"he was referring to the people who believe..."
No, sorry. It would be like seeing statues of Hitler and his generals in Germany, Mussolini in Italy, and Lenin/Stalin in many East Europe countries - part of their history? Surely, but a part that has no place in squares as symbols of it - because they are symbols of evil. And slavery is evil.
Most of the Confederate statues were also built later, well after the war, when - not to honor the fallen, but to establish supremacy again. It would take another one hundred years to honor the freed people rights.
Sure, many extreme left-wing activists are no better than their right-wing counterparts - but there are situations where there are no "fine people on both sides".