That's one of the main criticisms of Dysons' ideas (one he himself brought up IIRC), the idea that any race that had the knowledge and capacity to build one, would be so staggeringly advanced they wouldn't have the need to build one (beside bragging rights).
But on the subject of harming your planet... not so. If you build one in our solar system, you'd do it out near Jupiter's orbit, the Earth, Mars, Venus, Mercury, would be INSIDE the sphere and it wouldn't have much impact on them at all.
The shear amount of living space of a Dyson construct is really hard to get your mind around, you'd have land equal to billions of Earths, capable of housing hundreds of trillions of people without any crowding at all... Hell it would be like Earth with only a few million people living on it... It's brain hurting.
Though you'd need a lot of people outside bringing you raw materials to keep the civilisation going... that's stripping mining on a stellar scale kind of stuff... Hell you could dismantle every planet in our solar system and still not have enough raw materials for even a small percentage of it...