Speaking as a glider pilot, I don't have a problem with RC models because they are under the control of a human operator who should be able to see and avoid any full-size aircraft where he is flying. In addition, the RC model has to remain close enough to its operator for him to see not only where it is, but its attitude. Without this visual feedback its impossible to control the model. This is also why many models have different colouring top and bottom and often use assymetric colour schemes. I used to fly a bit of RC and found that using this type of paint job made the model far easier to control.
However, the thought of drones operating in class G airspace is very scary. Almost by definition these will be either autonomous or outside visual range of an operator but none of them, as far as I am aware, give the operator anything like the field of view or the fine-grained visual resolution that any GA pilot has and I don't think any of the autonomous drones have any optical see and avoid capability. In other words, current drones have little or no ability to stay clear of gliders, paragliders, microlites or balloons. These aircraft types do not usually carry transponders, so an autonomous drone that can't reliably use optical sensors to see and avoid a full-size aircraft is just an accident looking for a place to happen.
Recent reports confirm my assessment: American military drones have collided with manned aircraft (a C-130 no less), crashed because the operator didn't realise it was inverted, and had to be shot down by F-16s when the radio link failed: a whole litany of crashes and failures which all prove that drones have no place in civilian airspace or over towns, at least until the failings that led to these accidents all have proven, reliable solutions installed in every drone and subject to regular inspection and certification. Achieving this will take considerable time and is unlikely to be cheap.