Say it again
The standard 'off the shelf' consumer drones usually rely on higher frequency transmitters with limited range - due to a need to send video back to the operator. Whilst you can buy custom R/C gear that travels further, if you're flying a drone bought off the shelf, your limit is usually around 2-4,000 feet (up to about a mile). Battery limitations kick in if you're trying to ascend or travel long distances. You *can* buy aerials to improve range, but you have to be pretty committed to get it all working reliably.
If you want to annoy aircraft, you can buy the kit to do so, but you're better off making a balloon with a payload (hello, Register).
Funnily enough, before there were drones, aircraft pilots commonly reported UFOs. In at least a few cases, I suspect they see what they expect to see.
I can understand the 'lets ban them before someone gets hurt' argument - but have to point out that
a) The people flying irresponsibly with custom kit will still do so
b) It's actually pretty hard to hit something up there
c) Planes are tested for major air strikes
d) We don't seem to apply the same logic to autonomous cars and other technologies, or even the idea of Amazon carrying 1 kilo parcels a few hundred feet above public property and people.
e) Drones are being used responsibly in a whole range of new areas, and the technology is changing incredibly quickly. Heavy handed regulation at this point will stifle many emerging uses.