This is the same argument pro-gun folk use in the USA to explain that the apparent correlation between their massive number of gun-related crimes/deaths and the lax restriction laws is actually just a coincidence... the Bad Guys would get guns either way after all.
If you can only get a drone with difficulty, then of course they'll still be used by criminals but only by more organised, serious criminals. Like guns in the UK... of course you can get one if you know where to look but most burglars do NOT have guns. Casual criminals... some chav flying a drone for a lark... will not make the effort.
If you REALLY clamp down on things, then you have to find a smuggling ring just to get the things into the country. They get much more expensive and you have to associate with really murky types. Genuine terrorists are really quite rare compared to idiots or those who seize the opportunity because it's easy to do so.