There's another downside too. The resources ploughed into breaking encryption, are resources that can't be used for (eg) following up on reports that Dodgy Bob has been going around muttering about 'killing all the infidels', and maybe someone should pop round and check he's not trying to make a bomb out of tin foil and Swan Vestas...
The trouble with breaking encryption, or tapping more lines of communications, is that it just increases the size of the haystack that the security services have to search through.