On the other hand..
Twice I've been asked to stop taking pictures in London. Once by a private citizen who told me it was against the law to take pictures of buildings! I told her she was mistaken and carried on. The second time was by a police officer at Earls Court station. She said it was due to "anti-terrorism law". I stopped as I'd already got the shots I wanted.
Of course if they do ask you to remove the pictures you could always delete the images while the Old Bill are looking over your shoulder and then run an "undelete" program on your card when you get back home. ;).