Blackberry moment
As Snowden put it back in 2013:
Snowden then moved on to Blackberry Limited, chastising the company for “following an AT&T model” and accusing it of providing access to its clients’ personal messages to the governments of Canada, the United States, and India. “Blackberry will be erased from the pages of history” for its operating procedures, he said. “The customer is not really their customer, the state is their customer.”
Here Apple have attempted a dishonest walk back, but the problems still remain.
AI is an imperfect algo with lots of false positives, even if it was 1% FP, then given the small number of actual target images among the huge number of personal and family photos, it means statistically ~ 100% of images reported will be false positives. Even if they attempt to only flag if 32 images flag... its still 32 x noise = noise. And of course that 32, will become 1 as soon as challenged.
Showing your review staff only some photos makes the review less accurate, showing them more, makes the review more invasive. Showing them less detail makes it less accurate, showing them more makes it more invasive. There is no fix here, for false positives that is customer friendly, you are calling your customers pedos then asking your staff to secretly spy on those pedos customers to prove it!
They've realized the issue with various rogue states demanding their image sets be used for political and draconian purposes. Saying "we will only use sets from 2 countries" ignores the reality of that situation. The cat is out of the bag. US and UK, Russia and Belarus, Iran and Pakistan, China and Hong Kong, Saudi Arabia and Afganhistan, these pairs might do a little double act to humour Apple, but I think they will just ignore Apple's condition and still make the demand anyway.
Apple will comply, there is no legal basis for applying one countries limits to overrule another countries laws.
It's all too late, once they implemented that, announced that, they cannot unimplement it, they cannot walk it back.
FBI got their Apple backdoor, just like RCMP got their blackberry master key.
Too late to walk it back now.
https://www.vice.com/en/article/kz9kaa/exclusive-canada-police-obtained-blackberrys-global-decryption-key-how