NSA to world+dog: We're only watching 1.6% of internet, honest The US's National Security Agency (NSA) has issued a document titled The National Security Agency: Missions, Authorities, Oversight and Partnerships (PDF) that explains some of its operations - and includes a claim it “... touches about 1.6 per cent... “ of daily internet traffic and “...only 0.025 per cent is actually selected …
I have no issue with the NSA monitoring traffic that the legal system has deemed they can capture. No secret courts are required. Courts can seal records and issue warrants without the recipient(s) knowing. Until the EFF, ACLU and the likes can perform their own review of the NSA program, how can you trust anything that the NSA has said or what they will say in the future? Much like the government couldn't prove backdoors in Huawei gear, but given that they used Huawei couldn't prove otherwise, maybe we should use the same system to the NSA and any government program? If the NSA has done nothing wrong, then they have nothing hide and should be welcome to an independent review that will clear them, right?
While I am against things such as the UK's Snoopers Charter, or any system that monitors ALL internet traffic & communication, I am fully in favour of targeted intercepts, and if that means they need infrastructures in place to intercept, fine, as long as it takes a court order to do so, AND it is only used for serious crime i.e. Murder, People Trafficing, Drug Smuggling... Anyone using these powers to catch a minor criminal should be punished severely...
As soon as you take away judicial oversight of interception, you pave the way for 1984, where the thought police are abound... Although right now the thought police already are moving upon us, as soon as they criminalised pseudo-images, they became thought police...
The interesting question for me is not what the NSA is *trying* to do, but what is in fact possible.
The chances that they or any other body can deliver value for money on this technologically ignorant and pork-laden political wild goose chase approaches 0.025% of 1.6%. There are no obvious incentives to apply an economic rationale. Even if there were, failure and success are both secret.
The NSA effort is politically inspired, but not politically accountable. It creates special interest groups that can use both secret failure and secret success to appropriate more resources.
It's sheer genius. It's the perfect scam. All that's missing is to charge the victims for their own interrogation.
Harry Buttle
Unpleasant it is, but NSA developed *one* way (there surely are others) of finding *some* targets. With all the indignant commentary, I have seen no suggestion of an alternative that allows both potential advance warning and going back for a short period after a missed event (NSA says 30 days, I think, for most retained data).
"the Internet carries 1,826 Petabytes of information per day."
Much of which is duplicated en masse due to being -say- on the BBC news site, much of which is porn, much of which is Beyonce videos on YouTube. By the time you cull out all of that heavy-bandwidth traffic and mass publications, I suspect that the entire world's email and messaging is somewhat close to that 1.6% figure.
So basically it's a mis-used statistic that's intended as a publicity sop.
I wouldn't discount the porn or the Beyonce videos, only their duplicates. When any new video or image is uploaded, that is 'touched by the hand of NSA' but when it is downloaded, only the TCP headers (or 'metadata') are kept. Those are freebies and don't count towards the 'data' total. The 1,826 Petabytes of information consists largely of duplicate client requests for previously 'touched' data. The NSA likely only needs to 'touch' 29 Petabytes in order to capture everything - every header, every porn file, every mundane image upload on every image sharing site as well as the more savory web pages, emails, spam (just one copy of each) and all. The only duplicates that the NSA are collecting in their 29 Petabyte Total Internet Trawl are the files that they were unable to determine were duplicates at the time of interception.
Anything that the NSA don't have from the Internet remains uncollected only as the result of an error, and that will be fixed and collected at a later date.
and that's why I only give you selected statistics and half-truths.
The whole existence of the NSA is based on the concept of undercover surveillance. Turns out the people are a bit scared about undercover surveillance and prefer an open, democratic society. No spin will resolve that muti-billion-dollar dilemma.
You wouldn't even need to capture the headers. The courts have already ruled that the headers are fair game. Your ISP has full right to that information, the payload they do not. The reason why the NSA doesn't need to collect the headers, there is a far easier way; NetFlow/IPFIX. The ISP's already have tools to collect this and virtually all the devices they use, support it in hardware. You can have multiple destinations or they could have their collector also forward it to additional destinations. This would give the NSA all the data they need and not actually have to install a clot of collectors like they would to collect headers. That would require the sue of taps or SPAN/Monitor sessions on the ISP's gear. NetFlow/IPFIX is far cheaper and easier to implement. Many devices you just cannot send headers, so the collector would need to discard the rest of the packet.
The figure that grabbed my attention isn't the 1.6% but the 0.025% which looks a lot more significant to me - that means that 1.56% of the traffic they receive is put through additional analysis. Filter out the video, spam, advertising, p2p, and porn and you must be up to 10% of everything else. In other words, anything remotely interesting. That could be you talking to your boyfriend about coming out as gay or your strategy in tendering for that multi-million pound contract against the US Megacorp.
This isn't about terrorism or targeted surveillance of specific subjects of interest: their own figures and a little common sense show this is a blanket trawl of any juicy tidbits.
"This isn't about terrorism or targeted surveillance of specific subjects of interest: their own figures and a little common sense show this is a blanket trawl of any juicy tidbits."
Exactly
And remember the rest can always be archived for later "review" should you become a person they have become interested in.
2,000,000,000 billion internet users, and they are watching 1.6%...?
2,000,000,000 * .16 = 320,000,000
With 320mil possible reviews, is cutting jobs sane? If one person could invalidate/validate 100 a day per (and that's a lot), that would mean it would take 3,200,000 people to review them all in 1 day. Or 3,200 people in 1000 days.
I'm bringing up these goofy numbers because the review process is supposed to stop terrorism (supposedly). With these type of numbers, how could they ever stop an attack on time? If the NSA gets a lead on a terrorist attack, then the attack would basically have to take place 2+ years in the future for them to stop it, or else the lengthy review process won't stop the attack.
So are they sure they aren't doing something else with the data?
By "touching" they almost certainly mean automated scans for names, phrases, flagged addresses, and suspicious activity like exchanges of encrypted mails (WHAT ARE YOU HIDING, CITIZEN?). It's been common knowledge that this has been done since at least the 90's. That doesn't take any people at all, but is a big part of the reason the NSA has as big server parks as Google.
By "reviewing" they mean that the stuff the automated scans have flagged are checked by an analyst. Nearly all the flags will be clearly innocuous, and each of the several thousand analysts can probably process several flags per minute. A very *very* small percentage will receive closer investigation.
Prior to Snowden spilling the beans the NSA lied to Congress about its activities, then further lied about the extent of surveillance until further releases caught them out again; the use of one-sided secret courts and gag orders pisses on just about every principle of accountability and oversight I've ever heard of.
You could pick any of a number of US denunciations of the Soviet Union and satellites from the 50's to the end of communism and they'd fit the NSA's core business nicely, yet we're supposed to trust them because this is some kind of 'good' spying on your own citizens.
Now, on an almost weekly basis we get an endless stream of hot air filled with facts and figures justifying this State Surveillance for how little is really done and many bad people its stopped - facts that cannot be checked or verified. Given the lack of real information offered, if it was all so bloody benign, why wasn't it simply revealed before?
I'd be amazed if there was anyone outside politics stupid enough not to see the NSA's fluffy facts and soft soap for what it is - the authors of the constitution saw these people coming, and they've been dead for two centuries. Rather than PR untruths, maybe they'd be better off spending the time penning their statements for the Truth and Reconciliation Commission I sincerely hope they'll be facing one day.
An interesting comment I saw buried deep in an article, I think on the Washington Post, is that members of the Intelligence Oversight committees gave up trying to get the Patriot Act amended for one very simple reason: they couldn't discuss the reasons for wanting the amendment as it relied on compartmentalised information. It's very hard to make a coherent argument for changing a law when you can't tell the people who will vote on the proposal why the amendment is needed.
The committee members have to read their intelligence briefings in a secure room and can't take any of the data out of that room.
Since all the committee does is ask questions (as I understand it they have no real authority to change anything without a vote of the full house), which makes the entire oversight process a waste of time. The only real effect of the oversight committee is that the electorate probably think that the committee is there to stop abuse of power. i.e. yet more security theatre.
After reading the WP article I concluded that the Senators' and Representatives' comments, including Mr. Wyden's, were mostly self serving CYA. Almost all of them were lawyers and all have access to lawyers capable of analyzing the effect of law changes that were not at all secret since they became part of the USC. The oversight committee members are not required to vote out a bill they have reservations about any more than they were to approve and repeatedly fund the Iraq war.
Sen. Diane Feinstein may say thing that now are politically quite incorrect, but she is at least honest about this.
Means any company that might be tendering for a US contract, any that might compete with a US company or any that make sales in the US and the IRS might like to take a look at.
Imagine how useful it would be for the inland revenue to have all Starbucks internal discussions with their tax lawyers (well not at all of course because they are a US company and so the UK government will roll over like good little puppets.)
Assumption: the NSA does most of it's gathering on data that transits US soil, since most data is sent via oceanic fibre it can't be sniffed off satellites or radio (yes, I'm deliberately discounting the assertion Snowden made that they've spliced beam splitters into Chinese fibres)
The obvious conclusion from that assumption is that they're probably very deliberately using a very large figure (total global Internet traffic) and figuring out what percentage of that is caught in their sniffers.
A more relevant statistic is probably the percentage of USA traffic that they capture. I suspect it's quite a bit higher than the 1.6% from their publication.
If you read the NSA white paper, you will see that they mention 30 countries that they have information sharing with, so think in terms of pretty much everyone in NATO, plus Japan, Korea, the Phillipines, Singapore, Australia and New Zealand being part of the system.
And yes, given what has been declassified about Cold War taps placed on Russian undersea cables, expect that the NSA has splitters on most of cables that come into China from the Pacific.
EMPowering Command and Remote Virtual Control of IT, without Leading Media Mogul Maintenance and Compliant Explicit and Implicit Complicit Participation, is the NEUKlearer Great IntelAIgents Game and Alternate Virtual Reality for Engaging SMARTR Futures and Derivative Markets Players. ........ and not at all a Trivial Heavenly Pursuit, for ITs Rewards are Bonded Boundless Bounty and an Absolute Dream of a Nightmare to Oppose and Seek to Destroy and Disrupt.
*And why would you think that they ever would need to share such intelligence services with you, should you find it so easy to disbelieve and dismiss the notion and its programming to allow ITs Beta Players free rein and dominion in all domains entered and accessed.
And what would think you could possibly do, which would have any discernible disruptive or positive effect upon pre-programmed outcome, if you were advised such is a SecuredD IntelAIgent Stealth and Advanced Active Astute Autonomy Program floated by GCHQ and CESG for Markets Capture and Capitalisation/Virtual Realisation and Monetisation ..... Price Fixing? And/Or would you fully expect it to be plausibly denied by systems heads/incumbent controllers/anonymous spokespersons fully briefed to speak on the cloudy matter and CHAOS** for their service ....... or if you prefer, fully debriefed to speak on the cloudy matter for CHAOS in their service.
Methinks though, whenever something is so powerful as to be invaluable, is that fixing markets price exercise really pointless and only necessary to stop the Markets Games from collapsing in on itself with mass realisation of its artificial core purpose and the abusive use for personal profiteering which its misuse by right ignorant and arrogant dodgy traders results in.
**Clouds Hosting Advanced Operating Systems
And what you may have to consider is the real truth as shared by Adam Curtis [of BBC documentary fame] ……
BUGGER… maybe the real state secret is that spies aren't very good at their jobs and don't know very much about the world….. and how it is worked and IT works it. ……. http://www.bbc.co.uk/blogs/adamcurtis/posts/BUGGER
Which then would present one with, when one is exceptionally good at what one does and makes one a person of interest, an ab fab fabless opportunity to provide states with that which they need to counteract what they be missing.
"We'd like to spy on more of you more often, but we just don't have the facilities to do that in real time. Good thing we have lots of backup capacity so any of you step out of line we can just do a quick restore and see what you history says about you. Trust us, we're here for your safety."
Somehow I find my myself strangely unconvinced.
E.g. in Canada 1/3 of all evening Internet traffic is Netflix, which the NSA wouldn't care about. Ditto for most video streaming (oh user pulled a Justin Bieber video, who cares). So that 1.6% of traffic of everything is probably a significant chunk of the actual Internet traffic that is interesting (email, VOIP, instant messaging, etc.).
he said ' this whole privacy thing is like a pendulum, it swings one way (TBL WWW 1989) then it swings another way (NSA/GCHQ 2013)"
The problem that I see - is that the pendulum of privacy is not guaranteed to ever swing back in the direction of 'free & private correspondence' - In Germany, who have their election in just a few weeks, they are already lining-up people to fire as 'responsible' for the BND/etc/NSA co-operation. (Steinmeier?)
in practise, all EU governments (interior ministries justice departments) signed up to the post-echelon PRISM system quite some time ago, If I can take all you honorable ( and the dishonorable lurkers) back to 1995
this was the writing on the wall
"Memorandum of Understanding on the Legal Interception of Telecommunications
The "Memorandum of understanding with third countries" (later described as the "Memorandum of Understanding on the Legal Interception of Telecommunications") was discussed at the EU K4 Committee in November 1994. The significance of the "Memorandum" is that it extends the agreement on the surveillance of telecommunications to non-EU countries who are being invited to adopt it - and with it the "International User Requirements".
The Memorandum of Understanding was signed by the 15 EU Member States on 23 November 1995 at the meeting of the EU Council of Justice and Home Affairs Ministers
The contact addresses for signatory countries and for further information, which confirms the EU-USA link, should be sent to:
"a) Director Federal Bureau of Investigation,
Attention: Information Resource Division,
10 Pennsylvania Avenue, N.W.,
Washington D.C. 20535
b) General Secretary of the Council of the European Union,
FAO The President,
Rue de la Loi 175,
B-1048 Brussels,
Belgium."
The number of signatories to the "Memorandum" was open-ended, any country can join providing the existing member states agree. It invites "participants" because "the possibilities for intercepting telecommunications are becoming increasingly threatened" and there is a need to introduce "international interception standards" and "norms for the telecommunications industry for carrying out interception orders" in order to "fight organised crime and for the protection of national security."
The strategy appears to be to first get the "Western world" (EU, US plus allies) to agree "norms" and "procedures" and then to sell these products to Third World countries - who even if they do not agree to "interception orders" will find their telecommunications monitored by ECHELON the minute it hit the airwaves.
Source: "Memorandum of Understanding concerning the lawful interception of telecommunications", ENFOPOL 112, 10037/95, Limite, Brussels, 25.11.95<
"not a significant document"<<<<<<<-----------------------WOW!
- the Home Secretary
The Chair of the Select Committee on the European Communities in the House of Lords, Lord Tordoff, took up the "Memorandum" with the Home Secretary, Michael Howard, in an exchange of letters on the Committee's access to documents for scrutiny.
On the subject of the "Memorandum of Understanding on the Legal Interception of Telecommunications" Mr Howard told Lord Tordoff:
"The Memorandum of Understanding is a set of practical guidelines to third countries on the lawful interception of telecommunications. It is not a significant document and does not, therefore, appear to meet the criteria for Parliamentary scrutiny of Title VI documents."
It is quite clear from this Briefing that the "Memorandum" is not an insignificant document concerning as it does a EU-US plan for global telecommunications surveillance.
The "Memorandum" itself is just two pages. It is the full text of the "Resolution" attached to it which demonstrates its full meaning.
However, not only did Mr Howard not think the "Memorandum" was "a significant document" he also apparently believes the attached Resolution also insignificant as he allegedly did not submit it to the House of Lords Committee for scrutiny prior to its adoption in January 1995 or thereafter.
Source: Correspondence with Ministers, 9th Session 1995-96, HL 74, pages 26-29.
Letter to international standards bodies
In December 1995 COREPER agreed a letter to be sent out to "international standardisation bodies in the field of telecommunications" (IEC, ISO and ITU) also ETSI. The letter said:
"Modern telecommunications systems present the risk of not permitting the lawful interception of telecommunications if they have not been adapted, at the standardisation and design stage, to allow such interception."
These bodies are "invited" to take account of the requirements of the Council Resolution of 17 January 1995 and told that Member States would be applying "these requirements to network operators and providers of services".
The December 1995 letter to international standards bodies and the publication of the main Resolution in November 1996 in the Official Journal announced to manufacturers of equipment and service providers that they will be expected to meet the "Requirements" allowing surveillance for any new contracts within the EU and via the "Memorandum" that these standards would also apply to any countries signing up to it - for example, the USA.
Source: "Draft letter to be sent to the international standardisation bodies concerning the Council Resolution of 17 January 1995 on the lawful interception of communications", Council General Secretariat to COREPER/COUNCIL, ENFOPOL 166, 12798/95, Limite, 14.12.95."
sorry for that long chunk of 'not significant documents' thanks to Statewatch & cryptome for hosting some of the sources - the MOU & attachments “ENFOPOL 112 file number 10037/95” has not yet been found online.
of course, all the above 'subversion' of the internet was done in the best possible taste, only the FBI is mentioned - but Snowden showed that the 'FBI' data goes straight to 'NSA' - and then perhaps some of it is returned to the 'FBI' for the purposes that it was nominally acquired for?
“This self-reporting is part of the culture and fabric of NSA,” the document continues. “If NSA is not acting in accordance with law, policy, or procedure, NSA will report through its internal and external intelligence oversight channels, conduct reviews to understand the root cause, and make appropriate adjustments to constantly improve.”
They should have finished with the word 'honest'. We all know that makes them sounds more believable.... honest
....given the actual grunt needed to get this stuff to fly, I would imagine that the percentages (given they will be spun for public consumption) are based on their current ability and not their current desire.
Also there needs to be a shedload of collusion between the NSA and those with the raw data (MS, Google, world+dog inc.) otherwise what you end up with is data noise that'll take forever to filter.
At the end of the day and after all this new "reasonable" cobblers to try and blag their way out of this, you haven't a cat in hell's chance of finding out what the true SP is, so take precautions.
It would be rather remiss of the security services if they weren't using the resources of Google and others.
The major commercial players have leading edge software and massive data capture as a matter of course, so are in a position to assess a large part of the internet relatively inexpensively. Google, for example, has been analysing net traffic to pick up hints of developing influenza epidemics for some while.
This is one reason why the NSA 1.6% figure may not be wrong, because other organisations are doing the pre-analysis for them. What is wrong is that no details of the arrangements are made public.
Paedos only rape 0.0001% of children, so that's OK too?
Terrorists only kill 0.001% of people, so is that OK?
Muggers only blight the lives of 0.01% of people, that must be fine.
Seriously NSA, you are meant to be smart. An immoral act is an immoral act, regardless of the number of people it affects.
"Released on Saturday with little fanfare, the document's prologue explains that the NSA lacked tools to track one of the 9/11 hijackers."
Interesting. So they had the ability to track the others?
But they still somehow missed that whole "Let's fly some airliners into big buildings" thing?
I'm feeling very reassured now that whatever they're doing is totally worthwhile.
WANTED - for friendly local media company in Washington DC.
Fancy sitting around watching the Internet and reading e-mails for a living? Then, we kid you not, there is a vacancy available at our friendly local company. Apply now! Credit and background checks will be conducted. Applicants must be able to demonstrate a love of cats and pussy in general.
Those sound such small numbers, don't they?
But, of course, they aren't *numbers*, they're percentages and is a neat way of burying the fact that actually huge amounts (in numerical terms) of data is being scanned on what is nothing more than a massive fishing expedition in the hope that, somewhere in all the dross, they'll come up with something useful.
Since the NSA are playing silly buggers, why not return the favour.
We need a small piece of software that will create an email with keywords from a suggested list (some but not all of the words) - thus plot, bomb, New York etc.
The same piece of software then needs to adjust a 250kb image of say the president of the USA, make sure the CRC changes, that the image metadate is different, date and time would be current to the program.
It would then take your outlook (maybe other e-mail clients) and mail a slightly different message of different length to each person in your address book.
In turn it should be able to receive such e-mails adjust the content slightly and maybe reply.
Finally we get a load of americans to use this tool for a couple of days, and thereafter on occasion to friends outside the USA.
From an individuals perspective, you get a bit more junk mail that you would normally get in any given day (but its in a good cause!).
From the internets perspective there would be an increase in email volume comprable to a large SPAM campaign, or the release of some topless photos of a famous celeberaty.
From the NSA perspective..........
I have a similar idea based on corporate bullshit generators... the ones you can supply your own vocabulary to. Then we could have
-the jihadist manifesto bullshit genetator
-the drug lord shipment schedule bs gen
-the people trafficing bs generator
The potential is endless...
Nice to know they select 'foreign entities which might be of interest'. With of course terrorists as example. And so it goes on - NSA is fighting the terrorists, for your safety, what's wrong with that? I'll tell you what's wrong. It has already been revealed that the European Community, and especially it's trade policy, is very high on the NSA watch list. Yes, their trade policies could harm "US interests" - their economic interests, that is. Whatever bad you can say about the EC, terrorists they aren't. And that is only one example that has been revealed - who knows who else is on the watch list?
...you're competing against US companies for contacts....
http://news.bbc.co.uk/2/hi/503224.stm
"Journalist Duncan Campbell has spent much of his life investigating Echelon. In a report commissioned by the European Parliament he produced evidence that the NSA snooped on phone calls from a French firm bidding for a contract in Brazil. They passed the information on to an American competitor, which won the contract. There's no safeguards, no remedies, " he said, "There's nowhere you can go to say that they've been snooping on your international communications. Its a totally lawless world."
The "NSA joke" is done by using different (self-contained) fonts for parts of the document (show up if you use acroread to convert it to Postscript) and then using character codes that only make sense visually with these glyphs
This stops you cut&pasting sensible text.
Presumably it also stops things like Google indexing it as expected.
So if you don't want the NSA to snoop on messages you send. perhaps you should take a leaf out of their books and send everything as PDF using substituted fonts? [Unless they've got a system for handling that anyway, of course...]
This post has been deleted by its author
The NSA is currently building a data storage centre in Utah for storage.
not Petabytes, nor Exabytes or even Zettabytes - indications are that they're aiming at YOTTABYES of storage.
That's more data than has so far been generated by the entire human race through recorded history.
IE: they want to sneeop every single byte passing their snoop points and keep it effectively forever.
Thiunk of all those selfies you'd prefer the world not see. Too late.
Without intending any disrespect, I would note that NSA is an intelligence agency doing mostly what such agencies do and that U. S. Constitutional and legal protections apply, generally speaking, to U. S. residents. Others are subject to other laws governments and might well wish to question those governments about internet and other data collection and analysis activities including, but surely not limited to, those of the USNSA.
@tom dial
WTF? Your naivety is stunning! Is the EU permitted to spy on every US citizen? No! The US would never permit it! So why is it ok for the big bully to spy on the entre world yet chastise China or Russia for the same conduct? The USNSA is elevating itself to an omnipotent position and justifying itself in moral terms, when in reality its spying program has a secret business agenda! You of all people need to study this :-
http://news.bbc.co.uk/2/hi/503224.stm
"evidence that the NSA snooped on phone calls from a French firm bidding for a contract in Brazil. They passed the information on to an American competitor, which won the contract"
“According to figures published by a major tech provider, the Internet carries 1,826 Petabytes of information per day. In its foreign intelligence mission, NSA touches about 1.6% of that. However, of the 1.6% of the data, only 0.025% is actually selected for review. The net effect is that NSA analysts look at 0.00004% of the world's traffic in conducting their mission – that's less than one part in a million.”
I'm concerned about the numbers along with several others. I would quibble with an order of magnitude.
0.025% of 1.6% is 0.0004%
That would be 4 ppm, not "less than 1 ppm." While an order of magnitude doesn't mean much when talking about millions, saying "less than 1 ppm" sure sounds good.
However, a larger issue, as others have noted, is disingenuously using percentages to distract from the actual numbers. The NSA is still collecting 29 petabytes while reviewing 7 terabytes - *daily*. As for the 1.6%, about ten years ago, I heard a telecomm researcher say about 70% of all network traffic is P2P. I think it's not too hard to ignore several other broad categories of traffic and realize that 1.6% represents important communication. The NSA spin doctors are twisting and misreporting (or miscalculating) the numbers.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
