back to article Identifying terrorists: Let's find a value for needle in haystack

So you're looking to stop a terrorist attack. What do you do? The choices are: (a) build the largest haystack about all the population because you know that the needle has to be in there “somewhere”; or (b) have the powers to look at all the relevant smaller haystacks that are around when you have inkling as to what kind of …

  1. Blank-Reg
    Big Brother

    No matter which way you look at it, this gets us to 80-90% government surveillance. The government will be able to search databases and know of your phones location, who your friends and acquaintances are, who you talk to and message on a regular basis (and in some cases the contents of those messages), what you've bought/sold, where you work, your salary, holiday destination, motor vehicle trips, elements of public transport journeys, communications you've made and your personal interests.

    The only reason they can't do 100% (yet) is there's no widespread facial recognition being used, bank notes aren't tracked and nor are bicycles displaying ID's. Oh, and covering your face from CCTV isn't an offence. Yet.


    1. Anonymous Coward
      Anonymous Coward

      it's worse than 'surveillance' look at JTRIG

      Specifically the GCHQ JTRIG psychological work on mass deception that was farmed out to the [named] British Psychological Society accredited University Psychologists

      I have been enquiring about this, but it seems like I'm the only one on the planet who cares, apart from the nice ppl at Cheltenham/Hanslope/Scarboro' etc

      The Internet of Stuff will probably screw up privacy 100% for ever - it's expected we'll have SEVENTY pinging micro/nano-items around each person within a decade, but what if everything that we read/see is also modified slightly from reality, all webpages subtly compromised . . .

      over to the sock-puppets . . .

      1. Anonymous Coward
        Anonymous Coward

        Re: it's worse than 'surveillance' look at JTRIG

        here are some "spy & ex-military" antics

    2. Anonymous Coward
      Anonymous Coward

      Storage is getting cheaper

      So long as it's legal to collect the data, it can be held until the software exists for deep analytics.

    3. Anonymous Coward
      Anonymous Coward

      Covering your face

      Re "...covering your face from CCTV ins't an offence. Yet."

      Actually, that can be an offence as per the recent Million Mask March in London, where:

      "The Metropolitan police had earlier imposed Section 60AA of the Criminal Justice and Public Order Act 1994 for a number of hours, which provides powers to remove masks when police fear a crime will be committed."


  2. Your alien overlord - fear me

    Someone has to say it - it's DIP shit.

  3. Anonymous Coward
    Anonymous Coward

    I'm sick of these half-baked anti-terrorist measures. Why don't they just finish the job properly and have everyone in the country imprisoned in solitary confinement?

    1. Anonymous Coward
      Anonymous Coward

      Alternatively, the govt could just make and then wear ISIS masks/flags and start bombing/killing/beheading anyone that wasn't confirm to their ideals, it's all in the name of "$Deity", which should make things hunky dory

      1. O RLY

        Why not jail everyone? You still need to produce; someone's gotta pay for all this oppression.

        1. Graham Marsden

          @O RLY

          > You still need to produce

          Yeah, but IDS will tell all the prisoners that they have to work or they don't get fed.

          He'll probably also tell them that work will make them free...

          (Yeah, that's a Godwin, so sue me...)

  4. Anonymous Coward
    Anonymous Coward

    Offed by a drone because of software bug.

    This will lead to you getting offed by a drone due to a flaw in the haystack searching software.

    We all know how unbuggy software is.

  5. Aristotles slow and dimwitted horse

    I hate to say it, but...

    I have a sad, dark feeling that the reason that our Illustrious Home Secretary has ommitted a lot of the safeguards and cross-referencing and aligning to other "higher ranking" fundamental bills of civil right is that this bill is essentially fait accompli.

    I'm not sure I want to overplay it too much, and looking at in the context of the overall political processes currently in play surrounding the civilian referendum on the EU exit, to my mind this bill as it currently stands represents nothing less than the rape of our democratic process unless a similar referendum is allowed on this insidious piece of "potential" legislation.

    1. Dan 55 Silver badge

      Re: I hate to say it, but...

      I'm also starting to think this bill would allow a complete full population database to be held by this or a future government or an alphabet agency should it feel like it.

      It is necessary and proportionate to have all the data if you want all the data.

  6. Anonymous Coward
    Anonymous Coward

    Needle in an database stack

    Putting civil liberties to one side (much like our civil servants and politicians do), finding a group of suspects using a single database would be about as quick as the computing power used to filter it.

    Yes, you will need an immensely powerful system, but we're not talking science fiction scales here.

    You have a criteria of a threat, eq Russian ecco-terrorist likely to poison a dignitaries food.

    Filter the populace for anyone frequenting Russian language, cooking and ecological sites. From that you further filter with any more known factors, after that best guesses and you'll wind up with a manageable number of suspects to scrutinise more closely.

    As easy as shopping.


    The major problem is how wide open it is for abuse by organisations well known for abusing even the smallest oportunity.

    1. frank ly

      Re: Needle in an database stack

      "... abuse by organisations well known for abusing even the smallest oportunity."

      Also, where are the legal penalties for agency employees/contractors who misuse the data?

      1. Mayhem

        Re: Needle in an database stack

        So one of my colleagues has a friend who works with the Met police, running interference between the Met and various agencies. His biggest issue is that MI5, MI6, GCHQ etc do not - ever - write anything down.

        The Met is required by law to document everything they do. His role lives and breathes paperwork. The agencies are pretty much expected not to. Also, no one working with them is allowed to say what they do, or expose any resources used by those agencies or even disclose that such things exist at all. While documenting everything. As expected, the documentation is usually so vague as to be utterly useless.

        In an environment where no one ever records what they do, how will you ever know that something was accessed inappropriately. After all, if they accessed it, they had a reason. That reason was verbally instructed to them, and they had access by virtue of their position, therefore their access was appropriate. Prove otherwise.

        Also disclosure of inappropriate access means discussing what it was they accessed, and that is against a different law. Have fun.

        1. Mayhem

          Re: Needle in an database stack

          Not to mention that any inappropriate access was clearly done by an individual, working alone and solely responsible for that access. After all, if it was sanctioned there would be a record, but there isn't, so it wasn't.

        2. Khaptain Silver badge

          Re: Needle in an database stack

          "In an environment where no one ever records what they do, how will you ever know that something was accessed inappropriately."

          As soon as a computer is used, then it is usually very well known who is doing what. Giving the information out is another matter.

  7. Pen-y-gors

    Bit of a no-brainer really

    I'm sure the vast majority of people would agree that the police should be able to investigate crimes, or (strongly) suspected crimes, and that, in the case of serious crimes (which needs to be clearly defined), would include asking a judge to give them the authority to access various records relating to the activities of known or suspected criminals. So far, so obvious.

    Whether people agree that unaccountable 'Security Services' should have any role in domestic law enforcement is another matter.

    The job of the police is to investigate crime and criminals, all crimes, including those committed by MI5, therefore they need data about criminals. They have absolutely no need to gather or access data relating to people who are not serious criminals.

    It may not be easy for them to work out how to differentiate, but given their general success with addressing serious crime already without draconian data trawling, I would suggest that they have more than adequete data available now, without the need to make us pay for our own surveillance through higher ISP costs.

    Serious crims will always find a way to avoid surveillance (El Reg commentards have already suggested many possible options) so only the law-abiding will have their communications tapped. So why bother?

    Dear Mrs May, I am not a serious criminal so kindly fec off!

    1. John G Imrie
      Black Helicopters

      Dear Mrs May, I am not a serious criminal so kindly fec off!

      You have shown anger towards am member of HMG and frequent a web site that by your own admission is frequented by people who know how to avoid the government's security measures.

      Welcome to the watch list citizen.

    2. Anonymous Coward
      Anonymous Coward

      Re: Bit of a no-brainer really

      "Dear Mrs May, I am not a serious criminal so kindly fec off!"

      Your El Reg user profile is easily accessible via Google - showing all the comments you have made since you joined. (Which is also true for everyone who doesn't use "anonymous").

      Black helicopter icon is assumed.

    3. Vic

      Re: Bit of a no-brainer really

      Dear Mrs May, I am not a serious criminal so kindly fec off!

      Doesn't matter. Although all the accompanying blurb always talks about "serious crime", note that the draft bill tself is rather less prescriptive: Section 46(7)(b) says:

      It is necessary and proportionate to obtain communications data for a purpose

      falling within this subsection if it is necessary and proportionate to obtain the data ... for the purpose of preventing or detecting crime or of preventing disorder

      You can drive a coach and horses through that. Any crime is sufficient - and they only need to be trying to prevent it; said crime may not have been committed (or even planned).

      Then, once the data has been purloined, there is no requirement for them to destroy the dataset, nor to hold it purely for the purposes for which they said they wanted it. Oh look - they get everything they ask for.

      I am dumbfounded that MPs en masse have not told May to sod off. This bill is possibly the worst one a Home Secretary has triued to push through in recent years. It must be defeated.

      IPB delenda est!


  8. Anonymous Coward
    Anonymous Coward

    "[...] the following appear to be targets for dataset acquisition [..]"

    That looks like it would also cover public library records and customer purchase records held by online book stores. Presumably store loyalty card usage records would also link a person to their brick store purchases.

    "Thought crime" is already effectively with us in some prosecuted cases - but this would allow trawling of the population for any desired profile.

  9. Anonymous Coward
    Anonymous Coward

    I'm torn on this. Part of me thinks 'nothing to hide, nothing to fear', but on a personal note, I've bought several books from Amazon which some could see as questionable ( Mein Kamp, Marx's Communist manifesto, etc ).

    To a dispassionate reader, that could be seen as worthy of putting me on a dubious characters list. The fact that I'm reading them out of historical interest, not out of support for those ideologies is not something I'd trust somebody else to guess.

    Even that's assuming a perfect state with no theft of information for blackmail purposes using embarrassing information such as MP's porn habits.

    Part of me thinks that racial profiling probably isn't a terrible idea for this, but that's hardly fair ( infringe other innocents liberties so I can be left alone ).

    1. Warm Braw Silver badge

      Mein Kamp

      Is that the German equivalent of Scouting for Boys? If so, I can see why you might be on a list...

    2. Anonymous Coward
      Anonymous Coward

      I think if they can read all your purchase history, they can probably temper your outwardly extremist literature choice with the other items and reduce your risk rating.

      1. Anonymous Coward
        Anonymous Coward

        "[...] they can probably temper your outwardly extremist literature choice with the other items and reduce your risk rating."

        Not if they apply the principle of the best place to hide a tree is in a forest.

        As has been said elsewhere - people who are on a mission to catch anysomeone will tend to from suffer confirmation bias. The less evidence there is - then the more weight they will give to any small thing they clutch at as being suspicious - even if in normal circumstances it would be perfectly innocent.

    3. Bernard M. Orwell

      "I'm torn on this. Part of me thinks 'nothing to hide, nothing to fear',"

      Consider this distinction: "I have nothing to hide, so I have nothing to fear" vs "I have nothing to hide, so WE have nothing to fear". What may be true for you may not be true for everyone. It's akin to saying "I have nothing to say, so I don't need a right to free speech, and therefore no one needs such a right."

      Even if we completely trust this government (hands up if you do trust them 100%), how can we be certain that we trust all future governments with this power? History is littered with the abuse of laws like this by governments with hidden or selfish agenda. To assume that such could *never* happen here is supreme hubris surely?

  10. Harry Stottle

    I'm now, officially, an Enemy of the State

    they have forced me into that position by becoming my enemy. Naturally, the arrangement is reciprocal. I suspect this conclusion will be shared by most other UK based Reg readers. Others have their own reasons for opposing their own states.

    In short, I'd much rather face the miniscule risks of a terrorist attack against me and mine, than the colossal risks they're about to impose on us, without any realistic opportunity for us to opt out.

    1. Anonymous Coward
      Anonymous Coward

      Re: I'm now, officially, an Enemy of the State

      The risks might be miniscule to you, but slightly likely to the population as a whole.

      Not that we'd find the risk ratings ourselves to make such judgement.

      1. Pascal Monett Silver badge

        Re: "miniscule to you, but slightly likely to the population as a whole"

        Like car accidents then ?

        Whatever happened to "we will fight them on the beaches, . . ." ?

        Gone, like tears in the rain.

  11. Warm Braw Silver badge

    Half a dozen cases where access to bulk communications datasets have produced results

    So what?

    Even in the unlikely event that there was no other evidence available in any of those cases, the ends do not justify the means.

    1. Richard 12 Silver badge

      Re: Half a dozen cases where access to bulk communications datasets have produced results

      So six cases since 1984?

      That's such a miniscule hit rate that psychics can beat it.

      Or darts thrown over your shoulder at a UK map. While blindfolded and newted as a piss.

      1. Anonymous Coward
        Anonymous Coward

        Re: Half a dozen cases where access to bulk communications datasets have produced results

        "[...] newted as a piss."

        The spoonerism is often quoted as "nissed as a pewt".

  12. dogged

    But the real question is not about about the capabilities - vile as they are. It's about the fact that the Bill seeks to make knowledge of those capabilities illegal and therefore, there never can or will be discussion of them.

    Nobody's going to take this one to the ECHR because it will be illegal for them to admit to knowing what they know.

    [citation provided]

  13. Anonymous Coward
    Anonymous Coward

    I wonder if all this data might lead to new and interesting ways for the bad guys to hide and get missed. If you know the data is being collected you could put up a front that looked very legitimate. Humans are fundamentally lazy creatures so most people even if they are suspicious will only look at the easily collected data. With all this data sloshing about who could be bothered to do the hard work of finding out if there are any secret communications taking place. Basically I think well end up with the situation where the idiots get caught and the clever criminals continue to get away with as they always have done. Worse this puts a pressure on the criminals to improve what they are doing e.g. move to fully encrypted systems.

    1. John H Woods Silver badge

      "Worse this puts a pressure on the criminals to improve what they are doing" -- AC

      But it does relieve pressure on the poor terrorists; after all, the security services were already too busy to prevent people on their watch lists from committing terrorist acts, so once there are very many more leads the terrorists can shelter in a very much lower signal-to-noise environment.

      A 99.99% effective terrorist spotting algorithm is going to give you at least 10,000 UK suspects. It's going to require about 90,000 field agents and at least 10,000 support staff to watch them 24x7; the salary costs alone would be around five billion pounds sterling per year.

      1. Anonymous Coward
        Anonymous Coward

        Vote Labour?

        Adding 100,000 civil servants to support the War of Terror sounds like a Tony Blair move.

  14. teebie

    "Annex 9 to his report lists half a dozen cases where access to bulk communications datasets have produced results"

    Does it? Or does it just claim to do that (which we really should be expecting by now)?

    From a quick skim

    case 1 claim someone was caught, but gives no proof of this

    case 2: claims there was a conviction, gives no evidence of this (case reference, name of the person convicted)

    case 3: unsubstantiated (and admittedly, probably unsubstantiable) claim

    case 4: see case 2

    case 5: so unsubstantiated it doesn't even name the country it supposedly happened in

    case 6: see case 2. Also, this was in Syria

    They might as well have added the following

    "case 7: Bulk data allowed us to identify that a teenage in america was babysitting for a family, but got 'stoned' on marijuana cigarettes, the family called and asked if everything was ok, she replied everything was fine, and she had put the chicken in the oven. Analysis of types not available through conventional interception allowed us to identify that there was no chicken, and she had put the baby in the oven"

  15. Rich 11

    Option (c)

    (c) Burn down the entire haystack and sift the ashes with a magnet.

    Well, that's what it feels like. Civil liberties and accountability don't count for much anymore, not in comparison to a few politicians crapping themselves that someone (possibly them, but not likely) might be killed on their watch. Not that deaths seem to bother them if it's a UK citizen being driven to suicide by a harsh and unjustified withdrawal of disability benefit.

  16. scrubber

    Dear government,

    I fear you more than the terrorists.

    To quote Blackadder, "please, please, please stop"

  17. cynic56

    Forgetting my annoyance at this disgraceful debacle ...

    I particularly liked this clarification.

    in addition “data” is defined to include “any information which is not data”.

    So the information is data because it isn't data?

    1. scrubber

      Re: Forgetting my annoyance at this disgraceful debacle ...

      Did Brave New Work and 1984 have a love child and we all* voted it into power?

      * I didn't because I don't vite so none of this macabre lambada is my fault. In fact I warned people this would happen. No one listened, said that by not voting I had no right to complain. Well, all parties are for this BS, so who should I vote for?

      1. Anonymous Coward
        Anonymous Coward

        Re: Forgetting my annoyance at this disgraceful debacle ...

        "Well, all parties are for this BS, so who should I vote for?"

        The Lib Dems used their coalition power to stop the last Snoopers Bill. If they had still been in a coalition no doubt they would have done the same to this one. However the general public blamed the Lib Dems for the things the Tories did get through - and so reduced the number of Lib Dem MPs who would have voted against it. Thus giving the Tories a clear run - as long as they can sweet talk the SNP.

        1. Vic

          Re: Forgetting my annoyance at this disgraceful debacle ...

          However the general public blamed the Lib Dems for the things the Tories did get through

          No we didn't.

          We blamed the Lib Dems for sacrificing what they claimed to be a "personal pledge" when the offer of a coalition power-share came along.

          Nick Clegg then compounded the situation by apoogising - not for breaking his pledge[1], but for making it in the first place.


          [1] This was a personal pledge, remember, Something he claimed he believed in. So don't talk about how manifesto pledges can be ignored if the party fails to achieve a majority - this is something he said he personally believed in, as did his colleagues.

  18. Captain DaFt

    "Needle in a haystack"

    Nope, wrong metaphor for what they're doing.

    What they're trying to do is look for a needle in a mountain of pins using a scrapyard electromagnet, without a clue as to how to find the needle after they've lifted the lot.

  19. Unbelievable!

    Yyou are fined one credit for a violation of the Verbal Morality Statute

    I've a number of intelligent comments to make, but that would take too much time.

    So this instead:

    "Why don't you just shove a leash up my ass?"

    "<insert name here> you are fined one credit for a violation of the Verbal Morality Statute."

    So how long before we lose the £ and find our selves in 'credits' ?

  20. rtb61

    Let's drop the bull puckey. All this spy vs terrorist crap, is crap. It is all about corporations versus political activists, whom the corporations want to be able to treat as terrorists because of political activists threat to unlimited power and profits.

    The corporations via their control of government want to destroy political activists, stick you head up to complain and they want to cut it off. This via prosecution as persecution, denial of access to public transport, denial of access to corporate or government employment, denial of access to financial services and this not just for the political activist but for their entire family.

    For all of that they need to track everyone all of the time, so they can put boot heel to head the very instant you approach the focal point of any political activism either individually or as a group (in that political activism include Labour Unions, peace activists, environmentalists and those who support free and fair elections as well as justice systems with honesty and integrity).

    Absolutely nothing what so ever to do with terrorists, unless of course you use the typical far right extremists US position where all those mentioned have been within three letter agencies declared terrorists (at least amongst the corporate appointees).

    This with the four of the five eyes all sliming along behind, four of them under threat of egregious political targeted espionage activities (so really a one eyed Cyclops and four blind mice).

  21. Soap Distant

    What do the general public know?

    I was at the pub with a small group of friends, mostly of the older generation but savvy enough to do one or two things online, last weekend. I was asked about the TalkTalk hack and how could it happen, what it meant, etc.

    Naturally I used patronising, non-jargon type language to explain (well, there's a mummy computer and a daddy computer...) The end result was that they were absolutely mortified to learn that there was no guarantee that their online activities were, well, private or safe.

    To my point then, Joe Public seem to be very unaware of the risks to them from so-called hackers (crackers/scammers): they seem to understand the risks of letting HM's Govt. frolic with wild abandon through their personal data even less so.

    I noted that this week's Panorama focused on hackers and identity theft, perhaps they'd like to educate the British public on the scope and ideas behind this new bill? I don't think the public are aware to any significant extent what this means to their personal data.

    A close family member and I sometimes discuss privacy and the Intertwebz, he was in the "nothing to hide, nothing to fear" camp, until I pointed out that everyone has something to hide, even if they don't realise they wished they'd hidden it more carefully later...

    Will we see a fall off in the usage of online services I wonder? I've noticed I'm more reluctant to do things online for sure.


  22. MrTuK

    I think you all have hit it on the head !

    Great feedback to this article, but like someone has already stated we are all probably on the hit list now since we read this article let alone responded to it.

    Which is really scary ;

    a.) The fact that George Orwell got it right !

    b.) The fact that George Orwell got it right all that time ago !

    c.) The fact that George Orwell let us know what was gonna happen and we let it happen anyway !

    d.) All of the above !

    e.) So will we have pre-crime justice system soon ?

    1. Anonymous Coward
      Anonymous Coward

      Re: I think you all have hit it on the head !

      "e.) So will we have pre-crime justice system soon ?"

      I believe we have. You can be convicted for possessing perfectly legal material if the court decides you were thinking illegal things when viewing it.

    2. DropBear

      Re: I think you all have hit it on the head !

      I have a small issue with c), as these things go usually nobody (of the public) gets asked whether they wish to "let it happen" or not. Actually the only way to make your dissenting point known seems to be with a hockey mask and a chainsaw, but that apparently gets frowned upon, so do it at your own risk...

  23. banalyzer

    Rejuvenate your mattress

    and download porn from different sites to /dev/null 24/7. Make VM work for that extra few quid every month.

    Then apply for a job to a government agency

    1. Anonymous Coward
      Anonymous Coward

      Re: Rejuvenate your mattress

      "Then apply for a job to a government agency"

      You would probably get the job. Remember that Tory Whip who said that MPs should bring their problems to him so that he could get them off the hook. His quid pro quo was that he then had a hold over them if he wanted something doing that they possibly would normally refuse to do.

  24. Anonymous Coward
    Anonymous Coward

    Missing (and really obvious) safeguards

    There are some strange oddities in the bill if you trawl through. For example, section 16 provides "additional protection for Members of Parliament etc" which requires the Prime Minister to be consulted on warrants affecting MPs, MEPs, MSPs etc. However there appears to be no similar obligation for warrants issued by Scottish Ministers.

    1. Anonymous Coward
      Anonymous Coward

      Re: Missing (and really obvious) safeguards

      GCHQ-Scotland will surveille the English ministers and swap data with GCHQ-UK who watch those in the Republic of Scotland and the Principality of Ulster.

  25. Whiskers

    Make more data

    As 'they' are so desperate to collect as much data as possible, surely it is our patriotic duty to generate as much data as we can. Let 'them' drown in it.

  26. Anonymous Coward
    Anonymous Coward

    Very unpersuasive evidence

    Does anyone else find the case studies in Annex 9 of the Anderson report remarkably unpersuasive?

    Cast study 1 covers two items, (1) tax collection by HMRC (how is that relevant, and what is MTIC fraud?), and item (2) about bulk data collection allowing the arrest of terrorists in the late 2000s. But if they were arrested why no details - surely by now they would have been convicted so the names could be revealed?

    Case study 2 is about an airline worker with links to Al Qaida - they surely doesn't need bulk surveillance, just interception of those working for airlines? That was in 2010 - again no details of his name and conviction.

    Case study 4 is about a known list of extremists who had been to Pakistan - again no need for access to data for the whole population, just for those under suspicion.

    Case study 6 is about men contacting children online - but these appear to have been specific overseas individuals. Obviously the authorities were right to check on these individual's communications, but there seems no evidence that bulk access was necessary.

    And there have only been six cases in the last 15 years, apparently. No names are given at all so we can check on these cases. But in the Appendix 10 which follows this section, names are named. So why so much secrecy here? These cases are so weak and unconvincing one wonders if there is anything in them at all.

  27. Dr Dan Holdsworth

    Data overload time

    As soon as you know that the spooks are watching for certain patterns, it is childishly simple to start overloading them with false positives. Most home computers in this country run Windows, and most of these Windows machines are not fully patched or are even running completely out of date operating systems. Therefore most computers in the country are vulnerable to a lot of malware that is out there.

    So, consider what happens when some smart malware authors get into the fray. These people would first of all try to make a mildly malicious bit of code that would try to infect as many systems as possible, and which once in a machine would start making HTTP GETs to as many dubious, dodgy websites as possible.

    Jihadi forums, pornography, radical politics, bulk nitrate fertiliser suppliers, cat litter bulk suppliers, red diesel and heating oil merchants, carbon credit brokers, EU VAT law forums; everything anyone can possibly think of to throw up spurious interesting coincidences.

    Do that in a few thousand households, and very quickly the GCHQ database is so skewed that only law abiding and frankly exceedingly boring people will ever get picked up, on the grounds that they're way too squeaky-clean to be real.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like