back to article UK govt sneaks citizen database aka 'request filters' into proposed internet super-spy law

A secret database of citizens' personal lives and habits isn't explicitly spelled out in the UK's latest surveillance law. No, instead, it's described as a set of "request filters." The term is buried in the draft Investigatory Powers Bill (IPB), which was introduced to Parliament on Wednesday. Turn to page 254 of the 299-page …

  1. Camilla Smythe

    OK... That's StalkStalk.

    Every URL you visit will be fed up the back end of a Huawei GreenNet device,

    and passed through a 'request filter' after the Chinese and various UK teenagers have had a snout at it... then they will try selling socks to you based on your browsing history in order to support the cost of the equipment.

    1. davemcwish
      1. Camilla Smythe

        Re: OK... That's StalkStalk.

        Obviously Huawei is 'good' enough for us Peons as defined by Ms Perry and Ms Dido but Ms Perry needs to know her e-mails are 'protected' by, NSA, professionals...

        ITMT my local elected numpty is quite happy to spaff all of his personal communications to Google...

        camilla@smythe ~ $ dig mx

        ; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> mx

        ;; global options: +cmd

        ;; Got answer:

        ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28976

        ;; flags: qr rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 1


        ; EDNS: version: 0, flags:; udp: 4096


        ; IN MX

        ;; ANSWER SECTION: 14400 IN MX 10 14400 IN MX 10 14400 IN MX 1 14400 IN MX 5 14400 IN MX 10 14400 IN MX 10 14400 IN MX 5

        Last I did not hear Ms Perry's sons did not manage to haxor the pron filters or get arrested for hacking TalkTalk... becoz.

  2. Anonymous Coward
    Anonymous Coward

    Proxy or carrier-grade NAT?

    Presumably this 'filter' requires the ISP to be able to trace back through their proxy or CGN device to associate an outgoing connection with the subscriber making the request. No citizen database required, just proxy server and NAT logs.

    1. JetSetJim
      Paris Hilton

      Re: Proxy or carrier-grade NAT?

      Yes, I'm curious as to why this needs a database of the proles - I read it as "when we make a request, we may include other identifying information that the respondent ISP is required to attempt to match in a defined way" - e.g.

      Please tell me all about IP address aaa.bbb.ccc.ddd, where subscriber name is similar to "J Doe", and where geolocated, it was near Hyde Park on the 3rd Jan 2015. If a match is made with a pukka subscriber, give me everything to do with them.

      Or is this more a generous interpretation that the civil servants will attempt to create a database with this legislation?

  3. frank ly

    Can someone explain this?

    "However, each IP address cannot be resolved to a single individual because at the known time it has been simultaneously shared between many internet users. "

    I know what my domestic IP address (from Virgin Media) is and it's been the same for over four years. If I go to and similar sites, it tells me. Also, from outside my home network, I can access my domestic FTP server using that address. I thought that nowadays it was only people on dial-up connections who had an IP address that was the same address as someone else had, but at a different time. What have I misunderstood or missed here?

    1. Andrew Jones 2

      Re: Can someone explain this?

      Multiple people who live in your residence and therefore the bill seeks to be able to separate details of what sites they have visited from what sites you have visited. Likely using information from browser headers - but the only way I can see of them being able to do this - is to build up a profile of each user of that IP address - and I don't like the sound of that one bit.

      1. Dadmin

        Re: Can someone explain this?

        Two methods for identifying you by hardware ownership, behind a NAT, are: your hardware MAC address, and a profile of your cookies from your browsers, gathered in zero days and ISP bulk traffic slurping. And, of course, your Government Approved Spy Firmware updates you may not have known you requested yet, phoning home in the dead of night to tattle on you to the town fucking council about your secret habits. Citizen, use the convenience of the Internet at your own peril. You're being tracked in real time.

        1. g e

          cookies from your browsers

          So, actually, incognito mode can help here by destroying cookies and demanding new ones every browsing session? (Or just setting your cookies to session-only, too, I spose)

          That does sound like a little more than collecting the domain name you're visiting though, seeing as cookies are in the request header data, not the GET itself. I am right thinking they claimed only the domain from the GET was being 'remembered' ?

      2. frank ly

        @Andrew Jones 2 Re: Can someone explain this?

        A family home, yes ... but it said "... many internet users ...". Maybe they mean libraries, internet cafes, universities, companies, etc? (I should have thought of that, thank you.)

        1. Andrew Jones 2

          Re: @Andrew Jones 2 Can someone explain this?

          I believe there is already a law that requires commercial use to be recorded - eg you must have some form of identifying information about who is using the connection in case records need to be accessed in the future. For example - McDonalds wants postcode and email address (though it can be argued that's more for marketing), Tesco want Clubcard number. Internet Cafes usually require some form of indentification. Companies of course have your login details as do Universities, Schools, Libraries etc. I don't really remember much about it now - other than it was designed to tackle copyright and a lot of Internet Cafes and smaller commercial establishments complained it would effectively kill business. Anyway there is a Wikipedia page with more information or you can Google "Digital Economy Bill"

      3. OldBiddie

        Re: Can someone explain this?

        "the only way I can see of them being able to do this - is to build up a profile of each user of that IP address - and I don't like the sound of that one bit"

        The bill, of course, makes provision for expanding its remit to make this level of data gathering legal without any need to pass further laws, in fact, there are clauses in the guise of "training" and "R&D" dotted all over the place that are so loosely worded they could be used to bypass any kind of judicial oversight.

    2. Anonymous Coward
      Anonymous Coward

      Re: Can someone explain this?

      "What have I misunderstood or missed here?" Mobile phones. The carriers have NAT and proxy systems to condense the millions of subscribers down to a smallish number of IPs.

      1. Sebby

        Re: Can someone explain this?

        Best guess: I think they're talking about address sharing / CGN. In which case should consider the potential long-term benefits of not trousering dishonest money given to them by ISP lobbies from the short-term implementation of unsustainable and unjust IP address extension plans, and instead get this dank little island on to IPv6--that is if they can find the time between spying on everyone and making them bankrupt through cuts.

        But I could be wrong.

        1. Ken Hagan Gold badge

          Re: Can someone explain this?

          "and instead get this dank little island on to IPv6"

          Sshhhhh. Don't tell them about IPv6. With any luck they haven't noticed yet and the legislation will only apply to IPv4 connections. IPv4 is *next* year's panic (and the ISPs will no doubt say that they'll need a bung four times larger to record the data).

    3. scrubber
      Paris Hilton

      Re: Can someone explain this?

      I have my rather large connection open for all and sundry to use so there is some reasonable doubt as to who it was that downloaded whatever thing the government has a hard-on for this week.

      Paris - open to all and sundry.

      1. Bloakey1

        Re: Can someone explain this?

        <snip good point>

        "Paris - open to all and sundry."

        The difference twixt Paris and the Eiffel Tower is that only four million people have been up the Eiffel Tower.

        Saunters off singing I love Paris in the spring time ...

    4. iainr

      Re: Can someone explain this?

      Not everyone on ADSL or Fibre has a static ip address, some ISPs still provide dynamic ip addresses which can change over time.

      1. Ken Hagan Gold badge

        Re: Can someone explain this?

        "Not everyone on ADSL or Fibre has a static ip address, some ISPs still provide dynamic ip addresses which can change over time."

        But at any given moment, even with CGN, the combination of IP address and port number must resolve to just a single customer or else it doesn't flipping well work.

  4. Anonymous Coward

    Governments, they can all fuck themselves.

    1. Destroy All Monsters Silver badge
      Paris Hilton

      I do think it would be more expedient if WE fucked them?

      1. Bloakey1

        "I do think it would be more expedient if WE fucked them?"

        In a spirit of adventure I would be ok with prostitutes, rent boys and the odd vagrant of either sexual persuasion but I have to draw the line somewhere. Politicians, journalists and lawyers are a rubicon I will never cross, unless by "fuck" you mean terminate with extreme prejudice.

        It is indeed quite frightening the kind of accurate profiles that can be built up with metadata, browsing histories, contents of your dustbin etc. I once saw a specialist produce a frighteningly accurate description of someone based on their dustbin contents over a few weeks, they then repeated it and told me about his visitors etc. based solely on his Sainsbury nectar card details.

        I do not have any of that old crap and live quietly in the hills of a third world EU country where the average wage before tax is 100 Euros a week.

  5. phil dude
    Black Helicopters

    thought experiment...

    Let's say you have a vpn gateway in your house/flat. i.e. all traffic goes $ABROAD

    Let's say DNS is pointed at Google.usa , so all DNS is via VPN.

    How long until you get a knock on the door?

    Answers on a postcard...


    1. Anonymous Coward
      Anonymous Coward

      Re: thought experiment...

      Don't know, I'll let you know, however I won't be using Google's data gathering services.

      1. Tom Samplonius

        Re: thought experiment...

        "Don't know, I'll let you know, however I won't be using Google's data gathering services."

        I'm sure the mandatory data gathering service that the gov't is forcing ISPs to assemble, will be more effective anyways. Google just wants to click on ads. The UK gov't wants to know if you are planing to blow something up.

  6. Anonymous Coward

    does this mean

    I should start with and make an http/https connection to every reachable site on the internet? I'm sure if all of us did that the disk drive manufacturers would be ecstatic.

    1. Mark McNeill
      Black Helicopters

      Re: does this mean

      Be that as it may, I'm going to cc my MP into all my emails from now on, and invoke the Wilson Doctrine if Theresa May wants to read them.

      1. This post has been deleted by its author

        1. JetSetJim

          Re: does this mean

          The Wilson Doctrine isn't respected by Theresa & co

    2. Supa

      Re: does this mean

      I was just going to say the same thing. Let's kill them with information overload. You could automate this using a macro recorder and throw in a random number gen into the mix. The first person to make a browser extension to do this deserves a beer.

    3. choleric

      Re: does this mean

      You could do that, but if you wanted to eke out every last bit of annoyance/ecstasy you might want to start with instead :-p

      1. P. Lee

        Re: does this mean

        >You could do that, but if you wanted to eke out every last bit of annoyance/ecstasy you might want to start with instead :-p

        Oi!, gerrof my home network ;)

        NAT'ed internet-legal addresses FTW or lose, or confusification!

        I wonder if I can get the addresses from my private home network to show up in carrier proxy logs? Maybe I'll get Squid to pretend we're all using TCPWare on VMS from Apple's network.

        I think it might be relevant that government IT is now probably large enough that you can't flood them with data as a DDOS or, likely, signal/noise defense.

        One question would be, if NAT is a problem at an ISP level, Why is it not a problem at the home gateway level, where there are no logs. Are they just relying on mobile device use - as the geolocation suggests?

        Do you think they know this will end up badly for everyone and just don't care or do they really think that (a) it will do some good and (b) they will get away with this in the long run when the general populace realises what's going on.

      2. g e

        Re: does this mean

        And save time by skipping 10/24 and 192.168/16, etc too :o) (Yes I probably got the CIDR wrong but you get the idea)

  7. Chris G

    How long

    Before anyone using a VPN is automatically assumed to be a bogy. If you are not doing anything wrong, why are you hiding behind a VPN?

    1. Anonymous Coward
      Anonymous Coward

      Re: How long

      If you are not doing anything wrong, why are you hiding behind a VPN?

      Oh, that's an easy one. Simple question: does the government want me to exercise my duty to my staff to filter their Internet use from dodgy sites or are they prepared to accept that responsibility themselves? A VPN means I haul the traffic back onto the DMZ and filter outgoing, no DMZ means I break various laws they themselves imposed. Duh.

      Can they serve me with a warrant for access? Of course, any government in any country can do this as it's the law, but at that point I know who does what and who I have to drag into court if they expose my data or that of the customers. The activities that happen behind my back are the problem and that I endeavour to make intercept as difficult as possible.

      Fortunately I saw this one coming, not inconsiderably helped by the previous government which had a genuine fetish for surveillance. We have it covered.

    2. A. Nervosa

      Re: How long

      @ChrisG - Two words: traffic shaping.

      I'm on O2 and it's very obvious indeed that they're throttling the Apple Music download speeds to my phone over cellular. Streaming is fine but any attempt to download the tracks so I can listen to them on the Underground and the download speeds are barely any faster than streaming. 30-40 minutes to download an album for offline listening is a complete piss take.

      Of course when I connect to my VPN and O2 can't see what I'm doing, blam, those tunes beam through the airwaves at the very limits of my 4G connection speed.

      Unless of course you consider getting the service that you actually f--king pay for as 'wrong'.

  8. chris 17 Silver badge

    The meat on the stick

    So the request filter is the piece that differentiates user from user where the Internet connection is shared. To do this they are obviously recording more than just ip and url.

    It won't be long before its known exactly how the filters work and techniques to obfuscate what they are looking for are wildly shared rendering the whole snooping initiative pointless.

  9. Anonymous Coward
    Big Brother

    Archibald Buttle

    What scares me the most about this legislation is the thought that the people implementing these 'filters' are the same highly-skilled, highly-paid, dedicated programmers who developed the customer data security regime at TalkTalk.

    1. Triggerfish

      Re: Archibald Buttle

      You should hope they are.

  10. Anonymous Coward
    Anonymous Coward

    Buy disk shares now

    There are 44 million users of the internet in the UK. They are going to generate a shed load of data.

    Add all those applications that use http as connection/data protocol

    Then add applications like like BT's use of backhaul on domestic routers.

    Then add all that adware we get hit by.

    Then add all the analytic data.

    Oh and just imagine somebody develops an application that generates random web site access just to fill up ISP logs

    and your really stuffed. You ISP have to record all this, in a form that plod/spook/council/man on street/hacker can access

    Meanwhile the bad guys use vpn and dark web to avoid detection and joe public pays more to his ISP to have his own data recorded as he has to pay for the storage needed to record his activity.

    1. Chozo

      Re: Why Imagine

      Roll your own URL spaffing widget today with a copy of the free and simple to use Djuggler web scraper package. Use this grab the list of freshly registered domains from the likes of GoDaddy and others then have it spidering merrily away. Works on windows & linux,

    2. Martin-73 Silver badge

      Re: Buy disk shares now

      It won't just be the bad guys using VPNs... it'll be the good guys who just had enough of theresa may's ridiculous orwellian ambitions to make hitler look like a nice guy.

      And no, it's not invoking Godwin when the comparison's accurate

      1. choleric

        Re: Buy disk shares now

        Actually it is a Godwin comment because the whole point of that truism is that somebody somewhere on the internet thinks at least one of your opinions is as bad as the Nazis, just as you do Theresa's.

        Just because it's a Godwin comment doesn't make the comparison invalid or inaccurate though.

  11. Anonymous Coward
    Anonymous Coward

    How long...

    How long before an unencrypted copy of the "request filter" database is left on a train || lost in the post || sold by a corrupt employee?

    1. Neil Alexander

      Re: How long...

      Not long at all, given's track record.

      1. Anonymous Coward
        Anonymous Coward

        Re: How long...

        have an upvote .. but, on second thoughts . given's track record, it's probably already happened

  12. Otto is a bear.

    The point of investigations

    When an incident occurs that requires investigation, the investigator is faces with many pieces of information which they need to filter to discover their relevance. This is an iterative process which draws the focus onto the correct target. This is true for any investigation, and any source of information, be it internet or otherwise. The difference between public safety investigations and any other is that the targets will not give their consent to be investigated, if they are known at all. The question you need to ask, is, how could it happen if public safety officers can't do this. They are always far too busy to follow up irrelevant leads.

    All other investigations carried out for research by public or private organisations is consent driven, you give your consent to have your life examined to determine the colour of your next sock offer through your loyalty card or purchase contract, in effect because, no matter who you are the investigation's outcome is beneficial to both sides. Public safety investigations generally have a winner and a looser.

    1. Warm Braw

      Re: The point of investigations

      Except in practice, that's not how investigations occur, particularly when the incident has a high public profile. What happens is that the investigators typically fix upon someone who fits their vision of the target and then try to find information that confirms their prejudice. If you keep a large enough data set on every individual, there will always be something that looks suspicious - more so if you have a means to filter out the context.

  13. Crisp

    They aren't even pretending to hide their snooping now.

    What's it going to take to convince them this is a bad idea? Everyone in the UK putting on a Guy Fawkes mask and marching down to London?

    1. billse10

      Re: They aren't even pretending to hide their snooping now.

      "Everyone in the UK putting on a Guy Fawkes mask and marching down to London?"

      well, it's a good day for it ....

      1. Bloakey1

        Re: They aren't even pretending to hide their snooping now.

        ""Everyone in the UK putting on a Guy Fawkes mask and marching down to London?"

        well, it's a good day for it ...."

        Let us celebrate the last honest man to enter the portals of parliament.

  14. Anonymous Coward
    Anonymous Coward

    I hear AirVPN is quite good. Cheap too. Time I stopped procrastinating and sort out my comms for the modern age.

    How long before I'm on a watch list I wonder. Though, if they go through all that effort, all they'll find out is that I like wargaming sites, clothes and, uhhh, gentlemens downloads...

    1. g e

      You probably already are

      Be ruddy surprised if I weren't, even a tiddly low-level one.

  15. Bronek Kozicki

    The petition to revert this [unspeakable] is here

    1. NP-HARD


      You should too.

  16. Anonymous Coward
    Anonymous Coward

    Protest Idea

    It doesn't happen often but I've had an idea...

    On Monday the 9th November, the day after we remember the men and women that fought for our freedom, rather than throwing your poppy away mail it to you local MP (include a note if you want). Perhaps if they get enough poppies they will understand what freedom means.

    If you like this idea please spread it.

    1. Anonymous Coward
      Anonymous Coward

      Re: Protest Idea - template letter

      > rather than throwing your poppy away mail it to your local MP (include a note if you want)

      Great idea, but for the penny to drop with our MPs, a letter will be required. I suggest:

      Dear [your MP],

      I'm sure you will have immediately noticed the poppy pinned to the top of this letter. This poppy has done its bit this year, helping to commemorate the many thousands who gave their lives defending this country and its freedoms. Now that Remembrance Day has passed, I enclose it to emphasise just how tragic it would be if our freedoms were to be given away cheaply.

      The Home Secretary, the Rt. Hon. Theresa May MP, recently presented to Parliament the Draft Investigatory Powers Bill. When debating and voting on this Bill please will you do your utmost to ensure that:

      - all access to collected and processed data is subject to judicial oversight and approval (otherwise no one will ever be held to account regarding improper, overly broad or overly long-running requests);

      - only the police and the security services are permitted to see collected data (otherwise we will see repeats of Councils using the information to check that parents live near their choice of school and other, completely inappropriate, 'group-think' ideas); and

      - there is a 'sunset clause' included in the legislation (otherwise we are effectively admitting that the current threat levels will remain forever).

      Three simple points. If this legislation is as vital as the Home Secretary claims then it is vital that it is good legislation: please hold her to account over it.

      Yours sincerely,

  17. Werner McGoole

    I seem to have lost the plot

    GCHQ slurps all the packets off the UK's internet pipes, so they know exactly who connects to where and when. They almost certainly have tools to query this information pretty extensively.

    So why are we arguing about ISPs duplicating that data and there being another way of querying it?

    Someone please explain what I've not understood.

    1. Anon5000

      Re: I seem to have lost the plot

      Police will be able to access all this information too. Without a warrant.

  18. Teiwaz

    I will not be filed, stamped,indexed, briefed, debriefed or numbered...

    A dataabse...

    So this is the future of 'investigations' now is it?

    Develop a set of 'profiles' that describe 'dangerous online activity' and run it through the database every few months, then 'visit' those identified, cart them off and then sort out which can be prosecuted....

    Maybe this isn't what they intend with this, but it's how it'll end up.

  19. Andy The Hat Silver badge

    OOOh nasty thought ....

    If some script kiddies decide to implement a massive DDOS attack - or two or three simultaneuously - will this mean that, under law, the ISPs involved will have to record each and every web page request? I would have thought this would not only kill the target of the DDOS but the entire ISP's infrastructure too?

    Thinking out loud, are there exclusions to address recording? What about DNS requests, ntp and all the other stuff that is not usually typed at the keyboard by the originator (although it could be) but nonetheless is a perfectly formed request packet which could be used to disguise data ... If it's only "www." addresses that are stored will gopher and ftp and usenet start to make a comeback? If it's every IP address accessed it could make the system grind to a halt.

    1. This post has been deleted by its author

  20. davemcwish

    Request Filter

    Given that this is the 2nd attempt to introduce this I think it's highlight likely that it already exists and they're after retrospective authorization to use and demand ISP's populate

  21. PatientOne

    So... you go to a website that has linked images (Like this one or the BBC).

    Your browser connects to the websites where the images are stored in order to retrieve/display them.

    This is recorded so the police/GCHQ/random hacker can look at what sites you've been visiting.

    Am I the only person who can see a problem here?

    (No, I really don't think they'll put in any form of failsafe to filter out such connections).

    1. jay_bea

      Even better, hack the Ad network so that requests are redirected to dodgy sites, not only obscuring any actual criminal activity, but also landing lots of people on a watchlist.

      The poor state of security on an ad network you have never heard of could flag you as a person of interest subject to much closer surveillance, without you ever knowing about it (at least not until the Police come knocking at 6am for your computers).

      1. Anonymous Coward
        Anonymous Coward

        Posted this on el reg a couple of months ago, you may be interested

  22. Richard Morris


    Unfortunately, it was meant to be a warning rather than an instruction manual.

  23. dickiedyce

    Wait a sec...

    " The request filter will mean that when a police force makes such a request, they will only see the data they need to. Any irrelevant data will be deleted and not made available to the public authority."

    Sounds like a table view to me... oh dear, I've just realised...


    It's not a database. It's an Excel Sheet.

    1. davemcwish

      Re: Wait a sec...


      They'll also be on the 2003 version so be limited to 65,535 records.

  24. Anonymous Coward
    Anonymous Coward

    It's our old friend the ID card system, this time on steroids & minus the little plastic card

    No central database to complain about because now it's a distributed database.

    No increased taxes to complain about because now it's our ISP's who have to put their prices up to pay for it all

    The IPBill not only legalises currently illegal surveillance activity, it introduces a hell of a lot more & gives powers to people who don't currently have them

    We have to stop this, now.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like