back to article Cloudflare's bot bouncer blocks weirdo browsers

Users of some of the less well-known web browsers are getting blocked from accessing multiple sites by Cloudflare's flaky browser-detection routines. Aside from reporting it on Cloudflare's forum, there appears to be little users can do, and the company doesn't seem to be paying attention. Cloudflare is one of the giants of …

  1. wolfetone Silver badge

    "According to some in the Hacker News discussion of the problem, something else that can count as suspicious – other than using niche browsers or OSes – is something as simple as asking for a URL unaccompanied by any referrer IDs. To us, that sounds like a user with good security measures that block tracking, but it seems that, to the CDN merchant, this looks like an alert to an action that isn't operated by a human."

    It's also something poor scrapers/bots do too. Often they'll just trawl through a load of links and go direct. So one tactic CloudFlare (and others) do is block or slow down requests that don't contain that tracker information. But then good scrapers/bots will always go to the root of the site, then simulate clicks through to the desired page to scrape it.

    But this method that they're blocking on is for the low hanging fruit, the "swiss cheese" approach if you will.

    1. katrinab Silver badge
      Flame

      Take the Wall St Journal for example. My bot can scrape stock prices on it just fine, bypassing CloudFlare is waaay easier than it ought to be. Not going to publish the details of how I do it for obvious reasons.

      However, just visiting the website as a regular Chrome user on Windows 11 can be quite a challenge.

      1. Jamie Jones Silver badge

        Just like with the media industry and their anti-piracy efforts, it seems that the legitimate users suffer the most.

        As for checking "User-Agent", surely that's the first thing a bot spoofs (the bots that ignore robots.txt on my sites do, at least), so again, it's less known legitimate browsers that set a legitimate header that are penalised.

        1. andyprough

          "the legitimate users suffer the most"

          Which is what you would want if you were being paid to push more users to using certain popular browsers. It's the carrot and the stick approach. Or just the stick in this case.

          1. Goodwin Sands Bronze badge

            "Which is what you would want if you were being paid to push more users to using certain popular browsers."

            Well if Cloudflare are being paid to discriminate against certain browsers (either money or just a scratch my back type arrangement) then they & whoever is paying them are squarely in breach of anti-competitive law in any number of jurisdictions.

            Likely for starters

            abuse of dominant market position

            exploitative abuse / discriminatory behaviour

            exclusionary tactics

            and maybe collusion & vertical restraints as well.

            If a regulator gets interested then Cloudflare will regret what they're doing.

            1. katrinab Silver badge
              Meh

              Which browser / OS combination is more popular than Google Chrome on Windows 11?

              OK, maybe it is Android or iOS, but what if I want to look at it on a 28" monitor?

          2. bombastic bob Silver badge
            Unhappy

            I've sent nasty e-mails to my bank's "new" web interface maintainers for NOT supporting firefox... have not YET switched banks though. sadly too much trouble to do so

    2. cosmodrome

      The swiss cheese approach indeed. Supposed you're intending to catch all the holes and let the cheese pass. Referrer and user agent filtering were well established practices on 1990s porn sites. Never failed to annoy -strictly scientific- visitors and doing little to stop content scrapers. But maybe it's working better nowadays - if in doubt, just throw "AI" at it.

  2. Tubz Silver badge

    I have a couple of sites I use TOR to access and they too seem to have issues using Cloudfare.

  3. Jusme

    Another step towards CableTV-2.0

    ...You will use an approved browser (on an approved OS)

    ...You won't tamper with the content (block ads)

    1. Anonymous Coward
      Anonymous Coward

      Re: Another step towards CableTV-2.0

      Adblockers like Ublock have been fighting back hard look at youtube attempts to ban them.

      1. tinpinion

        Re: Another step towards CableTV-2.0

        Hello Manifest v3 gutting ad-blocking technologies. Why, what coincidental timing you have!

      2. bombastic bob Silver badge
        Unhappy

        Re: Another step towards CableTV-2.0

        also things like yt-dlp for downloading videos

  4. captain veg Silver badge

    power without responsibility

    My own experience is that Cloudflare really don't give the slightest fleck of turd for whatever deleterious effect its "services" have on ordinary end users. If you're not actually a paying customer or law enforcement agency then you don't exist.

    I've noticed them blocking access to a few sites recently on the basis of (correctly) geo-locating me in Andorra.

    Absolute scumbags.

    -A.

    1. Lazlo Woodbine Silver badge

      Re: power without responsibility

      By hosting ads on your screen, you technically are a paying user...

      1. AVR Silver badge

        Re: power without responsibility

        Well, you're the product at least. The payment isn't going to Cloudflare, they're not going to care.

        1. Lazlo Woodbine Silver badge

          Re: power without responsibility

          But you are blocking payment to Cloudflare's customer. If this happens often enough, they could be made to care...

          1. andyprough

            Re: power without responsibility

            How often would be often enough to get their attention I wonder? Cloudflare must already be turning away 10% or more of legitimate traffic, maybe more than 20%. If you look at the comments here and on Hacker News, people are being turned away based on browser identification, OS identification, ad blocking, geolocation, browser version number, cookie handling, etc. It would seem like the customers must already be feeling quite a bit of pain.

            1. frostknight

              Re: power without responsibility

              I hope cloudflare does something differently to fix this problem. The web is supposed to be a free one.

              Not in favor of only mainstream web browsers... or as intelligent people call them, corporate web browsers.

              And that my friend is a bad sign if its a corporate web browser.

              That means "Data Collection web browser" in my humble opinion.

              And Those web browsers are worthless, ugly and just plain bloated.

              Hate the UI of modern web browsers nowadays. Fisher price web browsers anyone? FISHER PRICE UI is so common now. Thanks microsoft, google, apple, etc...

              Sarcasm...

              Oh and the modern web is also awful looking. And captcha and trackers everywhere. Not to mention lots of ads that track people and anti adblock idiots refusing to acknowledge the reason people block ads is due to the stuff they do behind the scenes not the ad itself!

              Cloudflare is just the latest in this mess.

              1. andyprough

                Re: power without responsibility

                >"And captcha and trackers everywhere. Not to mention lots of ads that track people and anti adblock idiots refusing to acknowledge the reason people block ads is due to the stuff they do behind the scenes not the ad itself!"

                It is a rather vicious cycle - the web visitor has to jump through capcha's and turnstiles and lower their shields and accept cookies in order to access the content on a site, but then as soon as they are in they are bombarded with ads and trackers and their personal data is scraped and sold. It's like the Pink Panther movies, where Inspector Clouseau hired a Chinese manservant named Cato to attack and assault him at random times in his own house.

    2. myhandler

      Re: power without responsibility

      I manage a medium size site and putting it through Cloudflare was the solution for stopping the multitude of DDOS bot attacks.

      It has its uses.

      1. John Klos

        Re: power without responsibility

        I simply don't believe you and think you've believed the marketing hype from Cloudflare. Let's look at what you wrote:

        "stopping the multitude of DDOS bot attacks"

        Bots are ubiquitous on the Internet. Stopping bots, though, isn't something that should be an afterthought - that is, you shouldn't need Cloudflare to do it, even though it's nice to have fewer bots actually connecting to your servers. A server that can't stand up to bots on the Internet does not deserve to be on the Internet.

        Does "DDoS" actually mean what you think it means? That's where the disconnect is. You're almost certainly not getting a proper DDoS attack. You're just getting "attacked" by lots of bots. That's not the same as a DDoS attack. Please look it up if you're still unsure.

        Please don't be an apologist for Cloudflare on a technical site by implying that your site wouldn't be online if you didn't use Cloudflare. It's disingenuous.

        1. katrinab Silver badge
          Megaphone

          Re: power without responsibility

          It doesn't work like that.

          A bot that attempts to connect to the website and fails is still using server resources. There is nothing you can do on the server to stop that.

          1. Anonymous Coward
            Anonymous Coward

            Re: power without responsibility

            True, but if your server is on the internet, and not behind a firewall, you deserve everything they can fling at you.

            1. katrinab Silver badge
              Megaphone

              Re: power without responsibility

              And if it is the firewall that is running out of resources due to blocking the traffic?

        2. Ace2 Silver badge
          Thumb Down

          Re: power without responsibility

          This is the most remarkably condescending comment I’ve ever seen on El Reg, and that’s saying something.

        3. myhandler

          Re: power without responsibility

          @John Klos - yes I don't know what I'm doing at all - the site does over $10 mill business p.a. so something is ok

  5. SW
    FAIL

    Even blocking Chrome on a Chromebook

    How's that - can't get more "corporate" that a Google browser on Google hardware - yet still blocked me this morning.

  6. ptribble

    Incompetent or evil?

    As a niche browser user (Pale Moon) on a niche OS (illumos) I get hit by this.

    The question really is whether this is deliberate censorship, or inability. If the latter, then the idea they can accurately identify traffic is called into question.

    1. I could be a dog really Silver badge

      Re: Incompetent or evil?

      I refer you to Hanlon's Razor

  7. IamAProton Bronze badge

    Not sure how superficial is the browser check

    There are many user agent spoofers around, I use chameleon all the time and rotate the browser profile every 5 minutes to always haev a different fingerprint.

    The set of profiles/user agents can be chosen in advance

    https://addons.mozilla.org/en-US/firefox/addon/chameleon-ext/

    1. Gene Cash Silver badge

      Re: Not sure how superficial is the browser check

      This doesn't work though.

      My credit union (Trumpistani for building society, I think) used Cloudflare to block my older Firefox by locking the browser up with javascript, even though I had a user agent spoofed. The browser would suddenly start using all the cores in xosview, and when I did kill -9 or kill -11 it would take 5-10 seconds for the process to actually die.

      Then of course Firefox would be "helpful" by reopening the last page and locking up again.

      This started happening about a week and a half ago.

      This is mentioned in the sixth paragraph of the article.

      1. anonymous cat herder

        Re: Not sure how superficial is the browser check

        I wonder if that is why Firefox on Android has started regularly locking up when opening a page over the last month or two. Only solution I've found is to close it and start again.

    2. wub
      FAIL

      Shazam!

      Wow! I can't believe how well this works! I have been getting 403 You don't have permission on this server, or Secure Connection Failed, even though in at least one case, I'm have an account and a credit card linked to the retailer, so the cold shoulder welcome seems somehow wrong...

      Thanks for the suggestion!

      Have one on me-->

  8. nematoad Silver badge
    Thumb Up

    I have just sorted out a similar problem when using Palemoon.

    A kind A/c posted the following :

    You can get decent user agents by running a search for "The Latest and Most Common User Agents List". Give it a try, see if it works.

    I did, and it worked for my problem.

    It might sort out this nasty little "feature not a bug as well."

  9. Blackjack Silver badge

    Browser agents can be easily faked by botnets, this mostly just hurts users.

    1. Gene Cash Silver badge

      Yes, plus a Python script doesn't have whatever javascript exploit they're using to lock up older browsers.

      I wrote the script to download my credit union statement, because they broke Firefox.

      Which is ironic, using a bot script to get around the bot script blocking.

  10. Doctor Syntax Silver badge

    "Aside from reporting it on Cloudflare's forum, there appears to be little users can do"

    There is. They can desert sites that have this affliction.

    1. Gene Cash Silver badge

      Not really.

      Most of the banks and credit unions here in Trumpistani use Cloudflare.

      And of course they're useless when you report this as a problem. They just say "use the latest firefox" and assume you are hacking. The same thing happened when I questioned why they were setting .RU and .CN tracking cookies.

      I have reported it to the NCUA and the FDIC but they don't give a shit either.

      Now I have certainly stopped using a couple of vendors that use captcha. One of them was so retarded they didn't know captchas popped up. They were like "all you have to do is click 'I am human'" and I had to send them a screenshot.

      Edit: I did run into the CEO of one of the vendors at Bike Week. That was an interesting conversation. We'll see if anything actually changes.

    2. moonchild

      Unfortunately that isn't always an option. CloudFlare covers many thousands of sites, and I also don't think people will, e.g. abandon a local community site. CloudFlare has grown to be dangerously big in terms of traffic coverage and they really don't seem to have the right attitude for that kind of scale.

      The problem is that webmasters feel locked-in because of the increased "bad traffic" on the net, which is what should be combated at the provider level, so they don't want to stop using CloudFlare, and at the same time users of the sites don't want to abandon the sites because of a bad middleman. And quite often the two aren't directly talking about administration of the sites (because users generally have no clue about web administration).

      So CF has carved out a "safe space" for itself: Only respond to paying webmasters, don't have contact options for non-enterprise webmasters or users or other people who aren't customers of their service... and chasing more enterprise level "features" to sell off to their whales instead of doing something relatively simple like bot detection right.

    3. jasonbrown1965

      Ah yes, the purist approach, so useful in real life.

  11. Tron Silver badge

    I don't believe in coincidences.

    I suspect this is censorship. The big browsers can block downloading extensions, ad blockers etc. Alt browsers might not.

    Early versions of Opera used to dodge this by offering the option of reporting themselves as IE. Unfortunately, browsers have been getting shorter customisation option lists.

    The internet will become more restrictive with every passing week from now on courtesy of governments and big tech. The Empire is striking back. We can't do a lot about it other than hate them for ruining our net experience.

    1. Anonymous Coward
      Anonymous Coward

      Re: I don't believe in coincidences.

      The internet will never become more restrictive with every passing week because many are fighting to stop that. There many things we can do about it.

    2. logicalextreme

      Re: I don't believe in coincidences.

      You can switch user-agent string easily in Vivaldi, though not with that nice little switcher on the main window like they had in the Gecko versions of Opera (last time I checked, anyway). I'm not sure whether you can set it to report differently for different sites without an extension, though.

  12. Philo T Farnsworth Silver badge

    Firefox

    I've been seeing "are you a robot" challenges from Cloudflare for a few weeks using Firefox, though that's probably because I've got ad blockers, anti-trackers, and NoScript plugins.

    I also have been seeing intermittent blocking by Google News, alleging my IP address is engaging in "unusual" activity, which it is not.

    As of this morning, after switching over to Waterfox as my default browser, Google News has been CAPTCHA challenging me every time I click on a link, even though the browser's UserAgent string is pretty vanilla:

    Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0

    1. find users who cut cat tail

      Re: Firefox

      I am also seeing Cloudflare nonsense in normal Firefox – which I have as failsafe for sites insisting on a standard browser. Funny, a good portion of the sites work just fine in links and w3m.

      Can't anything about Google as I've not visited any their site since Search started hard-requiring JavaScript.

  13. Mage Silver badge
    Flame

    It's crazy

    Cloudfare are arrogant and stupid

    Also sites using them and using also captcha etc AFTER you logged in.

    1. ecofeco Silver badge

      Re: It's crazy

      They always have been.

      I cringe every time I have to unblock cloud flarte to view a web site. It is unnecessary and absurd and is counter to the very nature and purpose of the Internet.

  14. Korev Silver badge

    Screenreader and other assistive technologies

    Does this affect Screenreaders etc? If so then it's probably against the law in a number of countries under disability equality legislation.

    1. Anonymous Coward
      Anonymous Coward

      Re: Screenreader and other assistive technologies

      Screen reader user here. Mostly, it doesn't affect us any differently than others. Web browsers typically don't announce that a screen reader is in use. This was suggested a few years back, but I and others pushed back as hard as we could because this would become an excellent fingerprinting technique as users of that software can't opt not to use it in order to avoid sending the privacy signal. Having to try to remove the signal later would have been problematic. So mostly, CloudFlare has no way of knowing that a user uses one and therefore doesn't discriminate on that basis.

      There is one exception to this, however. CloudFlare uses, but did not make, the hCaptcha system. This captcha has an innovative way of dealing with visually impaired users, where innovative is not a compliment. Where other captchas have audio captchas, hCaptcha has decided that anyone who can't use the visual one should just get a tracking ID which can be submitted whenever they want to go through a captcha, although they've occasionally had accessible versions which they turn on and off at random. A sighted person can use the captcha and still have privacy, but a visually impaired one may not. That means that, while CloudFlare is probably not harassing blind people any more than they are sighted ones, the blind users have to give up privacy in order to get through whereas the sighted people just have the inconvenience of getting spammed with captchas.

      1. Korev Silver badge
        Thumb Up

        Re: Screenreader and other assistive technologies

        Thanks for the clarification AC

  15. milet

    This is a known problem, caused by "Disable Autoplay" extension. Enable autoplay on sites with CloudFlare protection and voila, problem solved.

    Of course, it would be even better if CloudFlare DID something about it...

    1. thosrtanner

      Where is this "Known" exactly? (And IIRC Disable Autoplay is a chrome extension, and not applicable to palemoon for instance)

  16. logicalextreme

    Cloudflare are certainly getting dafter recently

    As for general user-agent fuckery, Vivaldi gave up five years ago.

  17. John Klos

    Cloudflare know what they're doing

    We, at least readers of El Reg, aren't going to be fooled in to thinking that Cloudflare is this wonderfully clever company that can do all the things they're doing yet are somehow clueless and inept when it comes to this issue. They know exactly what they're doing.

    They keep going down the same path with no signs of changing course. For instance, they don't want people to report abuse, so they stopped processing abuse complaints sent to abuse at cloudflare.com and send an auto-reply that says to use their web interface. Their abuse reporting web page has gotten worse and worse, and will likely continue to degrade. Do they really not have the technical acumen to fix things? Really?

    For instance, the fields on the abuse reporting page only allow a certain number of characters. If you paste too many, you're stopped from adding more, but you can't submit the form and you're not told why. You have to know to remove a 100 characters or so from the abuse evidence field, as if that's supposed to be obvious. We wouldn't want to overload their poor servers!

    They've added a CAPTCHA to the abuse site. Apparently poor Cloudflare's web server that handles abuse complaints is just too fragile to work without it? Or do they simply not want to hear from people they decide are undesirables?

    They added a time limit which is greater than the amount of time a reasonable human can copy and paste a second abuse report. There's no reason in the world to do this aside from wanting to make reporting abuse as arduous as possible.

    Their abuse staff are either playing stupid or are actually incompetent. Some phishing sites show an error when you use certain browsers, but not others. But try to tell Cloudflare this in response to their "we see no evidence", and they just keep replying with the same form response.

    Are they REALLY this incompetent? Or are they evil, and want to recentralize the Internet around them, and want to protect the scammers they host? You decide.

    1. Kevin McMurtrie Silver badge
      Mushroom

      Re: Cloudflare know what they're doing

      Cloudflare serves the most sophisticated phishing operations I see on the Internet. It's not just some a fake web page. It's a very large number of interconnected sites - large enough to fool search engines. There's also the tooling, the accurate cloning of interactive content, click-through tracking, multi-level obfuscation, and high volume domain registration. The gang hacks web services on Amazon, Microsoft, and Google to send spams through existing business accounts. I know a victim and this gang even has high quality counterfeit products delivered to reduce suspicions.

      They've been on Cloudflare for probably over 10 years.

      I could infer why Cloudflare might be helping organized crime when they sell protection from organized crime, but The Register is a Cloudflare customer. My post would be deleted.

  18. StickWithIt

    Cloudflare should be ashamed of themselves. If they are not capable or willing to distinguish lesser known but perfectly valid browsers, they should NOT be allowed anywhere near the business of controlling Internet access.

    It certainly seems they are not offering real security, but bullying their way to keep promoting their pricey "security" plans.

    User agent tricks do not work in this case, mind you. Cloudflare simply allows a few browsers with deep-level checks, hidden from everyone's eyes.

  19. IGotOut Silver badge

    Can I just say...

    ...as a Cloudflare user, this is the website owners choice.

    In Cloudflare, you as the owner of the site, get to choose your level of threat protection, even on the basic mode.

    For example, if you try to get to my backend, outside of the UK, you're going to get blocked.

    Same for accessing the front end from many countries (I only sell to the UK, so really don't care if China can't scrape the content).

    However for legitimate traffic, there is multiple options on bot / malicious traffic detection, from js detection, captcha, blocking known bots etc etc.

    My guess is a lot of these sites are enforcing strict blocking, after all, how many here can't get to the reg?

    1. Anonymous Coward
      Anonymous Coward

      Re: Can I just say...

      The issue becomes when the most problematic feature of their platform is also the most aggresively marketed. All it takes is a single checkbox/slide in CF's dashboard to start locking out legitimate users just because of the browser they choose to use. I'm pretty sure if you have this active on your site, you by now have lost some of that UK market by intended sales not going through. Most customers won't have a strong reason to choose your shop over another, and if it throws up a barrier they are likely going to go elsewhere. So that "protection" is actually costing you more than you might realise.

  20. xyz Silver badge

    It's been like that for ages...

    IIRC opera mini (in pretty strict mode) used to get bounced by the cloudfare doorman.

  21. susanm

    It surely isn't that hard?

    I get it that there are many, many websites made by people who can't be bothered to test their work more widely than on the big name browsers. But Cloudflare surely have a responsibility to ensure that their technology makes an effort to support more than just a handful of browser types? I am going to guess that they are a successful business, and so they should be able to direct a tiny proportion of their profits to testing and fixing whatever their latest incarnation of checking for a human is, so real browsers like Pale Moon (and many others) can get past their gate keeping!

  22. sitta_europea

    If I see a page with "cloudflare" anywhere on it I know I need to close the tab.

  23. Johnny Footwrinkle

    Cloudfares behaviour is shameful, I use palemoon as my everyday browser and am becoming increasing frustrated and extremely disapointed at the attitude and behaviours of these companies that espouse to have 'our best interests' at heart. I believe that the end user should have complete autonomy to decide for themselves which software they are going to use on their own machine.

    The internet was hailed as great achievement by humanity, it was going to make our lives easier, make us more connected and unite us all, the reality is the interent is nothing more that a humungous data harvester used to extract revenue from the end users by selling every single concievable piece of data possible about them in order to saturate their lives with vapid and meaningless advertising and marketting drivel so that the corporations can continue to line their own pockets.

    In order to achieve this, 'gatekeepers' like cloudfare have taken the notion of 'protecting your security' and completey warped and disfigured it until it's nothing more than a means for them to carrol us into using their 'approved' browsers, essentially banishing the freedom to choose from our internet experience and leaving us vunerable to the data havesting and privacy invading behaviour of the marketting and advertising gimps. These people have a mental health condition and are obessed. They are OCD about advertising and generating revenue - this is a sickness!

    Cloudfare's attempts to force us into using browsers they aprove us is illegal, and shows just how fascist the internet actually is. Some of us don't want our data being taken, it's not like we see a single penny from the sale and resale of it all is it??? We are exploited daily and expected to quietly lap it up. Some us like our privacy and the freedom to choose for ourselves, this is our right, and we shall not have our rights eroded by a corporate monolith that employs lazy coders and tech staff who refuse to make allowances for the diversity of browsers available whilst literally shoving racial and social diversity down our throats at every oppurtunity possible, who refuse to adjust their code, employees who deliberately and mailicously target the indie browsers with their flippant attitude towards the number of users not being as high as the browsers they approve of .....here's a newsflash - Those browsers only have such high user figures because organisations like CF are forcing everyone to use them. People don't understand that companies like google are setting the web compatibility standards so that their browsers run faster and appear to give a 'better performance', people like cloudfare are aidding and abetting in this illegal attemp to monopolise the internet and tech.

    Cloudfare's response to this latest situation and the lack of channels for non-corporate users to communicate with them is just more evidence of the insidious nature of this organisation. The silence speaks volumes and the fact they only do anything AFTER articles like this one on The Register show they care more about their own image than they do about the actual service they provide or the users experience, but of course, we're all conditioned to this now, it's par for the course, we're the plebs expected to blindly jump through hoops and do nothing but generate wealth for a small minority who are mentally unwell and so obsessed with generating wealth to the point that their compassion and ability to function like decent human beings has been completely eroded. They display a complete lack of self awareness and are steeped in an outdated mental attitude rooted in bigotry and the immature and childish need to control other's behaviours. Cloudfare need to grow up and start actling like mature adults, and the whole tech sector needs to stop trying to punish and abuse us simply because we won't use one of their crappy pieces of data harvesting bloatware, y'all need to grow the hell up, it's 2025!!

    1. Daniel Pfeiffer

      Why should you care about some man in the middle? Your interlocutor is the provider of the web-site you wanted to reach. It's ultimately their fault, so yell at them!

  24. GSmith7

    CloudFlare is Big Brother

    Cloudflare's intentional denial of service to non-establishment browsers is The System saying "Comply". The "or else" is implied. This form of censoring the internet is unacceptable.

    1. Graham Perrin

      All the little babies

      Won't somebody please think of the children?

  25. Anonymous Coward
    Anonymous Coward

    It's unfortunate that services that once provided value inevitably become too big and become evil.

  26. Graham Perrin

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like