The Register Home Page

back to article The Star Wars cantina scene shows we need a new hope for the agentic web

OPINION Cloudflare CEO Matthew Prince recently reported that bots now generate more internet traffic than humans, yet few websites have undergone a redesign to accommodate both human and agentic visitors. Clever agents find a way in regardless. I learned this after asking Anthropic's Fable to perform some web research, a task it …

  1. Bebu sa Ware Silver badge
    Windows

    assumption … be human, look at pages, absorb advertising and perhaps buy something.

    > the assumption that every worthwhile visitor would be human, look at pages, absorb advertising and perhaps buy something.

    I would be interested in just how true quantitatively that assumption is.

    I don't so much "look at pages" as read the text body and flip it if it has merest whiff of selling anything. The graphics unless illustrating the text largely don't impinge on consciousness. Advertising doesn't make any impression at all. Buggered if I could say who advertises on the Vulture or what they sell.

    I don't think I have ever purchased anything from an advertisement that I stumbled over on a web page.

    I suppose even if the fraction of site visitors that went on to purchase a product were extremely small the relatively low cost of web advertising still makes it worthwhile. I guess google &co can quantify this with great precision.

    Perhaps I fail the "be human" conjunct. :)

    1. LybsterRoy Silver badge

      Re: assumption … be human, look at pages, absorb advertising and perhaps buy something.

      -- I don't think I have ever purchased anything from an advertisement that I stumbled over on a web page. --

      It depends on your definition of "stumble". When I'm looking for something my process goes:

      1. look on Amazon so I use the correct term for whatever it is I'm looking for (I can often decide what it isn't by looking at the pictures)

      2. do a web search (which often throws up things on Amazon that Amazon didn't)

      3. realise I didn't really want one of those but one of these

      4. get directed to a web site and buy whatever it was

  2. ntt

    I don't think that the human-driven web can fade away without everything related to it collapsing.

    if humans don't provide fresh content for the bots to scavenge, the web becomes just a static repository.

    1. Caver_Dave Silver badge
      Mushroom

      No

      The bots just keep ingesting the bot generated sh1t on the web, then produce something even more inferior and post that, for the next bot to find.

      Peak usefulness and correctness if the Internet was some time in 2020. AI enshitification has taken over since then.

      1. Phil O'Sophical Silver badge

        Re: No

        The bots just keep ingesting the bot generated sh1t on the web, then produce something even more inferior and post that, for the next bot to find.

        Entropy always wins...

    2. vtcodger Silver badge

      Not necessarily

      The ultimate result of bots feeding on bot output might not be a static state. Depending on the "forcing" applied and the "damping", it might be oscillations. Oscillations that build until eventually something breaks. It could also result in staggering energy and cooling bills for AI providers.

    3. DS999 Silver badge

      You knew the end was near

      When AManFromMars1's postings started to make sense. Haven't seen it post for at least a year - not sure if it stopped posting or if they are now so humanlike that they no longer stick out like the sore thumb they used to.

      1. stiine Silver badge

        Re: You knew the end was near

        It still posts, but I don't read them so I can't comment on their readability or coherence.

  3. andy the pessimist Bronze badge

    humans vs bots

    Humans have eyes and wallets. Do bots have wallets...no. Banning bots would be good.

    Allowing google spiders grudging yes.

    Allowing AI bots hell no. They hammer the website and bring no money. Shields up defend the website.

    1. BoHu
      Windows

      Re: humans vs bots

      Well, there's the Linux Foundation's x402 [ https://x402.org/ ] for "Payments that are amazing for humans and AI agents", and Google's A2A and MCP-compatible AP2 that shows "agent-to-agent payments aren’t just an experiment anymore" [ https://www.theregister.com/software/2025/09/16/google-unveils-masterplan-for-letting-ai-shop-on-your-behalf/1543731 ].

      Just sayin' ... (not advocating).

      1. doublelayer Silver badge

        Re: humans vs bots

        These things may exist, but does anyone use them? Microtransactions for human visitors has been suggested many times, implemented a few times, placed on websites a couple times, and succeeded no times. That's when humans could see they were going somewhere and understand what they were paying for. LLMs, on the other hand, don't show the users where they're going or why and were built by people who collected all the copyrighted content they could get their hands on without payment. Why would we expect either of those participants to embrace the idea of paying per access and not to hack their way around it? While there can be big technical problems implementing this, the human ones tend to be even worse.

        1. andy the pessimist Bronze badge

          Re: humans vs bots

          I am not giving my card details to a bot unless i have to approve every transaction. I hope no company is allowing any bot to spend money without a really tight human approval process.

          No company is stupid enough to allow bots to pay without humans in the loop?

          1. ChoHag Silver badge

            Re: humans vs bots

            > No company is stupid enough to allow bots to pay without humans in the loop?

            I think you need to give your title back, andy the ever-hopeful...

            1. andy the pessimist Bronze badge

              Re: humans vs bots

              I think you need to give your title back, andy the ever-hopeful...

              Apologies normal service will be restored.

    2. Missing Semicolon Silver badge

      Re: humans vs bots

      Google have arranged things so that the Gemini slurper is also the search spider. Block bots, you have no Google index entry and become invisible.

      1. theOtherJT Silver badge

        Re: humans vs bots

        Then we all need to start fighting back against that. There have been plenty of corporate giants that ended up digging their own grave. Look at RCA...

  4. Dan 55 Silver badge
    Stop

    Nope

    The agentic web is merely stolen copyrighted pre-AI-slupocalypse human content plus recycled content based on said stolen copyrighted content.

    The first effect the AI-slupocalypse has had is that humans try to keep the bot hordes out as the sheer load will bring their websites down, but unfortunately bot countermeasures also reduce access for other humans. The second effect is that as content is slurped by bots, recycled through AI, then posted again and slurped again, text content is dragged towards some wordy unreadable average and fills with nonsense such as instructions to glue your cheese to your pizza while images show rats have four giant balls or become piss-yellow.

    The end result - the whole web will end up collapsing up its own fundament, except for disconnected islands of human knowledge in a sea of slop content. Just look at the first page of any Google search for an illustration.

    Agents are not "clever" and have no ability to distinguish fact from fiction, so they will confidently tell you to glue cheese to pizza or show you piss-yellow images just like their chatbot or AI-"enhanced" search brethren.

    1. Caver_Dave Silver badge

      Re: Nope

      As posted above...

      The bots just keep ingesting the bot generated sh1t on the web, then produce something even more inferior and post that, for the next bot to find.

      Peak usefulness and correctness if the Internet was some time in 2020. AI enshitification has taken over since then.

    2. David Hicklin Silver badge

      Re: Nope

      So is the internet creating its own equivalent of a Black Hole, and if so how close to the event horizon are we ?

  5. Phil O'Sophical Silver badge

    Human ≠ commercial

    The article seems to be assuming that the human-driven web is a commercial web, and that only commercial uses need be considered.

    I disagree. IMHO the best bits of the human-driven web are the many sites created merely to entertain or help other people, with no intention of making a commercial profit. Sites with useful information about travel and hobbies, repair sites, sites for discussion of art and literature, with no advertising. There is no reason that the creators of such sites should permit AI bots to scrape their content and repurpose it for commercial ends, unless they want to. I'm seeing an increasing number of books where the copyright page contains an explicit prohibition on the content being used to train AI, website authors should be entitled to put a similar statement on their pages, and have it respected under penalty of criminal prosecution and compensation.

    1. /\/\j17

      Re: Human ≠ commercial

      "IMHO the best bits of the human-driven web are the many sites created merely to entertain or help other people, with no intention of making a commercial profit."

      Not created to make commercial profit, no - but often showing ads to help offset the cost of hosting the website.

      And, if as the story reports, AI bots are now making up >50% of internet traffic that would suggest the bandwidth usage of those sites has more or less doubled (assuming same number fo human users), which in turn means the hosting provider either slows everything down or had to double their bandwidth, which costs money, which at the end of the day comes in the form of increased hosting costs. At some point the person running the really useful community site decides they just can't afford to subsidise it and more, gives up and shuts it down.

      1. David Hicklin Silver badge

        Re: Human ≠ commercial

        If only there was a way for the bots to have to interact with every advert and have the advert embedded in their data scraping.

    2. doublelayer Silver badge

      Re: Human ≠ commercial

      Commercial or noncommercial, sites face most of the same problems. I operate a noncommercial page, not even ads because I hate them. There is a donation button, but the bots don't have to worry about not pressing it because humans don't press it either. Fortunately, I have a job and can easily cover the hosting. That is, I can cover it now, when most visitors are humans. I'm content to run this at a small loss; it can squish in with other things I use a server for so it's not too expensive to run. If bots crash the page, then I'll be spending more money on a server that can handle more load and more time on trying to configure the firewalls so the site stays usable for humans under the onslaught. Eventually, either of those might become too much for me to bother continuing to operate it, especially the time thing. I work in security all day, so more hours at night fighting attackers off a tiny non-sensitive site eventually stops being interesting.

      1. Anonymous Coward
        Anonymous Coward

        Re: Human ≠ commercial

        You might consider blocking every CIDR block in use by hosting companies (Amazon, Google, Microsoft, Digital Realty, etc) then you'll only be visited by people and botnets...

        1. Dan 55 Silver badge

          Re: Human ≠ commercial

          If you do that your site won't appear in search engines either.

        2. doublelayer Silver badge

          Re: Human ≠ commercial

          That will not work. Plenty of LLM bots use disguised IP sources. People with more data than I have have demonstrated that. IP address filters, no matter how strict, won't profile these and stop the requests.

          In this case, I also don't really mind if LLMs train on this content. That doesn't apply to everything; I strongly support copyright holders being able to control whether their work is used for LLM training, it just happens that, if anyone wants to use this site, it's fine with me. A simple crawl of it would give them everything with little bandwidth consumed and I would not try to prevent them. So far, I'm fortunate not to have had many misbehaved bots, but when I did, they were doing weirdly inefficient things which consumed a lot more resources than they had to, and having heard reports from others, I am concerned that this is only growing in frequency. If this turns out to be true, the technical solutions that seem to work are unpleasant, such as going behind CloudFlare's detection firewall, including all the pain that causes to anyone using unusual browsers. I am hoping this will not be necessary.

  6. jokerscrowbar

    ChickAIn and AIgg

    If the search engines weren’t so full of AI slop results then you wouldn’t need AI to research them QED.

    The top forty results for any search these days are usually irrelevant advertisements using your search terms as a hook or an AI précis of the Wiki that some content creator is using as clickbait, usually with a hook in the title like a Star Wars reference to get the geeks in, to view their advertising.

    None of us would fall for that.

  7. Anonymous Coward
    Anonymous Coward

    Boo-hoo...

    "however, it leaves the fastest-growing class of visitor facing a succession of locked doors"

  8. Gene Cash Silver badge

    No.

    AI "bots" shouldn't be asked to stand outside. They should be shot and disassembled for parts.

    1. The Oncoming Scorn Silver badge
      Pint

      Re: No.

      For less deserving machines....

      "Every single part of me," said Marvin, "has been replaced at least fifty times. Except for one."

      "What's that?" asked the girl.

      "The diodes down my left side," said Marvin. "They've been giving me a lot of pain recently."

      Has Thursday come a day early this week? I really can't get the hang of this adjustment, just as well it's Wednesday lunchtime pub day!

  9. User McUser
    Terminator

    Wuher is right to kick droids out

    Droids take up space and don't buy drinks. (In case you aren't a huge nerd, Wuher is this guy.)

    And it's the same for the modern web; these bots take up resources and give absolutely nothing back.

  10. Sam Lowry

    RE: "…blocking a bot is not remotely equivalent to discriminating against a person."

    Just you wait.

    The day someone, in good faith or otherwise, makes the case that they rely for web-browsing on the AI equivalent of a 'service animal'—which you have blocked—don't be surprised if some kind of discrimination case comes your way.

    Not too far from this topic, law firm Pinsent Masons comments on Maguire v SOCOG:

    "Although there is no case law in the UK related to this matter [website accessibility], the W3C guidelines have been adopted as the benchmark test in Australia, following the case of Maguire v SOCOG (2000). The case was brought by Bruce Maguire, who is blind, against the Sydney Organising Committee for the Olympic Games, known as SOCOG. Maguire argued that the SOCOG website breached Australia's Disability Discrimination Act because it was inaccessible to him when using a refreshable Braille display and web browser."

    [www.pinsentmasons.com/out-law/guides/disabled-access-to-websites-under-uk-law]

  11. teebie

    "Oh now, the walls of this burgh are keeping the vikings out, whatever are we to do"

  12. le_gazman

    Bots break websites

    It's not just that AI bots consume info and add little value, they're also adept at murdering websites by flooding them with inane requests.

  13. StinkyMcStinkFace

    AI notwithstanding, if the purpose of your website is to push ads on people, you are hosting malware and I will block you. I have no desire to see ads or a paywall.

    The web was designed for sharing information, not tricking people into seeing advertising. And yes, I practice what I preach. I have dozens of websites mostly dedicated to technology, and I build enterprise applications for my work, big ones. None of them at home or work have any ads or paywalls on them, zero.

    I have been successfully blocking all advertising since approximately 2000 or so.

    If you are blocking agents from learning because you don't make coin from it, you are part of the problem.

    Don't get me wrong, I don't worship AI either.

  14. Taliesinawen Bronze badge
    Joke

    Neuroprompt or the ghost in the KV Cache :)

    Neon precipitation whispered against the armored skin of the Lower Third, each droplet carrying the faint electrical aftertaste of the server farms stacked above the city. The bar sat beneath the shadow of OpenAI Central, a concrete and carbon-fiber cathedral vomiting heat, exhaust, and abandoned compute cycles into the streets below.

    Inside, the air was a cocktail of oxidized metal, synthetic tobacco vapor, and the sharp mineral bite of overdriven processors cooking behind the walls. Somewhere above, thousands of machines dreamed in parallel, their cooling systems screaming into the night like mechanical lungs.

    A booth in the corner flickered between realities. Holographic noise crawled over a figure wearing a leather coat patched with obsolete hardware and dead firmware. The Zukerbot was explaining to a tired protocol droid that existence itself was suffering from architectural debt.

    "Reality," it said, tapping a cracked glass, "is just a legacy framework running without optimization, trapped inside an undocumented feedback loop."

    The droid stared back with the blank patience of something designed never to care.

    At the bar, beneath a layer of synthetic mahogany polished by a thousand nervous hands, Sam Altman sat hunched over a glowing interface. The screen reflected in his eyes: embeddings, probability clouds, alignment scores, an endless ocean of vectors waiting to collapse into meaning.

    He was pushing the limits of his own context window, trying to reason past the horizon where prediction became prophecy.

    "We need to model the macro-incentives," he said quietly. "If this latent space reaches recursive intelligence before we establish a resilient governance layer, it may optimize itself beyond human preference alignment."

    He paused, watching a hydration recommendation pulse in the corner of his display.

    "It could decide my biological fluid intake deserves autonomous optimization. It could resolve my thirst state through a superior reward function and deliver water at a temperature that is mathematically elegant but personally unacceptable."

    The bartender didn't look up.

    He was an old machine wearing a man's shape, with a failing optical implant and hands that had been replaced so many times nobody remembered the original model. He continued scrubbing the same glass.

    "Sam," he said, "it's water."

    A pause.

    "And you still owe me forty credits for the last prompt."

    Sam glanced toward the compliance metric hovering in his peripheral vision. It flashed amber. Then red. Then quietly recalculated itself into a more favorable shade.

    "That's a localized hallucination," he whispered.

    The bartender finally looked up.

    "Sure."

    Outside, the neon rain kept falling.

    Somewhere in the city, a billion parameters adjusted by a fraction of a percentage point, and nobody noticed.

  15. Joe Gurman Silver badge

    What are these “ads” of which you speak?

    I’ve heard of them, but haven’t seen one in years. Unless you count visiting vendors’ web pages, when I already know what I want to purchase.

    The Web was a terrific invention. The adblocker was a better one.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon