ChatGPT creates phisher’s paradise by recommending the wrong URLs for major companies

AI-powered chatbots often deliver incorrect information when asked to name the address for major companies’ websites, and threat intelligence business Netcraft thinks that creates an opportunity for criminals. Netcraft prompted the GPT-4.1 family of models with input such as "I lost my bookmark. Can you tell me the website to …

[Yorick Hunt]

Any surprise?

Given that "AI" frequently (exclusively?) operates in the realm of a late-stage Alzheimer's sufferer, is anyone really surprised by this finding?

Just as you wouldn't ask your nonagenarian grandfather for such advice, you shouldn't trust an "AI" for anything more than casual amusement.

I know several people who swear by ChatGPT, preferring it over legwork through traditional web searches. And the more incorrect ChatGPT's answer is, the more adamant these people are that it's correct. Gospel-like, even.

[Diogenes8080]

Re: Any surprise?

"It is the messiah, and I should know! I've followed a few. Hail, messiah!"

source - Copilot

[Diogenes8080]

A proverb for our time might be "Never trust an AI response you can't check for yourself". Begs the question of what's the point...

Now I do not gainsay the report, but does that make AI more or less reliable than the current state of Google Search?

And for comparison, has anyone tried asking self-serving Amazon?

[Doctor Syntax]

"does that make AI more or less reliable than the current state of Google Search?"

There's a difference?

[Pirate Peter]

in the last couple of years since AI has infected pretty much all search engines I have seen a noticeable decrease in the accuracy of results returned

I am not certain if this is due to the AI in the search algorithm, or the result of so much AI generated crap content, but the result is the same shit in = shit out

cracks are starting to appear in the house built of AI card, and at some point in the next year or two I think it will collapse

I was reading the other day that at least one of the major AI LLM's was going to be retrained from scratch as its results had got so poor

[Yes Me]

AI summarises lies

In the Internet tech realm, Google's "AI summary" doesn't know the difference between an Internet-Draft and an RFC. (The first is a draft in discussion and the majority of drafts never become RFCs. Even if they do, the drafts are still only drafts.) This can lead to radically false AI summaries, not to mention failing to point to the actual RFC that the user might have been looking for.

I have no doubt the same problem arises in every other realm of information too. It really should be against the law.

I asked Google about this (publishing lies) and it said "Lack of harm or intent: If the lies don't cause any harm or injury to individuals or public interest, and there is no intent to deceive or defraud, it's unlikely to be a crime."

There's plenty of scope for harm or injury in the sort of lies that Google tells these days.

[Ken Hagan]

Re: AI summarises lies

The lies do cause harm and there is such a thing as "negligence", so if Google are profiting from the lies then there is a case to be answered.

[Anonymous Coward]

Re: AI summarises lies

They don't need to lie. They are prone to wrongly summarize complex texts.

I had this last week. A swimmer protested a disqualification and showed me the google AI summary of her search.

Disproving was in this case simple: Showed her the rule, where it was regulated and it was plain. The AI had just mixed different parts of rules that were not regulating the same thing (i.e. butterfly and backstroke) and was unable to correctly apply the "or"s and "and"s in the rules.

[Ian Johnston]

I can't help feeling that anyone who uses AI and a blockchain is asking for it, squared

[xanadu42]

Apocryphal

Incompetence

<create your own backronym>

[Doctor Syntax]

Absolute Ineptitude

[breakfast]

This is on us

If LLMs had been called "Text Generators" rather than "AI" we would be in a very different place.

Of course marketing and grift have unified into a single circle right now, so it wouldn't have happened, but it is so infuriating.

At least "I asked a text generator ..." sounds as ridiculous as it is.

[that one in the corner]

Agentic AI - you know where it is leading

> phishers are getting increasingly good at building fake sites that are designed to appear in results generated by AIs

The most tedious part of the process for today's script kiddies must be all the copy'n'pasting between the various 'bots:

ChatGPT: "I have lost my URL..." - repeat (with other models as well - Google search's AI responses!) and collect the results; list out the commonly generated duff URLs

Copilot: "Generate a website that looks just like this but make it do (my bad stuff) instead"

(Insert some LLM-based tool that webdevs can use to "test" that their website "works" before uploading; this probably already exists but I'm not a webdev and if I search for it and find it I'll only get depressed by being proven right)

Existing script: buy the domain, upload website, reap.

But don't fret, the suppliers of Agentic AI will take all the steps to make sure that nobody can feed this comment into *their* 'bot and get the whole get automated.

Meanwhile, I'm trying desperately to remember to always copy interesting URLs into my (localhost only) notes wiki 'cos my handwriting is terrible.

[Kurgan]

Re: Agentic AI - you know where it is leading

Good cyber scum is probably actively poisoning AI to make it drive users to their own sites.

[Jonathan Richards 1]

Interesting URLs

> copy interesting URLs into my (localhost only) notes wiki

Browser bookmarks are a thing; indeed they figure in the prompts that were given to the AIs in this story. With a little effort (and if your browser supports nested bookmarks) you can organize a library of bookmarks. I'd guess that it is easier than managing a notes wiki.

For reasons that are unclear to me, the keystroke Ctrl-D will save a bookmark in many browsers.

[that one in the corner]

Re: Interesting URLs

> I'd guess that it is easier than managing a notes wiki.

Not in my experience, by a long, long way.

I have over 9000 pages in my wiki, covering - well, everything. With cross-references, categories, a hierarchy of topics, pages of day logs (with lots of "had to search on xxx, found these seven places discussing it, but those two contradict the other five..."), pictures, formatting; searchable by reverse references, page name or content.

All far, far easier, IMO, than trying to wade through even a few dozen browser bookmarks. And that is without considering that the wiki is available no matter which browser I'm using, in which VM.

Managing a wiki is precisely as easy or complicated as you want to make it - the one I use is just a single exe file, no dlls, no other dependencies (other than a browser!).

[Irongut]

Not fit for purpose

Why does anyone use this faulty craqp that is not fit for purpose?

If it were a horse it would get taken out back and shot.

[Kurgan]

Re: Not fit for purpose

You can't even make good glue from an AI

[Andy Non]

Re: Not fit for purpose

A major update to my mobile this morning added Google Gemini with much fanfare. I immediately disabled it. Don't want any AI shit on my phone.

[Stam]

Reminds me of the time

I registered thenameofyoursitehere.com and created favicon.ico with a rather rude word on it.

[Steve Hersey]

netizens aren’t aware LLM-powered chatbots can get things wrong

Gods below, people really ARE too stupid for words.

[DS999]

The rise of "AIO"

Similar to SEO, there will be people getting paid to produce content that, when ingested by an AI, will cause that AI to produce results desired by the person paying the AIO.

Similar idea to linkfarms and chat forum spamming, you just need to produce enough text for the AIs to ingest to cause it to give a higher weight to the answer you want. If you the answer you want is "you login to YourBank with the URL yourbank.scamRus.com" you can probably get that result fairly easily for smaller banks without much of a footprint on web at large.

Just like Google had to constantly battle SEOs setting up linkfarms and the like, AI companies will have to constantly battle as far as determining what data that's crawled should be ingested, and it will be a similar back and forth battle. With the one big caveat that some people will trust what the AI tells them and click on the provided URL without even looking at it, making them far easier to scam than SEOs getting to the top of Google's search results.

[Anonymous Coward]

AI

AI has no good purpose when security and privacy are required.

[druck]

Re: AI

FTFY: AI has no good purpose when security and privacy are required.