Update turns Google Gemini into a prude, breaking apps for trauma survivors Google's latest update to its Gemini family of large language models appears to have broken the controls for configuring safety settings, breaking applications that require lowered guardrails, such as apps providing solace for sexual assault victims. Jack Darcy, a software developer and security researcher based in Brisbane, …
"Google just cut it all off. They just pushed a model update that's cut off its willingness to talk about any of this kind of work despite it having an explicit settings panel to enable this and a warning system to allow it. And now it's affecting other users whose apps relied on it, and now it won't even chat [about] mental health support."
How many times do we have to say this? It's somebody else's computer that you don't control.
There's a story here about social workers and agencies not being funded enough to have a real human helping victims. If LLMs continued to be used, it would be worthwhile to have a "stable branch" for these apps. Perhaps there room for a non-profit to step in and host LLMs specifically for these use cases.
This is also a story about a developer who put their faith in Google not arbitrarily shutting down products and services that people rely on.
I now aim to share an anecdote that amuses me: Google once decided that people would want to stream video games instead of running them locally, marketed the technology for months, won exclusivity contracts with game developers, built and shipped physical game console products, and then ate the entire cost of the endeavor just so they could add Stadia to killedbygoogle.com.
This is what happens when there are minimal regulations and you leave it up to the sales people to decide what product is for you. Leaving this in the hands of maniacal ads purveyors doesn’t bode well.
Or does it? I’m probably missing something obvious here…
VISUALIZATION: Egg frying sunny side up
EXPLANATORY TEXT: "THIS IS YOUR AI ON DRUGS"
=============================
We're going to entrust much of our our culture's future to this obviously flawed "technology"? May not be one of our better ideas.
"Why are (presumed) sexual abuse counselling services using LLMs in the first place"
A machine is clearly indicated to ... no it isn't: its utter madness even considering using one for this purpose.
One day some firm of Civils is going to throw a LLM at a bridge design which will get built and it will be fine. Many more will get built and yea, bridges will be automatically designed. Then the models will be trusted and running the whole thing through a proper analysis will be dropped. Then something like the Millennium Bridge will be created but it wont just wobble with a spot of resonance and be fixed with some hydraulic ram damping.
It will be Tacoma Narrows. Again.
Why are (presumed) sexual abuse counselling services using LLMs in the first place, given LLMs' propensity to generate unhelpful shit?
I was thinking this too. Trauma needs empathetic counselling by actual skilled humans and equally skilled and sensitive interviewers to collect a coherent statement.
Collecting the unguided and uncritical outpouring of suffering and grief from a sexual assault victim by a machine interface and have a LLM concoct a "structured" report destined for forensic use would seem to be a catastrophe in the making.
If I knew how to access AI/LLM one simple experiment I might undertake would be to take an otherwise innocuous stream of consciousness narrative†, chop it up into small but coherent pieces, randomly remove a piece or two, possibly adding an extraneous fragment and randomly reordering the lot before submitting it to the LLM to produce a structured text and compare that text with the original. How much was missing or conflated or inexplicably inserted would be interesting before even addressing questions of meaning or misrepresentation.
† I assume ChatGPT et al. have trawled Gutenberg, libgen etc and would recognise a published work.
It sounds to me like they aren't using it as a substitute counselor, but to help them get their story out in the first place. Some people who have trauma find it very hard to even begin talking about it, and talking to a machine instead of person might be less intimidating to them. Then when they talk to a counselor there's a huge head start because they have access to the victim's story as related to the AI.
Whether the process of telling the story to an AI helps the victim in the same way as it might to tell a human I'm not qualified to say, but if Google is cutting off the ability for that to work without any notice I can see where it would not only be disruptive to the process of learning whether/how well this works, but possibly cause further trauma to the victims if the time they spent talking to the AI so far turns out to have been completely wasted.
Hi!
I have actually interacted with HELIX during a demonstration at the Mater hospital.
From what we were told, and shown, it appears to keep all the logs and records in a local DB on-device, then push that+prompt+schema reqs to the Gemini corpo endpoint (which doesn't keep copies according to google's privacy policy) and the helix guys dont keep a copy
Hmmm ... Is this the Jack Darcy security researcher in Brisbane (with associated github), and is this the related Helix (through Australia's Medical Director)? Asking because neither of those places mentions "sexual assault survivors, rape victims, and so on" that I could see ...
Either ways, much as plenty of folks find it comforting to pen down their experiences in a personal private diary (eg. Anne Frank, famously), I can imagine that interacting with a private LLM can indeed provide a lower barrier to expressing oneself for some trauma victims. Irrespective of Google's Gemini terms however, I think this software should be run locally, which provides for much better securing of PII, and better versioning control (if done right).
It depends how well the various local models work, but there are a bunch of them and it generally just takes a computer with enough RAM to fit them. You may need to tailor the software to use one of those instead of Gemini, and those models might have different performance or quality. My problem is that the largest models from Google or OpenAI seem to have enough quality problems that I'm not confident putting them unsupervised into almost anything, so I don't know whether the best local models are equally good or noticeably worse.
This looks like a noble goal, I think their mistake was in using a remote LLM provider like Gemini (which looks fairly unreliable and frankly dangerous).
While I understand that autistic folks, introverts etc. can get a lot of use out of systems like this, especially in a legal context, Google and Microsoft are too unstable, shifty, and shady to touch, and this is a shining beacon of that lesson.
Isn't gemini 2.5 pro tagged as experimental/preview model, meant only for testing? I am curious why would anyone deploy it to actual users, let alone in such sensitive domain, (other than for testing purposes)? It's also a bit frustrating to see the media focus on sensationalism rather than in-depth reporting.
If you are using "AI" with any vaguely sensitive personal information then (assuming not self hosting the LLM so do not control it) then you would have a proper contract with hwatever LLM provider you decide to use.
You would need to ensure that data was not stored by the LLM provider, not used in LLM training etc i.e. basic confidentiality / never mind sensitive data safeguarding
This would all be in a contract as you would need to prove to potential customers that privacy & data security was paramount.
Given LLMs often have various censorship regimens, if the data to be processed was in any way likely to contain material likely to be "lost" due to censorship then your contract negotiations would also have made sure that your data would not be subject to any censorship.
You would definitely not be using an unstable preview / experimental release and (it would appear) had no agreement with the LLM provider on data privacy / censorship before implementing it in a product
you sell!
Ah... that's annoying, I hope that the victims who are effected find another way to organise what happened into a legal document. It can be so hard to talk about that stuff, and having something like that happen can be really discouraging when trying to go through legal processes. Legal stuff in itself can be retraumatising, so even just deciding that you want to go through the legal process in the first place can be a really difficult decision.
Yeah, like other people have said, why were they relying on outsourced experimental models? It would make more sense to use the current main model. It would be even better if they could create/train their own model for this since TOS for companies can change at any time.
I guess it feels easier to outsource models, but literally the only way to ensure that other internal company changes don't effect you is to train/host your own model. Very much easier said than done especially by a small team. It requires time, money and lots of effort, which can be hard when life in general is VERY unpredictable. Even just gathering the resources required for that could be really hard.
Ideally, you would want a team of people extremely knowledgeable in both law and LLM's to help you with the project. Those are two niches that are highly sought after by many big companies, and paying someone with those expirences a fair wage would be really hard as a startup.
The legal processes for rape victims can be such a fucking nightmare, I totally understand why someone would want a LLM to help them with it. I can also totally understand how talking to a robot could feel safer than a person if you've had horrible experiences with people.
I can understand why Google would not be okay with their LLM's giving legal advice on sensitive topics, it's a very niche use case that could land them in legal trouble depending on how it's used. But I definitely think that wanting to host a LLM to help rape victims is a wholesome endeavour that could be extremely helpful to people who have had horrible experiences.
This is the typical cloud computing platform, it isn't your computer, it isn't your software and it will be updated/replaced as the hoster sees fit.
For something this sensitive, surely running your own models with direct control is the only way to go? If there are changes to the model that is being used to people in very vulnerable situations, surely you need to thoroughly test changes to the model before letting it lose on real users?
If you are using a service that swaps out models willy-nilly, you are using the wrong service, if it needs to be relible.
If we can’t make money off you, we don’t give a fuck about you. MS will break stuff by accident, because they’re incompetent. Google will break stuff because it doesn’t pad the bottom line. And, yes, if you use a Google service that’s ‘free-to-you’, either it will be gone soon or Google is making money from it in other ways. There is no third choice.
Interesting note: I have DirectTV Streaming; certain people insisted that they couldn’t live without their tv fix, and comments about ‘a vast wasteland’ were met with hostile glares. (There was no way that I was getting a dish, given previous bad experience.) So there are four tvs attached to DirectTV Streaming, inbound using AT&T Fibre, nominally 1 Gb/s, not that we ever get even close to that. DirectTV Streaming uses Android devices with a lot of Google stuff, including voice control. Three of the devices have voice control turned off, one has it turned on. The device with voice control on is the only original device still operational; all the others have died and been replaced at least once, one of them twice. The one which was replaced twice is also the one on which almost all Google ‘optional features’ are turned off. Hmm. Co-incidence? Or Google handing out hints along the lines of ‘nice tv service you have there, it’d be a pity if there was a service interupption, now wouldn’t it?’ So far DirectTV is replacing devices for free under the warranty. (You have to use the serial number on the device to get a replacement, so it is clear which device failed.) We’ll see what happens later. I use the DirectTV app on my iPad, or log into the DirectTV site in a web browser on a computer, not a tv, on the rare occasions that I want to watch tv. No Google stuff in sight because I use Firefox, not Chrome… Google hates me. I don’t care.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
