Scientists claim >99 percent identification rate of ChatGPT content Academics have apparently trained a machine learning algorithm to detect scientific papers generated by ChatGPT and claim the software has over 99 percent accuracy. Generative AI models have dramatically improved at mimicking human writing over a short period of time, making it difficult for people to tell whether text was …
Katyanna, Hi,
A full reading of your post should surely have the headline and sub-text reading .....
Scientists claim <99 percent identification rate of ChatGPT contentBoffins and machines write very differently – and it's not easy to tell
Here's proof that Mars is cold. Though that burn would fit right in with the molten lead and sulfuric acid ambiance of Venus.
This papers claim is hilariously preposterous, as much bigger players have wasted much more resources on much bigger data sets and watched as the wheels fell off of their hit rate.
That sample size is a joke for a big data problem. The fools claiming GPT-anything will solve any problem you have money to spend on are the same kind of fool. An automated tool trying to detect fake text isn't any more clever than the one writing it, and without actual comprehension, reliable fake detection isn't possible.
Much like super spam, a bland and neutrally worded text that does not make any exaggerated claims will pass any test, human or machine, that does not reproduce the claims and results. This is a case where neither attacker nor defender will be able to provide high consistency with a naive LLM based approach. You may not need a "general purpose AI" either, but certainly something considerably more advanced then we have now.
Cool, i was wondering if this would be possible for detecting AI generated text.
My university research was on automated detection of plagiarism, using a fairly simple Bayesian classifier to pick out stylistic features of authors and then check a new text supposedly from that author. People really do have detectable stylistic features in how they write, this is often how the papers like 'Computers say Shakespeare probably didn't write King Lear' come about. I trained my classifier on novels from a particular author, then picked a chunk from another novel not used in the training, and inserted paragraphs from a completely different author. it could pick out the inserted passages pretty well.
What i was surprised by was it was also pretty accurate when it hadn't seen the authors work before, if you trained it on the single text and asked it for the outliers, it could find them.
So, based on that, i was wondering if chatGPT actually had a discernable style. If the classifier couldn't find anything useful to build the model with, then you might think it wouldn't work, but it always managed to find something when trained on human text. You could also see the model it was using and what features it had picked for that author, so if it found nothing, it might not be a human.
It would be interesting to see what it did on ai generated text where a human had edited it... One of the main ways to cheat in uni was apparently taking something from the internet, writing a new introduction and conclusion and slightly editing the text. This is pretty easy for the classifier to pick out, but also... lecturers said it stuck out like a sore thumb anyway :D (particularly on non native english speakers attempts to cheat.)
A few years ago (okay probably ten or more...) research by my undergraduate tutor was covered by El Reg.
I didn't even notice the name. But it had a quote from him, and I heard his distinctive voice in my head as I read it. And I had to stop, and think "why am I using such a strange voice? who's voice is it?" Until it finally dawned on me who I was hearing, and then looked back at the name and confirmed it.
We all have styles, and many of us can pick it up. And if we can, computers can. (But equally, we know we can all be fooled by a dedicated forger. And so, I imagine, can computers.)
Nailing one round of testing with arbitrary, small, and limited inputs isn't indicative of a system that will work in the real world. As soon as you build a filter, the generative systems will add it to their screening and adaptively tune their output till they pass. You don't even need to retrain an existing model to do this. So being able to catch a small sample of un-tuned output is easy at first, then fails rapidly as the output of the GAN adapts.
Everyone that is making this claim(high probability detection) is walking it back after the inevitable blowback once the claims go public. The spam filtering world has been working these problems for literal decades. If you think you found an easy solution to this problem, it's more likely that you failed to fully understand the scope of the problem. Bayesian classifiers won't cut it here, they didn't cut it before the current crop of LLMS showed up, or any of a dozen other problem and language domains. You also have to then contend with false positives and language bias, where the cost to those that submit work that is falsely classified as plagiarized or machine generated can be quite severe, and they likely will lack the skills and context to contest the accuracy of the automated tool.
if chatGPT actually had a discernable style
Unidirectional-transformer LLMs tend to end up in one of many, many "style basins" in parameter space based on the gradient traced by the context window. Remaining in that basin while generating text in that session helps verisimilitude, since human authors tend to a degree of stylistic consistency (as you saw in your own work, and has been observed all over the place basically since various cultures started developing their versions of rhetoric and literary analysis).
What the study described in the article shows is that populating the context window with a prompt for a particular type of scientific paper tends, with high probability, to land in a style basin that's detectably distinct from the style conventions that dominate in that genre. That's probably due to a combination of the imprecision of the model (ChatGPT has a lot of parameters, but obviously it's still very lossy; modern models have a lot more, so should have somewhat better precision) and the relatively small population of training texts from this specialized genre.
If you ask ChatGPT to write, say, Harry Potter fanfic,1 you'd probably get much better adherence to genre conventions.
1And I am not for a minute suggesting you do so, though I am reminded of Rowell's fine novel Fangirl now that I bring up the subject.
Apart from the caveats raised by the authors of the study, there is of course the issue of how ChatGPT and its ilk will develop, and we may be locked in a perpetual arms race between educators wanting to test writing skills of students, and increasingly sophisticated tools for students to hide their lack of said skills. A similar arms race may develop between editors of journals and authors on production and assessment of original work. Educators at least have the last resort of the written exam, in controlled conditions, but as editor of a journal it is going to be very hard to detect AI-generated text from original writing of the authors whose name is on the front page of a paper. Style changes would not necessarily mean much in multi-authored papers, as different people may be the main author of different parts.
AI tools are certainly not going to go away. We need to learn to live with them
We live in interesting times
Yes, I get it - use AI to spot AI generated material. Sort of eating your own dog food.
But, as we are talking about AI, the answer is to use the AI detection tool to reveal the reasons why the analysis decided it was AI generated. Then feedback the problems and correct them.
This is almost certainly happening already along with "tools" to make AI generated text styles similar to the "creators" using yet more AI to create a writing style template to use.
The cat is out of the bag so there is no easy way back now.
The publishers of the journals could detect it by simply phoning the institutions to check that the named researcher actually wrote it. Much easier than an arms race of computerised language analysis trying to detect computerised language generation.
So, do I get a phD for my proposal for a foolproof way of defeating the AI?
> check that the named researcher actually wrote it.
I'm not sure how that would work in practice. Consider a case where a "professor X" submits a work for publication.
The journal's editor phones the professor and asks. They say "yes, I wrote it". How much credibility does that add?
But we must also consider two other issues.
First, most published work has many authors. Often with the most prominently named authors being the most senior and frequently the least involved. Would we expect a single writing style from several contributors?
Second is the issue of false positives. A paper that a person did actually write getting flagged as AI. How should that be dealt with.
Finally, I'd say that the onus should be on the establishment the work was performed at to police their own staff. So that their reputation would suffer if their staff were found to be submitting work that was not as it appeared.
None of this should be surprising. Computers have been used in textual analysis since the days of punched cards. They were quickly into the "Who really wrote Shakespeare?" game - I remember reading about it in the New Scientist when I was at school and that's a very long time ago. With more and more CPU cycles and more and more storage available one would expect them to get better.
Just consider the last sentence of my first paragraph. I could have written it in passive voice - "it would be expected that...". Having chosen the active voice I could have chosen a different pronoun: "I", "we" or "you". I didn't have to be as emphatic with "more and more". I could have used "might" or "could" rather than "would", "anticipate" rather then "expect". I could have written "improve" rather than "get better" etc. Without trying too hard about alternatives I can think of almost 200 ways I could have written that one sentence*. If you (note different pronoun) were to look over my writing it would (switch to passive voice) be possible to build up a list of probabilities for my choices. Repeating that for different authors would produce different choice profiles and hence different voices.
On the other hand an automated pastiche generator producing text with no intrinsic meaning, no variation of emphasis to convey, no concept of elegant variation and no instinct for deliberate repetition for emphasis is going to deploy the relatively few phrases that come at the top of its statistical heap from the training material. In terms of the multivariate statistics I dabbled with (and dabbled is a very deliberate choice of word) about 50 years ago it will occupy a very confined part of the multi-dimensional space such statistics define and that, I think, is why this paper (and Turnitin referred to in the article) are claiming such high discrimination. It has one voice . A bot writes like a bot and, with maybe one exception, humans don't even want to do that.
* I didn't write the sentence intending to analyse it like that. I wrote it without any changes although I had changed words in the previous two. It was only looking at it after I'd written it that I realised the possibilities it held for taking the rest of the post along the route I did although the direction was intended. That's something a bot couldn't do.
My daughter at Uni uses ChatGPT to help understand things that are not clear in lectures, and finds things often /appearing/ to be explained better. The interesting consequence is that she feels her writing style is more like ChatGPTs than it used to be. We are what we eat. When we count on artifical reality for input, our natural outputs become indistinguishable from artificially generated ones.
If you look online, this isn’t how academics are using ChatGPT though (and they really are). It’s horses for courses.
Apparently it’s good to write a review article. ChatGPT is terrible for finding the papers and info, because it hallucinates. But as an author in the field, you *do know* the papers that are going to go into it, your job is to summarise and make a coherent end-to-end story. So, you get ChatGPT to ingest the relevant papers as a prompt, and output an essay article structure, identifying the key themes and waypoints. Then you get ChatGPT to summarise each paper, neatly into that structure; it successively generates several alternative paragraphs and you paste in the variants you like. It’s apparently possible for a PI to reduce a weeks work of writing 20-40 pages, to a morning. That’s a good win.
I have travel from one English speaking country, region or state etc. and the Spoken English varies. The written word is no different., look at the English dictionaries and or the age related dialects. I have visited in number of English speaking countries over the past x number of years and have found that regional languages/dialects and the written word often vary from one suburb to another let alone the spelling in different Islands,County,States,Metropolitan,rural, districts. So machine written texts would surly have the programmers syntax or style in their output?
So machine written texts would surly have the programmers syntax or style in their output?
Not really. Written (scientific and technical) English is much more uniform that spoken English. You never see "Howay man" or "Howdy y'all" in a paper.
Anyway, the language output isn't produced by an individual programmer anyway. The same is true when writing papers. I had a Chinese boss once, and his papers went through half a dozen people (with 3 or 4 native languages) before being published, to the point where there was no Chinglish in the output.
a Chinese room problem.
That literally is the problem with the bot output. It doesn't understand language at all, only sets of approximate rules and searches based upon that rather nebulous incomprehension.
It was accurately, if fancifully outlined in Watts novel Blindsight.
A growing number of university students are using ChatGPT to produce there university assessments thus doing no research etc. however citations give it away as how can you write a citatation into a chatGPT written document. sim,ple decide what citation you want and pose the correct question to the bot. answers then written to back up your argument.
Award first class honours degree or MA without writing or researching any thing.
This will help combat that approach being touted by university students.
It worries me a lot that the academics can't do this themselves. Scientific papers are meant to make sense and present novel ideas. So the need to use a statistical analyser to identify fake papers says more about the state of science than it does about ChatGPT.
Maybe...
The waters are becoming so muddied, that the only real way to find out what students know is to revert to one-to-one discussions - something that's been missing for a long time. Also, I can remember group discussions on a given theme where misconceptions quickly surfaced and could be corrected. That never seems to happen any more either.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
