back to article Scientists claim >99 percent identification rate of ChatGPT content

Academics have apparently trained a machine learning algorithm to detect scientific papers generated by ChatGPT and claim the software has over 99 percent accuracy. Generative AI models have dramatically improved at mimicking human writing over a short period of time, making it difficult for people to tell whether text was …

  1. amanfromMars 1 Silver badge

    Lost in translation does not equate to a disturbance in the force and source ‽ .

    Katyanna, Hi,

    A full reading of your post should surely have the headline and sub-text reading .....

    Scientists claim <99 percent identification rate of ChatGPT content

    Boffins and machines write very differently – and it's not easy to tell

    1. Roj Blake Silver badge

      Re: Lost in translation does not equate to a disturbance in the force and source ‽ .

      Here's the proof that generative AI has improved vastly over the last couple of months - an amanfromMars 1 post that's almost intelligible.

      1. Anonymous Coward
        Anonymous Coward

        Re: Lost in translation does not equate to a disturbance in the force and source ‽ .

        Here's proof that Mars is cold. Though that burn would fit right in with the molten lead and sulfuric acid ambiance of Venus.

        This papers claim is hilariously preposterous, as much bigger players have wasted much more resources on much bigger data sets and watched as the wheels fell off of their hit rate.

        That sample size is a joke for a big data problem. The fools claiming GPT-anything will solve any problem you have money to spend on are the same kind of fool. An automated tool trying to detect fake text isn't any more clever than the one writing it, and without actual comprehension, reliable fake detection isn't possible.

        Much like super spam, a bland and neutrally worded text that does not make any exaggerated claims will pass any test, human or machine, that does not reproduce the claims and results. This is a case where neither attacker nor defender will be able to provide high consistency with a naive LLM based approach. You may not need a "general purpose AI" either, but certainly something considerably more advanced then we have now.

        1. Mark Exclamation

          Re: Lost in translation does not equate to a disturbance in the force and source ‽ .

          That post was written by ChatGPT, and I claim my $5.

          1. Michael Wojcik Silver badge

            Re: Lost in translation does not equate to a disturbance in the force and source ‽ .

            I would have expected ChatGPT to remember the apostrophe in "papers" [sic].

    2. NeilPost Silver badge

      Re: Lost in translation does not equate to a disturbance in the force and source ‽ .

      … Boffin’s and Journo’s write very different, and it’s easy to tell.

      That’s probably a solid 100%.

  2. b0llchit Silver badge

    >99 percent identification rate

    I guess the remaining <1 percent makes up a major part of the internet (the rest is mostly porn, probably).

  3. Pete 2 Silver badge


    > prompted ChatGPT to generate paragraphs describing the same research

    I get the feeling that this 99% accuracy in spotting AI generated articles is akin to asking students to set their own exam questions and then complimenting them for getting all the answers right.

    1. steelpillow Silver badge

      Re: Circular?

      How long until we get AI-generated articles discussing the detection of AI-generated articles ...

      ... and how to avoid said detection?

      Disclaimer: I am not an AI, honest!

      1. Eclectic Man Silver badge

        Re: Circular?

        I'm waiting for the AI system that detects bias in AI systems based on incorrect assumptions, poor 'training' data sets and historical discrimination.

        Not holding my breath.

        1. dmesg

          Re: Circular?

          There may be a Halting Problem in that idea, struggling to emerge.

        2. NeilPost Silver badge

          Re: Circular?

          “Amazon recommends ChatGPT”.

          “Unexpected ChatGPT in bagging area”

    2. FrogsAndChips Silver badge

      Re: Circular?

      That would be circular if they were asking ChatGPT to detect ChatGPT-generated articles, but as I understand it they have trained their own model for the detection, and only used ChatGPT to produce the training material.

      1. steelpillow Silver badge

        Re: Circular?

        "That would be circular if they were asking ChatGPT to detect ChatGPT-generated articles"

        At least one academic has already done that, except they were exam essays not articles.

  4. Ben Bonsall

    Cool, i was wondering if this would be possible for detecting AI generated text.

    My university research was on automated detection of plagiarism, using a fairly simple Bayesian classifier to pick out stylistic features of authors and then check a new text supposedly from that author. People really do have detectable stylistic features in how they write, this is often how the papers like 'Computers say Shakespeare probably didn't write King Lear' come about. I trained my classifier on novels from a particular author, then picked a chunk from another novel not used in the training, and inserted paragraphs from a completely different author. it could pick out the inserted passages pretty well.

    What i was surprised by was it was also pretty accurate when it hadn't seen the authors work before, if you trained it on the single text and asked it for the outliers, it could find them.

    So, based on that, i was wondering if chatGPT actually had a discernable style. If the classifier couldn't find anything useful to build the model with, then you might think it wouldn't work, but it always managed to find something when trained on human text. You could also see the model it was using and what features it had picked for that author, so if it found nothing, it might not be a human.

    It would be interesting to see what it did on ai generated text where a human had edited it... One of the main ways to cheat in uni was apparently taking something from the internet, writing a new introduction and conclusion and slightly editing the text. This is pretty easy for the classifier to pick out, but also... lecturers said it stuck out like a sore thumb anyway :D (particularly on non native english speakers attempts to cheat.)

    1. Brewster's Angle Grinder Silver badge

      A few years ago (okay probably ten or more...) research by my undergraduate tutor was covered by El Reg.

      I didn't even notice the name. But it had a quote from him, and I heard his distinctive voice in my head as I read it. And I had to stop, and think "why am I using such a strange voice? who's voice is it?" Until it finally dawned on me who I was hearing, and then looked back at the name and confirmed it.

      We all have styles, and many of us can pick it up. And if we can, computers can. (But equally, we know we can all be fooled by a dedicated forger. And so, I imagine, can computers.)

    2. Anonymous Coward
      Anonymous Coward

      Don't get cocky kids

      Nailing one round of testing with arbitrary, small, and limited inputs isn't indicative of a system that will work in the real world. As soon as you build a filter, the generative systems will add it to their screening and adaptively tune their output till they pass. You don't even need to retrain an existing model to do this. So being able to catch a small sample of un-tuned output is easy at first, then fails rapidly as the output of the GAN adapts.

      Everyone that is making this claim(high probability detection) is walking it back after the inevitable blowback once the claims go public. The spam filtering world has been working these problems for literal decades. If you think you found an easy solution to this problem, it's more likely that you failed to fully understand the scope of the problem. Bayesian classifiers won't cut it here, they didn't cut it before the current crop of LLMS showed up, or any of a dozen other problem and language domains. You also have to then contend with false positives and language bias, where the cost to those that submit work that is falsely classified as plagiarized or machine generated can be quite severe, and they likely will lack the skills and context to contest the accuracy of the automated tool.

      1. Ben Bonsall

        Re: Don't get cocky kids

        Well, my research was about 25 years ago, so i would have thought they had come up with some better tools in the mean time :)

    3. Michael Wojcik Silver badge

      if chatGPT actually had a discernable style

      Unidirectional-transformer LLMs tend to end up in one of many, many "style basins" in parameter space based on the gradient traced by the context window. Remaining in that basin while generating text in that session helps verisimilitude, since human authors tend to a degree of stylistic consistency (as you saw in your own work, and has been observed all over the place basically since various cultures started developing their versions of rhetoric and literary analysis).

      What the study described in the article shows is that populating the context window with a prompt for a particular type of scientific paper tends, with high probability, to land in a style basin that's detectably distinct from the style conventions that dominate in that genre. That's probably due to a combination of the imprecision of the model (ChatGPT has a lot of parameters, but obviously it's still very lossy; modern models have a lot more, so should have somewhat better precision) and the relatively small population of training texts from this specialized genre.

      If you ask ChatGPT to write, say, Harry Potter fanfic,1 you'd probably get much better adherence to genre conventions.

      1And I am not for a minute suggesting you do so, though I am reminded of Rowell's fine novel Fangirl now that I bring up the subject.

  5. Michael H.F. Wilkinson

    Interesting stuff

    Apart from the caveats raised by the authors of the study, there is of course the issue of how ChatGPT and its ilk will develop, and we may be locked in a perpetual arms race between educators wanting to test writing skills of students, and increasingly sophisticated tools for students to hide their lack of said skills. A similar arms race may develop between editors of journals and authors on production and assessment of original work. Educators at least have the last resort of the written exam, in controlled conditions, but as editor of a journal it is going to be very hard to detect AI-generated text from original writing of the authors whose name is on the front page of a paper. Style changes would not necessarily mean much in multi-authored papers, as different people may be the main author of different parts.

    AI tools are certainly not going to go away. We need to learn to live with them

    We live in interesting times

  6. steamnut

    But for how long?

    Yes, I get it - use AI to spot AI generated material. Sort of eating your own dog food.

    But, as we are talking about AI, the answer is to use the AI detection tool to reveal the reasons why the analysis decided it was AI generated. Then feedback the problems and correct them.

    This is almost certainly happening already along with "tools" to make AI generated text styles similar to the "creators" using yet more AI to create a writing style template to use.

    The cat is out of the bag so there is no easy way back now.

    1. LionelB Silver badge

      Re: But for how long?

      > Sort of eating your own dog food ... The cat is out of the bag ..."

      Okay already, which is it?

      1. Doctor Syntax Silver badge

        Re: But for how long?

        Just bear with me while I click my mouse to find out.

  7. Howard Sway Silver badge

    researchers thought it would be useful to develop a way to detect AI-generated science writing

    The publishers of the journals could detect it by simply phoning the institutions to check that the named researcher actually wrote it. Much easier than an arms race of computerised language analysis trying to detect computerised language generation.

    So, do I get a phD for my proposal for a foolproof way of defeating the AI?

    1. Pete 2 Silver badge

      Re: researchers thought it would be useful to develop a way to detect AI-generated science writing

      > check that the named researcher actually wrote it.

      I'm not sure how that would work in practice. Consider a case where a "professor X" submits a work for publication.

      The journal's editor phones the professor and asks. They say "yes, I wrote it". How much credibility does that add?

      But we must also consider two other issues.

      First, most published work has many authors. Often with the most prominently named authors being the most senior and frequently the least involved. Would we expect a single writing style from several contributors?

      Second is the issue of false positives. A paper that a person did actually write getting flagged as AI. How should that be dealt with.

      Finally, I'd say that the onus should be on the establishment the work was performed at to police their own staff. So that their reputation would suffer if their staff were found to be submitting work that was not as it appeared.

  8. Doctor Syntax Silver badge

    None of this should be surprising. Computers have been used in textual analysis since the days of punched cards. They were quickly into the "Who really wrote Shakespeare?" game - I remember reading about it in the New Scientist when I was at school and that's a very long time ago. With more and more CPU cycles and more and more storage available one would expect them to get better.

    Just consider the last sentence of my first paragraph. I could have written it in passive voice - "it would be expected that...". Having chosen the active voice I could have chosen a different pronoun: "I", "we" or "you". I didn't have to be as emphatic with "more and more". I could have used "might" or "could" rather than "would", "anticipate" rather then "expect". I could have written "improve" rather than "get better" etc. Without trying too hard about alternatives I can think of almost 200 ways I could have written that one sentence*. If you (note different pronoun) were to look over my writing it would (switch to passive voice) be possible to build up a list of probabilities for my choices. Repeating that for different authors would produce different choice profiles and hence different voices.

    On the other hand an automated pastiche generator producing text with no intrinsic meaning, no variation of emphasis to convey, no concept of elegant variation and no instinct for deliberate repetition for emphasis is going to deploy the relatively few phrases that come at the top of its statistical heap from the training material. In terms of the multivariate statistics I dabbled with (and dabbled is a very deliberate choice of word) about 50 years ago it will occupy a very confined part of the multi-dimensional space such statistics define and that, I think, is why this paper (and Turnitin referred to in the article) are claiming such high discrimination. It has one voice . A bot writes like a bot and, with maybe one exception, humans don't even want to do that.

    * I didn't write the sentence intending to analyse it like that. I wrote it without any changes although I had changed words in the previous two. It was only looking at it after I'd written it that I realised the possibilities it held for taking the rest of the post along the route I did although the direction was intended. That's something a bot couldn't do.

  9. saif

    Life imitates AI

    My daughter at Uni uses ChatGPT to help understand things that are not clear in lectures, and finds things often /appearing/ to be explained better. The interesting consequence is that she feels her writing style is more like ChatGPTs than it used to be. We are what we eat. When we count on artifical reality for input, our natural outputs become indistinguishable from artificially generated ones.

  10. DuncanIrvine

    ChatGPT will be able to take the test itself and iterate results to avoid the tests. Then hand over a result that passes all tests.

  11. Anonymous Coward
    Anonymous Coward

    Back in the day El Reg would explain what "Two Truths and a Lie" is because I, for one, have never heard of it.

    1. FrogsAndChips Silver badge

      1. It is a game where you give 3 statements

      2. 2 of them will be true

      3. 2 of them will be lies

  12. jefffalcone

    These folks have simply trained AI to recognize their prompt engineering. I'm shocked that this got past peer review.

    The conclusions of the paper simply point to the current deficiencies in their prompt engineering.

  13. Justthefacts Silver badge

    Not the use-case….

    If you look online, this isn’t how academics are using ChatGPT though (and they really are). It’s horses for courses.

    Apparently it’s good to write a review article. ChatGPT is terrible for finding the papers and info, because it hallucinates. But as an author in the field, you *do know* the papers that are going to go into it, your job is to summarise and make a coherent end-to-end story. So, you get ChatGPT to ingest the relevant papers as a prompt, and output an essay article structure, identifying the key themes and waypoints. Then you get ChatGPT to summarise each paper, neatly into that structure; it successively generates several alternative paragraphs and you paste in the variants you like. It’s apparently possible for a PI to reduce a weeks work of writing 20-40 pages, to a morning. That’s a good win.

  14. Old Man Ted

    Regional and country of Origin

    I have travel from one English speaking country, region or state etc. and the Spoken English varies. The written word is no different., look at the English dictionaries and or the age related dialects. I have visited in number of English speaking countries over the past x number of years and have found that regional languages/dialects and the written word often vary from one suburb to another let alone the spelling in different Islands,County,States,Metropolitan,rural, districts. So machine written texts would surly have the programmers syntax or style in their output?

    1. Displacement Activity

      Re: Regional and country of Origin

      So machine written texts would surly have the programmers syntax or style in their output?

      Not really. Written (scientific and technical) English is much more uniform that spoken English. You never see "Howay man" or "Howdy y'all" in a paper.

      Anyway, the language output isn't produced by an individual programmer anyway. The same is true when writing papers. I had a Chinese boss once, and his papers went through half a dozen people (with 3 or 4 native languages) before being published, to the point where there was no Chinglish in the output.

  15. Wzrd1 Silver badge

    So, they've gotten to a 92% alleged accuracy rate in detecting

    a Chinese room problem.

    That literally is the problem with the bot output. It doesn't understand language at all, only sets of approximate rules and searches based upon that rather nebulous incomprehension.

    It was accurately, if fancifully outlined in Watts novel Blindsight.

  16. darklord

    Missing the point here

    A growing number of university students are using ChatGPT to produce there university assessments thus doing no research etc. however citations give it away as how can you write a citatation into a chatGPT written document. sim,ple decide what citation you want and pose the correct question to the bot. answers then written to back up your argument.

    Award first class honours degree or MA without writing or researching any thing.

    This will help combat that approach being touted by university students.

  17. Mike 137 Silver badge

    "trained a machine learning algorithm to detect scientific papers generated by ChatGPT "

    It worries me a lot that the academics can't do this themselves. Scientific papers are meant to make sense and present novel ideas. So the need to use a statistical analyser to identify fake papers says more about the state of science than it does about ChatGPT.

  18. Bob Whitcombe

    Isn't this the same tool that correctly identified 10 of 10 ChatGPT writings as those of AI - but erroneously identified 8 of 10 human authored works as done by AI as well? If the AI can't tell them apart and the humans can't tell them apart, seems we have passed Turning test milestone.

  19. Will Godfrey Silver badge

    Good News!


    The waters are becoming so muddied, that the only real way to find out what students know is to revert to one-to-one discussions - something that's been missing for a long time. Also, I can remember group discussions on a given theme where misconceptions quickly surfaced and could be corrected. That never seems to happen any more either.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like