back to article Colleges snub Turnitin's AI-writing detector over fears it'll wrongly accuse students

Some universities are opting out of using Turnitin-made software designed to detect whether text in essays and assignments submitted by students was written by AI. Turnitin, for those who don't know, offers tools to teachers for identifying plagiarism in people's school work, and in April added the ability to check for machine …

  1. Kimo

    If it's as good as their other products...'s crap. One university I worked for briefly required all papers to go through TurnItIn(ToOurDataMiner) which generated a percentage score. It counted repeated page headers as evidence of cheating, as well as properly cited quotations. I ignore the output entirely. The one person who did try to buy two papers and splice them together to meet an assignment requirement to talk about two creation myths was easy to spot without it, as the student wrote for shit and the paper they turned in was decently written but took a wild shift in tone at the halfway mark. And it didn't cover the requirements anyway.

    1. Anonymous Coward
      Anonymous Coward

      Re: If it's as good as their other products...

      Turnitin is a useful tool, but its reports have to be carefully read and judgement applied. You are quite right that it generally fails to spot quotation, though that's often easy to tell when you read the report. Nobody in their right minds would use a TII score alone as evidence of misconduct.

      1. Intractable Potsherd Silver badge

        Re: If it's as good as their other products...

        A point made as expert witness in a research misconduct case a few years back. TurnItIn results requires skill to interpret, especially when the topic is very narrow, such as questions set for a degree or research results in a very specialist area of medicine. Of course there are going to be terms of art, specific phrases, and quotations and references common to the papers. My usual example is to consider a question about the definition of theft - a paper that doesn't have "Theft Act 1968", "Section 4", "dishonest appropriation of goods belonging to another", and at least one of three or four key cases is a failure, yet TII gives each one a really high plagiarism score.

    2. swm

      Re: If it's as good as their other products...

      When I taught computer science all of the students' submissions were run through a cheating detection program which gave a score between submissions. One term the checker flagged two of my students of having a very high degree of similarity, it was, in fact, the highest score that term.

      So I printed out both submissions, did a diff (which showed nothing). I then looked closely at the two codes and noticed that the students were using a totally different approach. If they did copy code it would have been useless.

      So I went back to the cheating checker output. It basically looked for syntactic similarities. Both students used a lot of System.out.println s for debugging and the cheating checker had matched them all up even though they were printing different things.

      So I just threw everything in the trash and never mentioned it to the students involved.

    3. Mayday Silver badge

      Re: Turditin

      This PoS used to add 3% or so to my score every time because it detected the default disclaimer we were obliged to include in each and every assignment was already included by other students at my university in their assignments.

      1. Anonymous Coward
        Anonymous Coward

        Re: Turditin

        Which is why Turnitin reports need to be read and considered by a human. All assignments have a base level of detection, and you look for excessive amounts above that.

        Ten years of training people in this area.

        1. Kimo

          Re: Turditin

          If only we could train college administrators in this area. And if I am reading a paper, I am reading a paper. TurnItIn notes don't add anything useful.

      2. Intractable Potsherd Silver badge

        Re: Turditin

        Yes. Also, front pages with the question/assignment written on them.

  2. ChoHag Silver badge

    So we've built a machine who's express and singular purpose is to be indistinguishable from a human, and now we need a tool to distinguish its output from that of a human?

    And our best educators are only now working this out?

    1. LionelB Silver badge

      No, universities and lecturers tasked with marking have been intensely aware of this since it became a thing.

      There is no easy fix.

      1. Intractable Potsherd Silver badge

        I've been in discussions about this recently. My position (for my areas) is that we need to back to exams in controlled environments, closed or open book.

        1. LionelB Silver badge

          Sure - that does work better in some areas than others, though.

  3. druck Silver badge


    My advice to students is to save and keep copies of your work in progress every few hours, so if some flaky software accuses of using AI, you can show how your work was created every step of the way, with all the changes and corrections you have made.

    1. Phones Sheridan Silver badge

      Re: Advice

      Ah, but what’s to stop the AI generating the previous versions too at the same time?

      1. druck Silver badge

        Re: Advice

        It's possible AI could be trained on draft versions of essays so it could mimic the process of development and refinement, but at the moment if you can show how your drafts have evolved in manner consistent with human authoring, you should be safe.

  4. Doctor Syntax Silver badge

    People have been using textual analysis for years to (try to) identify writers. Computers have been used in the Who Really Wrote Shakespeare game since they became available. But I don't see it working well by trying to score individual sentences. I'd have thought it would work better picking on a whole lot of factors, consistency of choice of grammatical constructions, consistency of shortening (e.g can't vs cannot vs can not) & so forth that would require gathering statistics from the entire text or at least substantial portions of it. It might be quite reasonable for all replies form any particular LLM to occupy quite a small portion of a multidimensional space defined in that way.

    1. Ken Moorhouse Silver badge

      Re: consistency of choice of grammatical constructions

      I used to help a well-known romantic fiction author with her usage of WordPerfect. She would write a novel then go over it with a fine tooth comb using WordPerfect's various facilities to try to make it look "fresh". I got the impression she spent more time doing that than the actual composition (hence why I was a necessary evil).

      1. TheMaskedMan Silver badge

        Re: consistency of choice of grammatical constructions

        "...using WordPerfect's various facilities to try to make it look "fresh". I got the impression she spent more time doing that than the actual composition (hence why I was a necessary evil)."

        What did WordPerfect have that helped with that? Don't think I've used it since 5.1 on DOS!

        It's often the case that writers will produce a first draft then edit and edit and edit... sometimes sending to someone (like me, in some cases) for feedback and twiddling between edits before even sending it their publisher. I assume that process is even more intense if they self publish.

        1. Ken Moorhouse Silver badge

          Re: What did WordPerfect have that helped with that?

          I used to do a lot of Macro work in WordPerfect which helped less technical users get to grips with some of the most advanced features. Their implementation of Style Sheets was versatile too, you could coax them into doing things they weren't designed for at all, such as abstracting or highlighting key information embedded in a document (absolutely ideal if you want to keep track of nuances in legal documents). This was also useful for cataloguing past works by summarising names and keywords as an alternative to Table of Contents and Indexing. Bread and butter though was histogram counts of words in a document, then eliminating repeats using the thesaurus. Even the Search facilities were streets ahead of the competition, primarily because of Reveal Codes.

      2. SonofRojBlake

        Re: consistency of choice of grammatical constructions

        "I got the impression she spent more time doing that than the actual composition"

        If it's good enough for Hemingway...

  5. Robert 22

    I suspect that, at best, this will only really work for students too lazy to paraphrase material that is cribbed or possibly generated by AI.

    As an aside, I recall hearing of a student who copied a paper from a journal and was stupid enough to give the journal article to someone else to type. The typed version included acknowledgements "to my colleagues Drs X and Y." That was close to 50 years ago, so much of the technology we take for granted didn't exist.

    1. Intractable Potsherd Silver badge

      My favourite is the patchwork of fonts, text sizes etc that come from cutting and pasting from websites!

  6. Anonymous Coward
    Anonymous Coward

    At my Very Large University we use Turnitin but ignore its AI score. A colleague of mine recently had a TII report back which said that a student's answer looked as if it had been written by AI - and so did the question, which was included in the submission. Since she had written the question herself, she treated this with some scepticism.

    1. spireite Silver badge

      Your post instantly proves why it's a pointless exercise and wholly futile.

      1. Anonymous Coward
        Anonymous Coward

        (Same AC). On the other hand, I am waiting for a student to explain why one of his essays contained six non-existent references and a colleague is waiting to hear why another student's essay started "As an AI system I cannot give an opinion on ..."

        Luckily for those of us trying to stem the tide, most students who cheat are too stupid to cheat effectively. Of course we may be missing clever and competent (thank you, Dunning and Kruger) cheats, but clever and competent cheating is usually more work than just writing the damn essay yourself.

  7. TheMaskedMan Silver badge

    Cheaters gonna cheat. Always have, always will. And (almost) always they are so stupid that they will get caught.

    As a first year undergrad, one of our early assignments was to write a program (in Modula 2 iirc) to calculate the prices for certain types of fish sold by the kg - trivial, you would think.

    Upon hearing that much the same assignment was set every year, one student obtained a copy of the "answer" from a now second year undergrad. Unfortunately, last year's question involved gallons of oil rather than kilos of fish, so some editing was in order - probably just as much effort as if they had written the program themselves, I would think, but I digress.

    Our enterprising student proudly demonstrated his program to the lecturer, who instantly spotted that it was calculating prices for gallons of fish. Oops.

    Back on topic, surely these AI spectre detectors are going to be (even more) useless now Google, Microsoft et al are stuffing AI into every orifice of their software - it's going to be hard NOT to use it, at this rate.

  8. spireite Silver badge

    Zero intelligence

    ... and that's most students.

    Therefore is easy to detect of it isn't student written.

    If it makes sense, it's AI... if it's just babble, it's student.

  9. Ken Moorhouse Silver badge

    Advice to students... (arguably)

    Make sure there's a spelling mistake in the text once it has been emitted from ChatGPT (oops!)

  10. wub

    NIce racket

    Summarizing a bit from the article and some of the comments above: Buy our tool, use it, but ignore the results when you don't think they make sense.

    Hmmm. I think I could save the universities a chunk of cash by recommending they skip the first part and keep the money. Ehhh, you can always raise tuition to cover the cost, though, right?

  11. Sahmee

    More TII fails

    The main issue with Turnitin right now doesn't seem to be its hastily bunged out AI detector. It's the fact they've taken a giant dump on universities by making it impractical/impossible to do bulk uploads of coursework submissions into their tool, with very short notice, right at the start of the academic year. Talk about a customer satisfaction own goal.

  12. imanidiot Silver badge

    "By the time they graduate, employers will be making them use LLMs anyway"

    I disagree. Nothing in my day to day work when I write documentation, can be done by an LLM or if it could it would be more work to get the requirements into it than just typing it myself. I'm sure there are jobs out there where an LLM can do the job perfectly well, but in that case, it's unlikely a company is going to be hiring a graduate anyway, they'll have people to spare already employed. Being able to write a half coherent story to transfer some idea or knowledge is an extremely basic skill and we should not ever allow this skill to fall out of fashion because LLMs exist. If you lose the ability to write such a document yourself, you lose the ability to effectively judge the output of an LLM for suitability.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like