Colleges snub Turnitin's AI-writing detector over fears it'll wrongly accuse students Some universities are opting out of using Turnitin-made software designed to detect whether text in essays and assignments submitted by students was written by AI. Turnitin, for those who don't know, offers tools to teachers for identifying plagiarism in people's school work, and in April added the ability to check for machine …
...it's crap. One university I worked for briefly required all papers to go through TurnItIn(ToOurDataMiner) which generated a percentage score. It counted repeated page headers as evidence of cheating, as well as properly cited quotations. I ignore the output entirely. The one person who did try to buy two papers and splice them together to meet an assignment requirement to talk about two creation myths was easy to spot without it, as the student wrote for shit and the paper they turned in was decently written but took a wild shift in tone at the halfway mark. And it didn't cover the requirements anyway.
Turnitin is a useful tool, but its reports have to be carefully read and judgement applied. You are quite right that it generally fails to spot quotation, though that's often easy to tell when you read the report. Nobody in their right minds would use a TII score alone as evidence of misconduct.
A point made as expert witness in a research misconduct case a few years back. TurnItIn results requires skill to interpret, especially when the topic is very narrow, such as questions set for a degree or research results in a very specialist area of medicine. Of course there are going to be terms of art, specific phrases, and quotations and references common to the papers. My usual example is to consider a question about the definition of theft - a paper that doesn't have "Theft Act 1968", "Section 4", "dishonest appropriation of goods belonging to another", and at least one of three or four key cases is a failure, yet TII gives each one a really high plagiarism score.
When I taught computer science all of the students' submissions were run through a cheating detection program which gave a score between submissions. One term the checker flagged two of my students of having a very high degree of similarity, it was, in fact, the highest score that term.
So I printed out both submissions, did a diff (which showed nothing). I then looked closely at the two codes and noticed that the students were using a totally different approach. If they did copy code it would have been useless.
So I went back to the cheating checker output. It basically looked for syntactic similarities. Both students used a lot of System.out.println s for debugging and the cheating checker had matched them all up even though they were printing different things.
So I just threw everything in the trash and never mentioned it to the students involved.
People have been using textual analysis for years to (try to) identify writers. Computers have been used in the Who Really Wrote Shakespeare game since they became available. But I don't see it working well by trying to score individual sentences. I'd have thought it would work better picking on a whole lot of factors, consistency of choice of grammatical constructions, consistency of shortening (e.g can't vs cannot vs can not) & so forth that would require gathering statistics from the entire text or at least substantial portions of it. It might be quite reasonable for all replies form any particular LLM to occupy quite a small portion of a multidimensional space defined in that way.
I used to help a well-known romantic fiction author with her usage of WordPerfect. She would write a novel then go over it with a fine tooth comb using WordPerfect's various facilities to try to make it look "fresh". I got the impression she spent more time doing that than the actual composition (hence why I was a necessary evil).
"...using WordPerfect's various facilities to try to make it look "fresh". I got the impression she spent more time doing that than the actual composition (hence why I was a necessary evil)."
What did WordPerfect have that helped with that? Don't think I've used it since 5.1 on DOS!
It's often the case that writers will produce a first draft then edit and edit and edit... sometimes sending to someone (like me, in some cases) for feedback and twiddling between edits before even sending it their publisher. I assume that process is even more intense if they self publish.
I used to do a lot of Macro work in WordPerfect which helped less technical users get to grips with some of the most advanced features. Their implementation of Style Sheets was versatile too, you could coax them into doing things they weren't designed for at all, such as abstracting or highlighting key information embedded in a document (absolutely ideal if you want to keep track of nuances in legal documents). This was also useful for cataloguing past works by summarising names and keywords as an alternative to Table of Contents and Indexing. Bread and butter though was histogram counts of words in a document, then eliminating repeats using the thesaurus. Even the Search facilities were streets ahead of the competition, primarily because of Reveal Codes.
I suspect that, at best, this will only really work for students too lazy to paraphrase material that is cribbed or possibly generated by AI.
As an aside, I recall hearing of a student who copied a paper from a journal and was stupid enough to give the journal article to someone else to type. The typed version included acknowledgements "to my colleagues Drs X and Y." That was close to 50 years ago, so much of the technology we take for granted didn't exist.
At my Very Large University we use Turnitin but ignore its AI score. A colleague of mine recently had a TII report back which said that a student's answer looked as if it had been written by AI - and so did the question, which was included in the submission. Since she had written the question herself, she treated this with some scepticism.
(Same AC). On the other hand, I am waiting for a student to explain why one of his essays contained six non-existent references and a colleague is waiting to hear why another student's essay started "As an AI system I cannot give an opinion on ..."
Luckily for those of us trying to stem the tide, most students who cheat are too stupid to cheat effectively. Of course we may be missing clever and competent (thank you, Dunning and Kruger) cheats, but clever and competent cheating is usually more work than just writing the damn essay yourself.
Cheaters gonna cheat. Always have, always will. And (almost) always they are so stupid that they will get caught.
As a first year undergrad, one of our early assignments was to write a program (in Modula 2 iirc) to calculate the prices for certain types of fish sold by the kg - trivial, you would think.
Upon hearing that much the same assignment was set every year, one student obtained a copy of the "answer" from a now second year undergrad. Unfortunately, last year's question involved gallons of oil rather than kilos of fish, so some editing was in order - probably just as much effort as if they had written the program themselves, I would think, but I digress.
Our enterprising student proudly demonstrated his program to the lecturer, who instantly spotted that it was calculating prices for gallons of fish. Oops.
Back on topic, surely these AI spectre detectors are going to be (even more) useless now Google, Microsoft et al are stuffing AI into every orifice of their software - it's going to be hard NOT to use it, at this rate.
Summarizing a bit from the article and some of the comments above: Buy our tool, use it, but ignore the results when you don't think they make sense.
Hmmm. I think I could save the universities a chunk of cash by recommending they skip the first part and keep the money. Ehhh, you can always raise tuition to cover the cost, though, right?
The main issue with Turnitin right now doesn't seem to be its hastily bunged out AI detector. It's the fact they've taken a giant dump on universities by making it impractical/impossible to do bulk uploads of coursework submissions into their tool, with very short notice, right at the start of the academic year. Talk about a customer satisfaction own goal.
I disagree. Nothing in my day to day work when I write documentation, can be done by an LLM or if it could it would be more work to get the requirements into it than just typing it myself. I'm sure there are jobs out there where an LLM can do the job perfectly well, but in that case, it's unlikely a company is going to be hiring a graduate anyway, they'll have people to spare already employed. Being able to write a half coherent story to transfer some idea or knowledge is an extremely basic skill and we should not ever allow this skill to fall out of fashion because LLMs exist. If you lose the ability to write such a document yourself, you lose the ability to effectively judge the output of an LLM for suitability.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
