Cool, i was wondering if this would be possible for detecting AI generated text.
My university research was on automated detection of plagiarism, using a fairly simple Bayesian classifier to pick out stylistic features of authors and then check a new text supposedly from that author. People really do have detectable stylistic features in how they write, this is often how the papers like 'Computers say Shakespeare probably didn't write King Lear' come about. I trained my classifier on novels from a particular author, then picked a chunk from another novel not used in the training, and inserted paragraphs from a completely different author. it could pick out the inserted passages pretty well.
What i was surprised by was it was also pretty accurate when it hadn't seen the authors work before, if you trained it on the single text and asked it for the outliers, it could find them.
So, based on that, i was wondering if chatGPT actually had a discernable style. If the classifier couldn't find anything useful to build the model with, then you might think it wouldn't work, but it always managed to find something when trained on human text. You could also see the model it was using and what features it had picked for that author, so if it found nothing, it might not be a human.
It would be interesting to see what it did on ai generated text where a human had edited it... One of the main ways to cheat in uni was apparently taking something from the internet, writing a new introduction and conclusion and slightly editing the text. This is pretty easy for the classifier to pick out, but also... lecturers said it stuck out like a sore thumb anyway :D (particularly on non native english speakers attempts to cheat.)