Using 'AI-based software like Proctorio and ProctorU' to monitor online exams is a really bad idea, says uni panel

A committee at the University of Texas in Austin has advised against using AI software to oversee students' online tests, citing the psychological toll on students and the financial toll on academic institutions. Acknowledging that some form of online proctoring is necessary to discourage academic misconduct, the committee …

[Anonymous Coward]

They never should've used it in the first place.

Any program that takes over control of your devices (EG: mic & webcam) & prevents you from using them as you see fit (EG: not at all) is by definition malware that doesn't belong anywhere near the machine.

You want to make sure they're not cheating, fine, but requiring the installation of what amounts to destroying their privacy in the name of your security is the absolute wrong way to go about it.

What message do you think that sends? What lesson will the students learn from the experience? That their privacy isn't worth two shakes of a lamb's tail, that they can't be trusted not to act like mature adults, & that they should just suck it up & deal with it like the good little sheep you want them to be? What kind of adults do you want to train? What kind of future do you want to leave for the children that will then be in charge of your retirement home? How well do you think they'll treat you once the power shoe is on the other foot?

Just don't use the software, it's not worth it & never has been.

[Anonymous Coward]

Re: They never should've used it in the first place.

Uh... really? Hyperbole much?

Every school I attended in the before times did the same sort of stuff, just without technology. Every test I ever took had at least one person in the room to watch the students and make sure we weren't cheating. That means someone spent 1-2 hours constantly watching us. Making sure we weren't looking at our neighbor's answers, searching for answers on Google, etc. I'd like to think we all turned out ok even despite that surveillance.

AFAI can tell, that's all this software attempts to do. Though, it sounds like it doesn't do a great job of it. Poor pattern recognition and threatening messages suck, but I appreciate that it doesn't just fail the students, it refers the data to a human for verification.

Of course, cheating was rarely as common as some people would like you to believe. But once in a while you got a mass outbreak that made for a good story to tell every semester. So, it sounds like this made a good deterrent, even if it didn't catch many students. Convincing the students not to cheat is a lot better than catching them in the act.

Though, if their software is doing something more than just watching the student while they take a test, then they should be called out for that (then we can burn 'em at the stake). Pretty sure we had an article a while back about some software installed on underage students' laptops that didn't necessarily stop recording outside of class. Software that makes child porn never goes over well, not for the software company and not for the school that installed it.

[A Non e-mouse]

Re: They never should've used it in the first place.

Every school I attended in the before times did the same sort of stuff, just without technology. Every test I ever took had at least one person in the room to watch the students and make sure we weren't cheating. That means someone spent 1-2 hours constantly watching us

Was there one teacher per student? Was said teacher sit right in your face staring at you constantly for the duration of the exam? Whilst school exams may be 1-2 hours, at Uni exams can be 3 or more hours. Can you imagine the psychological stress induced after being stared at constantly for that period of time?

[Anonymous Coward]

Re: They never should've used it in the first place.

There is medical help available for mental disorders like that paranoia. Please seek it out.

[Anonymous Coward]

Re: They never should've used it in the first place.

"Uphill both ways in the snow and wind."

Funny how everyone always claims they have it worse than any other generation just because they have a bigger chip on their shoulder than other generations. The "muh rights" and "me first" generations are really downgrading society severely.

What ever happened to you *responsibilities*, kids?

[katrinab]

Re: They never should've used it in the first place.

The biggest problem seems to be that it refuses to let you take the test because it can't see you because you have the "wrong" colour of skin.

[Sandtitz]

Re: They never should've used it in the first place.

"Any program that takes over control of your devices (EG: mic & webcam) & prevents you from using them as you see fit (EG: not at all) is by definition malware that doesn't belong anywhere near the machine."

Well, the other options would be either take a in-person proctored exam (like we used to before Covid); or an honour based system with a pinky promise not to cheat. The latter is fine for perhaps sales (or pre-sales) certifications; I've done my share of them as well.

My latest technical exams were by Pearsonvue and required me to install their OnVUE software that took control over the mic and webcam. It also disabled Windows keyboard shortcuts (alt-tab, screenshots etc). I was also required to take pictures of the room I did the exams, and upload them before the session. And at start of the session I needed to show the room with my webcam as well.

When the session ended the program was closed, I regained full control of my computer once again.

I'm totally fine with this.

What you are proposing would result in a deluge of technicians and engineers who hold little knowledge in their supposed area of expertise, thus diluting the value of a real certification. No difference to $20 fake diplomas from Internet University.

[boblongii]

Re: They never should've used it in the first place.

"What you are proposing would result in a deluge of technicians and engineers who hold little knowledge in their supposed area of expertise, thus diluting the value of a real certification."

Oh, well, that would be an earth-shattering change, wouldn't it?

[doublelayer]

Re: They never should've used it in the first place.

"What you are proposing would result in a deluge of technicians and engineers who hold little knowledge in their supposed area of expertise, thus diluting the value of a real certification."

In my opinion, if that is true, it means the certification wasn't much good in the first place. The easy methods of cheating are where you look up answers online. That's kind of normal now for most jobs, because if you've forgotten something, it's better to just refer to the docs and therefore not make mistakes. If you're analyzing someone's experience with technical skills, then it makes sense to let them access those reference materials, because if they can successfully obtain the goal in the time limit, then that proves they know enough to find the information they needed and apply it properly. If they know very little, then they will most likely not be able to find the answer or tailor it to the situation in time, so they'll still fail. A test based on rote memorization isn't much use when you need quick analytical or problem solving.

In other classes, there's a better case for such restrictions. A student doing a math test shouldn't be able to just type the questions into a calculator. If you have an environment where the person really can't access the reference materials, then you might want a memorization test as well (but really, that's not many places). Most tests, however, aren't so basic.

[Sandtitz]

Re: They never should've used it in the first place.

"If you're analyzing someone's experience with technical skills, then it makes sense to let them access those reference materials, because if they can successfully obtain the goal in the time limit, then that proves they know enough to find the information they needed and apply it properly."

Of all the job application stories on these forums, the cases where the remote applicant Googles all the answers have been met with ridicule, scorn. Surely that's a resourceful technician if (s)he can come up with an answer within a minute?

The problem is not just Googling for an answer or reading reference docs. The exam taker may have ganged with other people to collectively answer the questions. Once again, in a workplace asking a more knowledgeable/senior colleagues for guidance if commendable (rather than messing up), but getting other people to answer test questions shows zero knowledge on the exam taker's part.

Yes, some questions may hinge on memorization of obvious limits. In a basic network exam I'd expect the exam taker to know what the difference between L2/L3 switch is; how many usable VLANs are available in 802.1q; selecting suitable technology for a specific scenario.

"In other classes, there's a better case for such restrictions. A student doing a math test shouldn't be able to just type the questions into a calculator."

Most of the math test questions from secondary school onwards were not pure equations but you had to derive the formula from the worded question or perhaps from a trigonometric drawing. Letting the calculator to calculate sines and tangents should be allowed once the student has shown prificiency in calculating them. It's the knowledge of using the functions that matter.

Also, most of the time just an answer wasn't satisfactory, you needed to write down the intermediate calculation phases as well, something my scientific calculator back then certainly didn't do.

[ThatOne]

Re: They never should've used it in the first place.

> What kind of adults do you want to train?

The best kind: Obedient and resigned to their fate.

[elsergiovolador]

Re: They never should've used it in the first place.

The system is built upon wrong assumption - it is based on a belief that everyone is going to cheat.

If you are starting from that position of distrust then you are not going to build anything on such rotten foundation.

The system should be designed assuming that _some_ people are going to cheat, but this should be found way before they even get to the exam.

For example, from my school years teachers knew exactly who is cheating, but they couldn't expel them, so at very least they had to monitor them while the rest of the class was rarely paid attention to. These AI systems assume everyone wants to game the system, which is insane.

[Anonymous Coward]

Re: They never should've used it in the first place.

"If you are starting from that position of distrust then you are not going to build anything on such rotten foundation."

But if that's all you have to work with, if you live in a DTA world...

[YetAnotherJoeBlow]

We don't need no education...

-- especially this way

[martinusher]

Its not really AI, is it?

There's a lot of software our there that takes a variety of inputs, munges them into some kind of index and then issues pass / fail type results based on that index. All of this software is characterized by it using unpublished, proprietary, algorithms and an aggressive legal attitude to anyone who questions the validity of the results. It doesn't matter whether its proctoring examinations, watching the work habits of employees or whatever, its a Black Box that both the vendors and the people who signed off on leasing it will aggressively defend as if their lives depend on it.

(Its been quite a long time since I had to do examinations but I recall that at anything beyond 'O' level it really didn't matter if you cheated because the questions were set up to probe subject understanding, not knowledge. Having all the facts at your fingertips is useless if you don't know how to apply them.)

[katrinab]

Re: Its not really AI, is it?

It is Machine Learning

The algorithm is the training data.

[doublelayer]

Re: Its not really AI, is it?

And the preprocessing done to the training data, the preprocessing done to the production data, the specific model or models in use, the output type of those models, the thresholds for consideration of those results, the algorithm to deal with conflicting results from different models, and changes made to all of those since last time it was clearly broken. They don't want to tell people details about any of those.

[Mage]

Re: Its not really AI, is it?

Indeed Machine Learning is a misnomer. Machines can't learn and AI today is a marketing term, nothing to do with AI as envisaged up to the early 1980s.

It's all human curated input (hopefully properly curated) stored in a specialist kind of distributed data-flow type database and then pattern matching. AI, Machine Learning and Neural Networks are all terms invented to make it sound cleverer and more accurate than it really is.

https://www.theregister.com/2021/08/22/in_brief_ai/

Such circumstantial evidence isn't good enough to put down a stray mongrel, and link includes idiot with total lack of understanding what unmoderated random people might do with a so called AI Images from Text generator. A bonkers idea.

AI medical diagnostics rely on human experts at the start curating a massive amount of data. The performance is exaggerated to get the data, really for ulterior motives (Google's medical subsidiary) or to sell a big system (IBM's system that didn't work and only shared a brand name with the Jeopardy winning system, which also wasn't actually AI, but a party trick more useful for Alexa, Siri etc, which are not AIs).

And if Medical AI did really work, who would be competent to "train" the systems in the next generation? A dead end solution.

[pip25]

Online exams do not seem like a good idea in general

It's like putting the open textbook next to the exam papers on the student's desk, then asking him not to look at it. These proctoring programs are problematic because they are trying to solve an unsolvable issue - and even as they invade the students' privacy, they still have a good chance of being wrong.

[Giles C]

Re: Online exams do not seem like a good idea in general

I have done several online exams during lockdown as the exam centres were closed.

To be honest next time I will go to the exam centre, as it is a lot less mucking about.

The online proctoring service the pearson Vue exams use does use the software to stop you accessing the internet, or opening up other files and as the exam is meant to be a knowledge test that seems reasonable.

Having said that I did use a spare laptop with nothing personal on it to do the exams.

But yes at the exam centre there are several cameras watching you constantly to ensure you aren’t cheating so you are under constant surveillance. The only other choice they could have is to lock each person in a cupboard with no access to anything. But that would probably be stressful for someone.

If you are taking an exam which is closed book, don’t try to cheat and if you are being observed then accept it.

[Warm Braw]

Re: Online exams do not seem like a good idea in general

it's like putting the open textbook next to the exam papers

Why unquestioningly assume that's a bad thing?

There was possibly once some justification for using memory as a proxy for academic achievement, back in the day when academic books were expensive, largely confined to libraries, retrieved by manually searching a card index and changed very infrequently.

There's no more intrinsic virtue now in being able to recall textbooks in detail than there is in being able to use a slide rule or log tables.

I'm somewhat surprised that I can find relatively little research into the efficacy of different forms of "summative assessment" and I suspect the value of such assessment is as much in measuring the efficacy of the teaching as it is in making a useful distinction between the taught.

However, what research I can find suggests that it's perfectly possible to design "open book, open web" exams that produce comparable results to traditional tests, you just have to design them somewhat differently. So if you can do it without spending a fortune on ProctorScope^TM why wouldn't you?

[M. T. Ness]

Re: Online exams do not seem like a good idea in general

Open Web exams look like a good idea. Only: are the university teachers willing and able to take all the time and effort needed to make them? And are they capable?

[Warm Braw]

Re: Online exams do not seem like a good idea in general

the time and effort needed

Well, of course, examiners rely on a recycled library of questions - which is why there was always a lot of emphasis on revising with "past papers". But compare with when Radio and TV came along: all those old Music Hall performers that had had the same act for 20 years needed to get new material or get out of show business.

And it's not as if the money going into surveillance couldn't be diverted into exam development.

[boblongii]

Re: Online exams do not seem like a good idea in general

"There's no more intrinsic virtue now in being able to recall textbooks in detail than there is in being able to use a slide rule or log tables."

I disagree. A textbook is something that someone has put effort into and has been edited, and possibly revised if it's a second edition, and probably 70k+ words of explanation. If you can read it and remember it, it's probably worthwhile doing so in most cases.

What are you suggesting is the modern alternative? The ability to use Google's half-arsed interface to Wikipedia where you can see what some unemployed moron typed in over a half-hour lunch break?

I don't bloody think so.

[doublelayer]

Re: Online exams do not seem like a good idea in general

In many cases, not having to memorize everything in the textbook. If, for example, I'm writing a program and I've forgotten all the possible magic numbers for a system call, I can find out those options from that call's documentation. I shouldn't have to memorize every system call and every option on every different operating system just to do that, because the information is available conveniently. The textbook can serve two purposes. First, it teaches the theory and provides useful techniques which the student will remember and apply later. Second, it provides reference information which the student can look up later. A good student will remember the former because they use it frequently, and can memorize those details which they use most often, using the book to recall those rarer examples.

[Warm Braw]

Re: Online exams do not seem like a good idea in general

That sounds like a false dichotomy.

The purpose of education is not to provide you with all the information you will ever need, but to give you the tools to learn and understand and some grounding in the current state of the art as a starting point. What you need to learn are the principles and how to weigh the various pieces of information available to you at any particular time.

The syllabus=>textbook=>examination paradigm is supposed to be a means of education and not an end in itself. You only have to look at English and Maths exams in the UK to realise the tail is wagging the dog: many of the grammatical concepts in English have been invented solely for the purpose of being examined and you'll be penalised if Maths if you don't use the specific methodology prescribed in the syllabus even if your approach is equally valid.

It's not a case of textbook or Wikipedia, it's a case of having the judgment to know what's likely to be relevant and reliable when you're not being spoon fed the "right" answers by an educator. Of course, that's both harder to evaluate and harder to rank than rote learning.

And it's sometimes difficult to escape the thought that perhaps the ranking - any ranking - is the primary consideration.

[lostcolony]

Re: Online exams do not seem like a good idea in general

So much this.

I never understood why so many tests in college tested -recall- ability. I.e., the thing computers are amazingly good at, and humans are really bad at. And which invariably I could cram for, and post test I'd remember nothing.

Pretty much every subject worth learning has ways to test for actual comprehension. For math, show your work. For literature, ask for how the material affected them. For sciences, apply it in some fashion.

I'm a CS major (shocking), and the classes that had tests were invariably the worst. The classes that had projects, however, were great. And as everyone who has ever worked on a group project knows, working together isn't a guaranteed recipe for success. If everyone is remote, too, there is no group peer pressure to 'work together', allowing others in the group to coast; you can just ghost people who aren't pulling their fair share.

Test for understanding; you can't google for it.

[NATTtrash]

Re: Online exams do not seem like a good idea in general

I think you're right. Furthermore, as with some more "nowadays" developments and ideas, it ignores basic things that always have been, and will not cease to exist all of a sudden. People are inherently lazy. Thus rather use some super software to do a humans job. Yes, that will also let them rather cheat than face the music that they might be sub-par, do it for real, and maybe fail. And let's not start about people dominating other people, telling them exactly what (they will allow others) to do, and accepting no variations? Nothing to see here, move on.

I'm not a techie like many of you here, more into human tinkering and fixing. Therefore I still hope that humans working with humans will stay. Since it proved and proves its worth. Or would you rather have an automated WhatsApp after the "machine" found that you have cancer, and decided you have decreased societal value, fixing cost too much, so won't be treated? "Nothing to see here, move on..."

[veti]

In other news.. .

... Professional drivers against self driving cars...

... Actors opposed to deepfake animation...

... Spongebob hates the idea of automated burger flipping. ..

It's human nature to be opposed to an innovation that threatens your livelihood. And from that starting point, it's not hard to come up with arguments against it. That in itself doesn't tell us much about the merits of the technology.

[emfiliane]

Re: In other news.. .

That definitely doesn't apply here, since the panel in question reviewed the service _after_ implementing it for a year. If anything, they pushed back too little initially, but realistically no one had really good answers until enough institutions all gave it a year to see if it shook out or not.

[doublelayer]

Re: In other news.. .

In the article, did we see anybody arguing against it who previously made their living as a professional student-watcher? The students themselves are complaining. Professors, who probably would really like not to have to monitor students because it's extra boring work are complaining about it. Those groups do not stand to lose their jobs, so maybe they have real reasons to complain.

[JDPower666]

How the hell can you sue someone for copyright infringement cos they linked to a public video??? And why does the person being sued have to be out of pocket actually defending such a ludicrous claim? Absurd.

[katrinab]

They are arguing that the url that links to the video is a creative work.

Which is BS of course, but even if it wasn't, it was created by YouTube/Google, not them.

And of course, there is fair use, which would allow them to copy extracts of the actual video as part of their criticism / comment.

[Daedalus]

I write software so I....

There's an online list of things that a software dev doesn't use because they know that software is usually crap, insecure and dangerous. Add online exams and online proctoring to that list.

This is a classic case of "let's pile on more crap to fix the crap we shouldn't have been using in the first place".

The worst of it is the use of low-bid software for tests in schools. Software that the schools don't understand, the teachers don't understand, and the developers don't give a monkey's about, if they're still around. The money people are happy, though. Yay!

[Anonymous Coward]

No need for this sort of thing

I'm a current student. Our exams this year were all "open book" - we had 24 hours to write an essay in response to a question and we were allowed to use notes, books, online resources and so forth. Had a student not done the work all year, they would have failed hard in this - you had to have done the work and made the notes and understood the concepts and key ideas and authors etc to be able to produce a passable piece of work in 24 hours. Because of the time constraint, you couldn't just go in blind, you had to know which resources to use beforehand.

But testing rote memory of a subject is less and less important. In the professional world, we all sit at computers with access to an internet full of information. The key skill the degree teaches you isn't the rote learning of that information, but the appropriate use of that information and the interpretation and understanding of that information to complete a task.

My department - one of the top departments in its field in the UK - is leaning very heavily towards not returning to the "sweaty sports hall" style of examinations when Covid is hopefully a thing of the past, and isn't planning on replacing them with this kind of software monitored approach either. Open book better replicates the real-world working conditions of the 21st century.

[Anonymous Coward]

"Introducing new and improved AI Snakeoil!!! Uses advanced AI technology to miraculously deliver on all the promises our original Snakeoil made but never delivered on. Available as an upgrade to Snakeoil for $49.95, or as a new license for $149.95. AI Operators are standing by..."

[Ken Moorhouse]

Panopto

The first thing that entered my mind?

Panopticon.

Spooky name for a company in this field.

https://en.wikipedia.org/wiki/Panopticon

"The panopticon is a type of institutional building and a system of control designed by the English philosopher and social theorist Jeremy Bentham in the 18th century. The concept of the design is to allow all prisoners of an institution to be observed by a single security guard, without the inmates being able to tell whether they are being watched.

Although it is physically impossible for the single guard to observe all the inmates' cells at once, the fact that the inmates cannot know when they are being watched means that they are motivated to act as though they are being watched at all times. Thus, the inmates are effectively compelled to regulate their own behaviour. "

[My other car WAS an IAV Stryker]

Re: Panopto

Nice allusion, which went on to be (fictionally) implemented in Nineteen Eighty-Four.

Early on, Winston muses on telescreens -- could they view them all at once, or only so many at a time? Thus, the Party populace had to act as if they were always being surveilled.

Icon, obviously -->

[Anonymous IV]

Proctology

On first (mis-)reading the headline, I assumed that the article was going to be about some magic method of performing a non-invasive colonoscopy (which itself is a sort of online examination). How disappointing that it referred to academic examinations.

[david 12]

If the software doesn't work, then dump it. If it costs to much, dump it. If exams don't work, them dump exams. If watching students causes anxiety, then don't watch.

But don't just conflate everything into one big objection to 'surveillance' , ignoring the weaknesses of the individual supporting arguments.

Frankly, the people I worked with who had the biggest objection to exams, and proctoring, and evaluation, were academics who knew that bad exam results demonstrated bad teaching ability, bad pedagogy, and lack of content.

[Michael Wojcik]

Linkletter's fund-raising campaign is currently suspended

Just wanted to note that according to his GoFundMe page, Linkletter has suspended donations to his legal fund. It's at nearly $86K CAD, so maybe he feels it's sufficient, at least for now.

In any case, more power to him. I hope his Anti-SLAPP petition succeeds.

[Simon R. Bone]

Not The Face

I did an online proctored exam a few months back and was constantly being told off for touching my face (!) I like to think holding and scratching my chin but apparently this could be some sort of Who Wants To A Millionaire cheat code...

[Meeker Morgan]

The real test ...

... is faking out the AI software. Pass that test and have it made.

Ar least they stopped this AI before it went racist.

[Kevin McMurtrie]

Used for job interviews too

This crap happens in job interviews too. You only have to read the terms and waivers to know how much of snake oil marketing trick these services are. I can't imagine how angry students would be to have this garbage suddenly dumped on them after months of studying.

At least in the job interviews, I can decline and not have wasted too much of my life.