'Meritless': Exam software maker under fire for suing teacher who tweeted links to biz's unlisted YouTube vids

A copyright-infringement lawsuit brought against a university staffer by Proctorio – a maker of software that monitors students during online exams – was this week dubbed "meritless" by an EFF analyst. "Proctorio should cease its efforts to muzzle critics," the foundation's Joe Mullin added. Last September, Ian Linkletter, a …

[Anonymous Coward]

Am I the only one that thinks closed book exams are not that relevant any more? Certainly in most work sectors you have others to bounce off, and we now have access to a massive library of cat videos at our fingertips, not to mention all the information in the world.

So what if someone passes tests by reading notes or using a calculator. Students need to be taught how to learn, how to research, how to work.

[John Riddoch]

I sat my French exams in 89 & 90 with an English-French dictionary to hand. Theory was that if you were bad enough to need the dictionary too much, you'd lose time looking up words to be able to finish the exam in time. You needed the base vocabulary and could look up the odd word which doesn't come up often.

I've also done an IT technical interview where I could use Google as much as I wanted; was stuff like "configure Tomcat to do XXX" or "this MySQL DB isn't working, go fix it", so it was useful for looking up the obscure error messages or "how to" bits for things I hadn't done before on Tomcat.

I've also had to sit certification exams where some of the specific knowledge you need is obscure and not required very often, so it seemed odd to include it in such an exam. I could have found the answers quickly in the documentation, but that wasn't allowed.

[onemark03]

Open-Book Exams

Sat a few of those in my time.

I can only endorse the comments by this commenter on wasting time.

[John Robson]

There is space for both sorts of tests - in the same way that there is space for both with and without calculator maths examinations.

They are testing two different things, one is testing your recall of the basic foundations of the subject, which you shouldn't need reference material for - the other is your ability to understand the topic, know it well enough to get to the correct place in the reference material and put forward a reasoned argument for something much deeper than could be achieved in a closed book exam.

In terms of the calculator I very rarely type anything into a calculator without having done the mental arithmetic to know what sort of number I am expecting to see come out (order of magnitude, approximate value of first digit, should it be rational...). And that's a key skill, for which explicit mental arithmetic training and testing should be encouraged.

[Jonathan Richards 1]

Ballpark estimation

> that's a key skill

... and one that was simply required in the days when a slide rule was the calculating engine of choice. I attended physical chemistry seminars in which the problems had to be calculated to 9 s.f. with logarithm tables - not because the precision warranted it, but because the lecturer then knew that we hadn't used one of those subversive new-fangled 8-digit electronic calculators he'd heard about.

[Michael Wojcik]

Sure. But it's not a skill that can effectively be tested for remotely, so it doesn't justify the sort of half-assed vigilance theater that Proctorio are selling.

[big_D]

The thing is, you need a base of knowledge on a subject to be able to tell whether the information you are looking up is relevant or accurate. This is something that is severely lacking among a large number of people these days.

But the exam is there to see how much you have learnt, do you actually know the subject being taught. It isn't an exam to see if you can look up random topics online. I do agree that they need to learn how to learn and how to do research, but that is not what an exam is there for; it is there to see if they have managed to learn and do their research.

I use search engines regularly to supplement my knowledge - the key being supplement. If I spent all of my time just browsing forums and reading online articles, I'd never get any work done. That basic knowledge has to be there to start with.

[Fonant]

Quite Agree: memory exams are mostly useless

All my University Mechanical Engineering exams were open-book. You could take anything you liked into the exam, so long as it wasn't too big and you could carry it. Books, notes, calculators.

This meant you had to really understand the subject, and know where to look for answers. Just like in the real world. No engineer would rely solely on memory, so why would testing that ability be of any use?

This was way back in the 1980's, it's not a new concept.

[onemark03]

Re: Quite Agree: memory exams are mostly useless

Agree entirely.

I sat some (non-tech-related but UK-based) open-book examinations back in the early noughties. IMHO, they are a two-edged sword. It's nice to be able to look things up BUT you still need to know your stuff otherwise you waste (a) valuable exam-time looking things up and (b) checking-time at the end. Consequently, the pass-mark was 60%, not 50%.

[DarkwavePunk]

Re: Quite Agree: memory exams are mostly useless

I think there's a certain prudishness against open book exams. Wen I waz studaying tu be a Enjinear in the early '90s it was really dependent on the department or even teacher as to whether it was allowed.

As you say, you need to know the foundations to make any sense of reference material in the first place.

Also fuck Materials Engineering for making me remember entire sets of Phase Diagrams off by heart. No disrespect to any Materials Engineers out there and fortunately enough alcohol has been consumed to erase most of my memory of such things past gone.

[Jonathan Richards 1]

Re: Quite Agree: memory exams are mostly useless

+1

I could once derive the 3-D stress tensor for a non-Newtonian fluid from first principles. That's 'once' as in 'on the day of final exams' - couldn't do it before, haven't been able to do it since, and more importantly can't imagine the circumstances in which anyone would have to do so!

[John Robson]

Re: Quite Agree: memory exams are mostly useless

"That's 'once' as in 'on the day of final exams'"

That's most of my quantum mechanics knowledge summed up right there...

At least merely doing a physics degree (rather than chemistry) we only had to be able to derive, calculate and prove things about a hydrogen atom, nothing more complex.

[Bitsminer]

Re: Quite Agree: memory exams are mostly useless

Yes, but....

Back in my engineering school days, there was an open-book exam, with a question about deriving the capacitance of an oval coaxial cable. The question exactly matched a sample question from Schaum's outline that gave the requested formula.

One too-clever student wrote copied the formula into the examination paper.

Along with the comment "by inspection".

He got full marks for his answer.

[ibmalone]

Re: Quite Agree: memory exams are mostly useless

This is particularly a problem with 'un-invigilated internet open book' versus 'you can bring in your notes' open book. It is much harder to create questions where an exact worked solution isn't out there somewhere, and at some point you want to be able to check the candidate can actually understand and apply the subject rather than simply copy someone else's solution. Additionally, student's commitment to helping each other out sometimes extends to sharing the answers with each other during the exam.

One Coursera course particularly sticks in my mind; students were to grade each-others answers after submissions for each exercise closed. You got used to noticing somewhat weird and quirky solutions, typing them into google, and finding exactly the same code on github dating to a year or two ago. Was never clear if the plagiarism reports went anywhere. (There was also the particularly special case of someone suggesting that following the marking rubric was the Nuremberg defence...)

[WolfFan]

I trained as an electrical engineer, at that time the university I went to didn’t have a separate IT degree. This meant that in addition to computer-type courses (FORTRAN, PL/1, C, programming logic, networks, etc.) I had to take the likes of electromagnetic field theory and electrophysics. Emag FT was extremely annoying. We were allowed to take a cheat sheet in (actually, up to four pages) with stuff of our choice which we thought might be useful. But… if you knew where to look for the correct what’s it, odds are that you already knew the correct what’s it, and looking it up just wasted time. And the tests were written expecting that we would have certain items either in our heads or on the cheat sheet, so that’s more like _required_ than allowed. I will say that knowing how antennae behave (and what an antenna is, it’s amazing how few know all the things which can be considered to be antennae… hint: that USB keyboard cable is an antenna….) made tracking certain network problems down easier. I still have the emag FT textbook… (I dumped the ephys text within five minutes of getting out of the final…)

[Gene Cash]

To quote my high level math prof: "you can use anything you want on the test, except another student, since it's your knowledge and cleverness I'm trying to measure"

He was famous for his extra-credit questions being derived from the very next chapter in the textbook. He did get us to read ahead, which was his goal.

[doublelayer]

There are some cases where the student shouldn't have a bunch of notes or external access, but usually the downside of having them can be mitigated by a time limit or a keen grader. For example, having a student in programming copy answers off Stack Overflow would be a problem, and similarly making sure an interviewee isn't going to take that approach can be important. There are also jobs where someone really needs the ability to memorize information and use it without access to reference materials, either because those materials aren't available or because there isn't sufficient time. Barring those, however, most tests don't really need to be that restricted.

[nematoad]

Which is it?

Protorio or Proctorio?

It comes across as a bit slapdash when the name of the subject of the piece seems to vary. I found that the impact of the discussion was lessened because of the inconsistency.

Nitpicking perhaps but this sort of thing is something that should be spotted before publication.

Oh, one last point. If the name of the company is Proctorio is their application called Proctoscope?

[Chris G]

Re: Which is it?

In the States, proctors is the term given to invigilators at exams.

I am now wondering if the study of proctors would be proctology the same word as that used to describe the study of arseholes.

[Prst. V.Jeltz]

What i find amazing about this story is that a company is actually selling sofware that claims to invigilate and detect cheating by looking into the students eyes.

also amazing that anyones bought it!

Its a great idea , but shurley to anywhere near usuable it would need another 20 years R&D

We've barely mastered face or voice recognition thus far after decades of trying

[AndersH]

I would guess the people making the purchase decision probably have no background in tech. They simply believe the sales pitch.

[Dave314159ggggdffsdds]

It isn't sold as being able to tell if someone's cheating. Just to suggest targets for observation by humans.

As long as the results are treated as being largely false-positives by whoever is getting them, it's no different to what invigilators do in exam halls when they look for e.g. students who aren't looking down at their desks, and watch them for a minute to check they're staring into space and thinking, rather than trying to crib answers.

[Electronics'R'Us]

But the computer says...

What you state may be the guidance, but the reality (I seem to recall some coverage of this a little while ago) is that it is used specifically to identify 'suspicious' activity and that is accepted by the shoe size IQ operator as 'cheating'.

[doublelayer]

I'm sure that's what they say, just like the police facial recognition systems probably say the same, but they'll be used in the same way. This program says the activity is suspicious and at best someone will look at the video and go "Hey. They did move their gaze. How should I know if they're looking at the window or at a screen behind this one. I'll trust it." and the student will get questioned or disciplined. I can see lots of possible eye movements during a test, from a bored student looking at the wall while thinking to someone unsure of an answer having involuntary reactions to their anxiety about their answer. Whatever the documentation may say, I think it's so unreliable as to be unacceptable to use it.

[Glen Turner 666]

Proctoring software, unreliability coupled with high stakes consequences

A common configuration for "online proctoring" software is that being "flagged" by the software halts the online exam. This supposedly prevents people blatantly cheating and selling the exam answers in real time.

Even more moderate conditions still suck. If the software flags the student and then the exams invigilators check the recorded video, and then allow the exam to continue, that's a time penalty which may result in marks lower than the student's actual knowledge and skill.

If the proctoring software fails, then the student's examination is invalid. Putting that another way, better pray your operating system is no good at detecting malware-like behaviour.

Proctoring software usually objects to other people in the room. Students can't reliably take an exam from a public library, or from a share house, or from a room with popstar posters.

The software objects to the student's face leaving the camera's frame. Better not drop a pencil, be harassed by your little sister, knock the laptop lid.

The software does eye tracking, the idea being to detect use of notes. Better not sneeze.

Some proctoring software fails to detect people present when their skin colour has insufficient contrast. Got to love that the "systematic racism" here involves two meanings of "system".

For high-stakes exams the invigilators will often want the laptop's camera taken on a tour of the room. Bedrooms are sometimes too complex a space to pass this inspection. Of course students don't know this until just before the exam.

The presence of proctoring software increases the stakes of already high-stakes exams for students. Why a university would choose to do this -- rather than rapidly redesign assessments -- in the midst of the most demanding teaching year since 1939 says a lot about the relationship of the university to its students. It's also a great illustration that education isn't only what is said in the course.

[Alan Bourke]

America is never shy

of trying to throw pills or technology at problems that would be better tackled by addressing the root cause.

[Nightkiller]

Re: America is never shy

"Americans can be counted on to always do the right thing, once they've tried everything else"

-- Winston Churchill

[Jonathan Richards 1]

Re: America is never shy

Rather "You can always count on Americans to do the right thing..."

You wouldn't ordinarily catch Churchill voluntarily splitting an infinitive. :Pedantic anti-Nazi: ==>

[werdsmith]

Re: America is never shy

I wonder how many of these wise quotes are misattributed. High 90s % I would guess.

[Michael Wojcik]

Re: America is never shy

As Shakespeare once wrote, "People will attribute anything to Churchill on the Internet".

[KittenHuffer]

It is a great idea, and stop calling me Shurley!

[Dr Scrum Master]

Presumably it's tracking pupil movement to see if the examinee is looking at concealed notes, etc.

[jason_derp]

I'd fail every time then. I was occasionally made to do tests alone in a room thanks to my wandering eyes. I get so bored during tests I look around for any mental stimulation besides the game "look at all the items in a room and conceive of a way to kill yourself with them". I'm not even saying I'm particularly good at tests, I just find them so boring I get exhausted and go searching for anything slightly interesting to keep me going. I've fallen asleep during tests too, which is why they started sticking a teacher in the room with me.

[Giles C]

Well my last exam was done at home and was a remote proctored Vue exam.

I needed to ensure I was in a place I couldn’t be disturbed (easy I live on my own) had to take pictures of my work area, and the camera ensured that I was in front of my laptop for the whole exam and didn’t leave shot (doing so would have been an automatic disqualification).

Given a choice I would prefer to go to a testing centre as it is easier and a nicer experience but during a pandemic it wasn’t possible.

It is a good job they didn’t use on chess players or Beth Harmon would have been chucked out of her matches for starting at the ceiling......

[Androgynous Cupboard]

Voight-Kampff test

Not just pupil movement. They're looking for capillary dilation of the so-called blush response. Fluctuation of the pupil. Involuntary dilation of the iris. In fact, here's one of the exam questions:

"Q1. The tortoise lays on its back, its belly baking in the hot sun, beating its legs trying to turn itself over, but it can't. Not without your help. But you're not helping. I mean: you're not helping! Why is that, Leon?"

[yetanotheraoc]

Re: Voight-Kampff test

"The tortoise lays on its back, its belly baking in the hot sun, beating its legs trying to turn itself over, but it can't. Not without your help. But you're not helping. I mean: you're not helping! Why is that, Leon?"

Since there's no tortoise, no sun indoors, and my name's not Leon, I would say their surveillance system needs work.

But seriously, watching for pupil dilation in this scenario is supposed to show what, exactly? That the eyes being monitored actually belong to Leon-taking-the-test? Oh, your pupils didn't dilate. The camera must have been focussed on your doppelgänger while you were cheating! No, no, it was me, I just don't really care about tortoises....

[Warm Braw]

Is there any aspect of education that hasn't been diminished by corporate rapacity?

I'm old enough to still find it breath-taking that students' "learning" is now barely more than a grudging, copyright-controlled loan from an educational megacorporation and that they're being deliberately inculcated with the notion that their future is inevitably subject to Orwellian surveillance.

There was a time education was specifically about the opposite.

[gobaskof]

"Proctorio uses an opaque proprietary system to measure 'behavior', calculate 'abnormalities', and then assign a 'suspicion level' to each student,"

Orange lied to us! We were promised a bright and orange future, not this dystopian hellscape where computers assign us suspicion levels from cameras in our homes. Fuck any university that thinks this is ok.

[Chris G]

You weren't lied to, you were just sold the potential benefits.

It's just unfortunate that the orange you bought was difficult to peel, sour and mostly pips but it's a nice colour.

[Electronics'R'Us]

Herein lies the problem

"Proctorio uses an opaque proprietary system to measure 'behavior', calculate 'abnormalities', and then assign a 'suspicion level' to each student,"

The entire premise here is that the students cannot be trusted which is not a good thing.

As others have noted, you have to have an understanding of the core subject to make any lookup (on searchengine fu) meaningful.

If I were enrolled in something like this (unlikely as I am well past that age by several decades, but still), I would insist they provide me with one of their computers (which would have very limited access to anything else on the very slow network it would be provisioned for) as I would most certainly not permit this sort of surveillance crap on any privately owned kit.

[gobaskof]

Re: Herein lies the problem

Working at a university (that luckily does not use this shit, all remote exams open book) you can insist all you want, they won't lend you hardware. This is such an insane violation of privacy I am not sure how I would act, but there would be lots of shouting and swearing. I was angry enough 12 years ago when a lecturer wanted us to sign up for iTunes to get podcasted lectures.

[TheMeerkat]

Re: Herein lies the problem

But the student really cannot be trusted. This is how exams work.

The only way to have proper grading and testing is to just have normal exams as we had then before the lockdown. Nothing else is going to work.

[Pascal Monett]

"protect our intellectual property"

Your intellectual property that is blatantly displayed in public video available on YouTube ?

Is this a joke ?

You're not protecting your IP, you're trying desperately to protect your reputation, just like Diebold.

And, just like Diebold, you're going to find out that using lawsuits to silence criticism does not work.

[Anonymous Coward]

Re: "protect our intellectual property"

Any number of companies host and lock their own training and sales videos themselves, not on YouTube.

Proctorio posted them on YouTube for the public to access because it was cheaper and easier.

EFF has it right, there's no eff-ing case.

[Claverhouse]

Re: "protect our intellectual property"

To be honest, there were allegations of faulty machinery when more votes went to Bush via Diebold than the MSM considered fair; but since the MSM have fully realised that criticisms of that machinery, now run by the new owners Dominion, by Mr. Trump, are totally unfair and that it gives good vote.

.

[ Diebold Election Systems, Inc. changed to Premier Election Solutions sold to Election Systems & Software, sold to Dominion Voting Systems ]