GitHub Copilot may be perfect for cheating CompSci programming exercises

Microsoft's AI code-suggestion tool GitHub Copilot is showing itself to be so capable that educators may have to rethink how they teach computer science. University of Massachusetts Amherst computer science professor Emery Berger earlier this month published a blog post warning educators that "students armed with [Copilot] …

[Anonymous Coward]

Not in academia

But I wonder if not grading those tests would be (part of) a solution?

There seem to be two issues at play:

* People putting themselves at a disadvantage by not learning/ practising the stuff (their problem)

* People putting others at a disadvantage by getting comparable or better scores without having put in the effort

Not grading would address the second issue.

(Edit: nah, this is a daft idea. Still, I'll leave it here for discussion)

[doublelayer]

Re: Not in academia

I don't think it is a daft idea. This is a problem for introductory assignments sent out as homework where you can't add complexity because the students don't know enough yet. It's the kind of thing that we can write in two minutes today, so there's only so much you can do to prevent someone getting a simple solution. Even plagiarism without being detected would be pretty easy with this kind of question.

My answer to this problem is like yours: put out those homeworks, perhaps give students a small grade for having turned something in, and give them feedback on how well it worked, but don't grade it. People who want to learn will do them anyway and get good results. People who cheat will get only a few points. Then, give everyone a test on restricted computers or even on paper (though I don't much like coding on paper, it can be done), and grade that. Someone who learned with the homework will do it again and score well. Someone who cheated will probably look at the questions and panic because their cheating tool isn't available. This also gives any student who wants a dry run in case they don't understand a concept, so they can learn it in time. The homework can be treated as a practice with feedback, and the test can be secured from cheating more easily.

[steviebuk]

Re: Not in academia

Or you'll have people with poor memories like me who's old but trying to learn the small assignments. Completes one but then for the next one still needs to refer to the last one due to already forgetting some of the formatting for some functions.

Another idea is telling the student "I want you to comment your code. Tell us what each bit is doing. Then when you've done one whole section. I'll ask you to come into my office and explain your code." Could even do it remotely if needed. Problem is, Uni classes are probably too large to do that.

[doublelayer]

Re: Not in academia

I don't think that will be a problem. In most courses, the first exam is at least a month if not two into the term, so you'd have lots of practice before it gets to the test. Referring to the old homework when doing the next one wouldn't be a problem under any course conditions I've experienced. By the time you get to the test, hopefully the experience of having done what would probably be ten to twenty of those tasks has helped you to understand the structure of a chunk of code so you can reproduce it.

[Brewster's Angle Grinder]

Re: Not in academia

If it's a locked down machine you could provide the language spec and API documentation. You could perhaps print them out if it's a paper test. (Hey ring binders! Now you're talking!)

Lets face it, 50% of programming time* is googling the spec. There is just so much stuff we have to remember that none of us can hold onto it all down to the tiniest detail.

_{* Humorous exaggeration. Except when I get sucked down some rabbit hole.}

[mtrantalainen]

Re: Not in academia

If the task is to compute Fibonacci numbers using recursive function calls, AI can fill in the comments, too.

Ask more complex (to understand, not to implement) questions and the AI cannot fill the response anymore.

If you want to test for understanding of revisitation, ask the student to write some function that demonstrates that.

In pretty sure that AI cannot yet write code to demonstrate understanding of recursion without giving actual algorithm to implement.

Maybe the task could be "write a function that demonstrates use of recursive function to compute something else but Fibonacci numbers or factorial".

That would guide human students to look for examples for those two cases and then decide about their own function.

[Anonymous Coward]

Re: Not in academia

My teacher used to call me a keyboard junkie. He always said to write out your project on paper in steps first. However I always ended up writing my project steps in the code as comments so my code always ended up documenting itself. Kind of like

# I want to read in x,y,z variables and verify

<then I would proceed to write my code snippet>

# I want a function to log

<then I would write my logging function>

# I want a function to handle failures

<then I would write my failure function>

You get the idea. It works for me, and I can usually go back to old code and quite quickly work out what I had done and even the reason behind my thinking.

[Anonymous Coward]

Re: Not in academia

> or even on paper (though I don't much like coding on paper, it can be done)

I know it can be done, and it's great. I was only allowed to touch the computer (at my dad's office, back then computers were extremely rare not to mention expensive) after I showed my dad that I could program. We went through an introductory LISP programming book together first and I had to do all the exercises with pen and paper. I was about eight.

The world could be very different today if every child was competent at programming by age ten or twelve. But I digress.

[doublelayer]

Re: Not in academia

"The world could be very different today if every child was competent at programming by age ten or twelve. But I digress."

Right, good luck getting that going. While you're at it, I have several other skills that could be handy if everyone knew it by age 12 instead of some people knowing it in adulthood. I learned it relatively early as well, but not everyone has the interest to do the work required to have that experience, and some people would prefer to spend their time learning something else by then. If you have a miraculous invention that enables everyone to know a lot more than they already do, by all means let's train everyone to write code. If not, let's not force people to do it if they don't want to and are planning to use their time doing something different.

[Anonymous Coward]

Re: Not in academia

Given the ubiquity of computers and their economic significance I would have thought knowing about them would be quite a priority.

Instead, for most people they are a means of passive consumption, a role that TV used to fill.

[doublelayer]

Re: Not in academia

You can make a case for a lot of things being critical knowledge that everyone should learn, but for most things, it's not true. We would probably get many benefits from teaching everyone medicine. I'm not talking first aid and biology. I'm talking about proper medical school training including basic surgery, including practical application as early residents. We also live in a society where laws and contracts are important, so let's give everyone at least a year or two worth of legal training. We live in a world of many cultures, so you'll have to be fluent in at least four languages from different language families. We don't have an extra few years to have everyone do that, so we don't. I'm guessing that, like me, you don't have a medical or legal degree.

We live in a world with a lot of computers. That doesn't mean everyone has to program them. Programming is one of the things that's easier to self-train, and I know many good self-taught programmers out there. By all means we should have resources available for those who want to learn it. I'd even be happy with a mandatory small chunk in the curriculum that introduces everyone to the idea, and they can decide from that taste whether they want to continue on to more advanced work. Training everyone to be good programmers by the age of twelve, on the other hand, is elevating what we like over a lot of more important things for little benefit.

[Ian Johnston]

Re: Not in academia

Given the ubiquity of computers and their economic significance I would have thought knowing about them would be quite a priority.

By the same token, all programmers should have a sound grounding in electronics and chip design.

[Blackjack]

Re: Not in academia

If you don't grade anything and only have like a final exam that actually counts, some people won't learn shit.

I say that as personal experience, I had a teacher that did that, I ended quitting after six months since not only I did really bad in the "Final exam" I didn't actually learn anything.

[Anonymous Coward]

Re: Not in academia

> I say that as personal experience

That doesn't prove anything. Were the exercises corrected, with relevant comments and such, or was it fire and forget?

I did part of my studies in France and at certain points during the year we would count the homework in kilogrammes per week (no joke). It wasn't scored as such but we had to correct and comment on each other's homework so you could be failed if you didn't turn yours in. It was a character building experience that was.

[Blackjack]

Re: Not in academia

"Were the exercises corrected, with relevant comments and such, or was it fire and forget?"

It was the early 2000s, the teacher didn't grade or correct the homework, just told us of the password to his website that had the correct answers. Yeah he changed the password every two weeks but that's not the point.

So not only we were supposed to do the homework, we were supposed to not cheat and to correct it ourselves.

Do note the class was only three times a week and the teacher just left after each class and he liived in another city far away and no, not making this up.

So while yes we could ask some questions if you got lost in the beginning that was it.

It was also programing and math, and we weren't told of any books we could buy and or read so for me at least it was a complete disaster.

[Pseudonymous Howard]

Re: Not in academia

We are talking academia, not basic school. As long as the tests are corrected and commented, there is no need for grading tests. The comments should tell the students where their deficits are, so they know what to focus on to pass the exam later. This would make cheating in tests totally pointless and failures in the tests are not a disadvantage but an asset which give students information on their learning requirements.

To be honest: I don't want people, who are not able to learn in such a way to be able to pass any exam.

Graded tests lure students into cheating, especially if those test grades also influence the final grades.

Any task solved with cheating will not yield anything useful for the student.

[Anonymous Coward]

There are anti-copilot techniques...

A Uni lecturer friend and I discussed a similar issue which was how to produce an exam that could be taken at home during the pandemic with no supervision? One option we came up with was to include a program that contained mistakes and ask the student to explain what was wrong and why.

For now, at least, Google and copilot can't do that.

The problem with the approach is that the student learns how to debug a program (not a bad thing in itself) but not how to write it in the first place.

[tfewster]

Re: There are anti-copilot techniques...

Level the playing field in both teaching/coaching and exam situations by giving them a controlled development & test environment such as a standardised VM to work in?

No direct Internet access or remote copy/paste, and if they type in a complete program from start to finish, they're copying from somewhere.

You could record the sessions to review their technique. Though that would still need a lot of supervision/coaching.

[Anonymous Coward]

Re: There are anti-copilot techniques...

tfewster,

Like the idea but it would be quite labourious to create and to review the sessions for every student.

Using a paper based system is not such a bad idea.

I know that it would be a disadvantage to people who work better on-screen with all the 'toys' *BUT* ultimately you are trying to teach something which they should be able to explain, in extremis, on a piece of paper/whiteboard etc.

Simply, ignore the obvious typos etc, which should not be there if you are being strict, and check if the basic approach is correct and the techniques make sense.

[steviebuk]

Re: There are anti-copilot techniques...

Problem is some of us learn by looking at others code. Currently trying to learn Python doing the free Harvard course. But struggle at times with the problem sets and makes me, at least, want to give up and move on. Looking at others code really helps when totally stuck. However I make sure I understand exactly what is happening for every line. And I've even changed the code in mine and done the solutions different ways.

My logic has never been great so when some take an input and convert to an int at the same time. I know I wouldn't of thought of that so do it the longer way of taking the input then on the next line converting it to an int. I never totally copy, I always make sure I understand what the code is doing.

[Anonymous Coward]

Re: There are anti-copilot techniques...

> makes me, at least, want to give up and move on.

You're on the right track. That's how most competent programmers feel about python.

[An_Old_Dog]

Re: There are anti-copilot techniques...

I like the restricted dev environment idea! They take the tests (do the programming) on ethernet-connected uni lab computers, with no general internet access, and proctors making sure nobody's using a personal device to access the Internet (or plugging a wireless USB dongle into the lab PC they're using).

[An_Old_Dog]

"Guess the number I'm thinking of"-like tests..

Decades ago I had a jerk teaching assistant pull that one on us. The test consisted of an uncommented program, with lines deleted. We were supposed to "fix" it, but we weren't told the program was supposed to do!

[Anonymous Coward]

Re: "Guess the number I'm thinking of"-like tests..

> The test consisted of an uncommented program, with lines deleted. We were supposed to "fix" it, but we weren't told the program was supposed to do!

That's the kind of tests that used to get me in trouble.

I'd probably have deleted the rest of the lines and left a comment to the effect of "your code was shite but not to worry, I got rid of the evidence".

[b0llchit]

Re: There are anti-copilot techniques...

One thing is "fix the program". That indeed should develop debug skills.

Another is to change or modify the program. Then you need to understand the current form and be able to write new code. One trick can be that most of the existing code must be reused (as part of the assignment).

[Anonymous Coward]

Re: There are anti-copilot techniques...

> The problem with the approach is

How about giving them the program with errors as you describe and asking the student to reimplement / refactor all or parts of it according to a new specification?

For some people, most of their programming career consists of maintaining someone else's code anyway.

[Anonymous Coward]

Re: There are anti-copilot techniques...

If you went into a job interview and said:

"Sorry they didn't teach me to code. I just spent 3 years fixing up other peoples programs"

You would get employed on the spot.

[doublelayer]

Re: There are anti-copilot techniques...

No, you wouldn't, at least not until you could prove they did so well. If, for example, they had you fixing bugs that were all a bit obvious, all revolved around incorrect behavior, etc, you're not competent for most programming tasks. One of the frequent requests of a programmer to modify existing code isn't to fix a bug, but have it do this stuff but faster than it's doing it now. That takes understanding of how the code interacts with the systems and how performance is affected. Many others are about adding features. Some even need functionality started from scratch. They want to test that you know how to do all of that, and if you only know how to fix someone else's bug, they might be able to find better people. If they can't even prove you know that, you're not a good candidate.

[Anonymous Coward]

Re: There are anti-copilot techniques...

In a perfect world, people wouldn't be allowed to write their own code until they had spent at least five years fixing up other people's poorly designed, unnecessarily complicated and badly written code.

It really is the only way to learn how to write maintainable code.

[doublelayer]

Re: There are anti-copilot techniques...

Only if you have good solutions. Otherwise, it teaches people a lot of bad habits. If you learn from a lot of people who weren't able to write good code in the first place, then you'll end up basing your knowledge on what's by definition the wrong way. Yes, some of that would be remembered as the bad way to do things and you wouldn't reproduce it, but other bad practices would probably still get through.

[Blackjack]

Re: There are anti-copilot techniques...

Just do both in the same exam.

That way the kid using Copilot will never get better that a 50% and that's a fail.

[Michael Hoffmann]

Use custom language?

Going way way back here, but when I went in for compsci in the 80s, they taught us with a turing complete custom language which nobody else used. Don't even remember what it was called, but it leaned heavily on Pascal with a bit of "verbosity for teaching purposes" so some COBOL thrown in.

Of course you didn't have a home setup or Internet anyway, so this comparison only goes so far.

But even if, we sure could not have used CoPilot. Unless we already knew other languages and could cross-port back and forth. Anybody able to do that is already qualified to probably teach the unit.

[Anonymous Coward]

Re: Use custom language?

The CJ Date approach, eh?

[An_Old_Dog]

Re: Use custom language?

In my college freshman Pascal class, for a while they had us writing/testing/debugging programs written in a synthetic language called "Gear's Assembly Language." The emulator ran on our CDC 3300 (which in turn ran a nifty, developed-in-house time-sharing operating system called "OS-3").

I'd love to get hold of a copy of OS-3 and/or G.A.L.

[Evil Auditor]

Re: Use custom language?

The following decade, writing «pseudo code» on paper were many of my programming exams.

[agurney]

It's not problem .. you have a chat with the student, let them know you're impressed with their solution, and ask them to explain it.

You are testing comprehension, not necessarily implementation.

So what if you are using someone/something else's code, does it answer the question that's been presented?

[Anonymous Coward]

> you have a chat with the student

Have you seen a university classroom in the last 15-20 years? There are countries with less population than a typical class these days.

[blueatria]

What is the problem here?

I don't get the problem. Having tests/exams set that require people to remember syntax and particular function calls is lazy. Testing should be about problem solving.Copilot is really just a more advanced version of code suggestion in IDEs. This is like the idea that programming on paper is a useful test, why? Testing problem solving and giving people motivation is what exams and testing should be about, whether or not I can remember the peculiar syntax of a specific language is not a real test of whether or not someone is learning. Memory tests are pretty much useless. Just my 2 cents as I am a current student after having worked in computing environments for many years.

[doublelayer]

Re: What is the problem here?

For most courses, you're right. For introductory courses, the first step probably should be teaching the inexperienced student how to use a tool. Their solution to each problem will be built from language concepts that they have to understand, so it makes sense for the first introductory class to test their ability to use a language. If you don't understand how a loop works, your solution to a problem, if you have one at all, will be hideously unmaintainable, so it's important that everyone learn quickly what loops are and how they're used.

[Anonymous Coward]

Re: What is the problem here?

> it's important that everyone learn quickly what loops are and how they're used.

Coming from a functional background acquired at an early age I've always been slightly confused by loops. I only started to grasp them when we did an introductory electronics course in school. Sometimes it helps to think of programmes dealing with mutable data in terms of electronic circuits.

[mtrantalainen]

How about just fixing the tasks? Don't create so easy assignments that AI can directly fill the answer.

If your intent is to train future software developers, don't deny access to tools that would be used in real work. Instead, ask questions that require understanding instead of formatting known algorithm to a given simple question.

[doublelayer]

I think this is most useful in the introductory courses. When you're teaching someone how to write really basic functions because they don't understand all the concepts they'll need to implement a solution, you can't just keep adding complexity. You can find simple examples that aren't in common use yet, but eventually you'll run out of good ones. There's a reason that Fibonacci and factorial are always the first two examples of a simple recursive algorithm, because they're easy to understand and write without needing extra knowledge.

If you're making good software developers, you need more than understanding someone else's code. At some point, they'll have to write their own code and it will be in the next class when the problems are harder. They should start writing stuff now, even if the answers are kind of basic.

[Anonymous Coward]

> Don't create so easy assignments that AI can directly fill the answer.

The problem is not the assignments being easy. The problem is that the AI has been trained on those very assignments.

[John Smith 19]

So why exactly do you need to train people to do programming at all?

That's the ultimate question

It can do what it's told to do.

What happens when it's told do something it doesn't know anything about already?

[doublelayer]

Re: So why exactly do you need to train people to do programming at all?

Spin it up and give it a try. I haven't tested it myself, but having seen a few examples, I'm pretty sure it won't be able to read a spec and spit out a program any time soon. It probably will choke pretty quickly on most assignments after introductory courses. We have to train people to write programs because this tool isn't an AI programmer with perfect knowledge, and we need to start by training them on the basics that this one can do because they need the easy building blocks to get started. If we want people to self-train the introductory steps, this is no problem. If we want to use the traditional get-taught-and-tested method, we'll need at least a few occasions where they don't have this to make sure they have a good starting position.

[Anonymous Coward]

Oh no, not the Uni grads

I'm sorry, but every Uni educated junior I come across is basically worthless until they get 2 years of actual software engineering experience anyways, unless they are actually self-motivated, self-taught and "have it", which is rare because Uni grads tend to be the most entitled insufferable hype-powered-technology-chasers of the bunch. So the worry about co-pilot making it harder to enforce making them do useless busy work drills to meet arbitrary syllabus goals made up by a 10 years out of touch professor so that they can be given a "degree" in knowledge that becomes outdated in matter of a few years is not exactly of concern to the real world at large.

TL;DR: If co-pilot is invalidating your CS course, it's probably not worth much anyways.

[Anonymous Coward]

Re: Oh no, not the Uni grads

I think it odd that you can get into engineering or programming without presenting a portfolio or performance, like fine arts and music require.

But then again I always thought of engineering and programming as part of the Arts & Crafts more than the Sciences. Like pottery with vectors.

[Falmari]

Co-pilot is not the problem the course is

If a CS course has a problem with co-pilot then there is already a problem with the design of the CS course.

Lets take the "find the nth element of a Fibonacci series"

I did a search on that exact text and found multiple examples here is one web page https://prepinsta.com/c-program/fibonacci-series-up-to-n-using-c/. So copy one of those examples make some superficial changes (formatting, variable names) job done. If the code is going to fail a plagiarism detector then so will Co-pilot generated or a students own code as it will look the same, there are only so many ways that that code can be written.

I think Co-Pilot may not be the best way to cheat on that problem. Normally these sort of assignments are to show that you understand what you have been taught, for example Recursion. So for the Fibonacci problem the solution would have to use recursion. There is no guarantee that Co-Pilot generated code will use recursion, the webpage above has four examples one of which is recursive. The last thing you want is for Co-pilot to produce some novel solution. "Copilot is different, he said, "It actually generates novel solutions."

So if your CS course awards marks for assignments that only require code then you have the problem that a student may not have produced the code themselves. Be that web search or Co-pilot. Even before the internet if one student could produce working code for an assignment they all will. Just think about it. ;)

[Anonymous Coward]

Signalling the inevitable

Computer programming will be another of the redundant professions once AI and machine learning really get going.

Eventually it'll be seen as quaint that you once needed to learn an abstract "language" to get computers to operate in the way you wanted them to.

[AceRimmer1980]

An AI with an Uzi 9mm?

There was a documentary about that.

[Electric Panda]

Memories...

Back in my day, people were caught using jad to decompile the lecturer's supplied Java Class files (used to test our solutions) and expose the grading criteria. They weren't failed or marked down but departmental policy changed after that.

[vincent himpe]

how many times has your boss asked you

- to write a recursive Fibonacci sieve ?

- hand calculate the eigenvector of a matrix ?

- solve an integral ?

- make a perfect 1inchx1cinhx1inch cube using nothing but a hacksaw , a file and a ruler. ?

the answer : NEVER

Industry spends billions on dollar to have cutting edge tools to do that work. Draft it in solidworks, send it to CNC.

Copilot is nothing but a tool that does the tedious repetitive stuff, so you can focus on the real work. So yeah, allow it in the classroom. Teach them how to use it to their advantage.

Too much time in education is spent in memorizing stuff that can easily be looked up, and solving little meaningless puzzles. You might as well fill out the sudoku and crosswords.

Teach the APPLICATION of things and show how it is used in the real world.

Drop a bunch of abstract formula and the students can solve them. Give them a real world problem and they don't even know what formula apply.

[Electric Panda]

Re: how many times has your boss asked you

It's a common joke I've seen in the comments of YouTube videos about 'coding interviews be like' and so on.

Interview: use an online whiteboard to show us how to balance a Red-Black Splay Tree in O(n/3) amortized time. Then, assuming we have a circular buffer queue of n elements and you want to find how many times the letter "A" appears on the heap, how many apples were sold in Grantham last year?

Actual day job: Updating the test cases for org.FeckarseIndustries.Java.JavaClasses.JavaPackages.EnterpriseStuff.IEnterpriseFactory.IEnterpriseFactoryFactory.JankyRubbish.OldStuff - this was of course written in Java 5 back in 2010, long after Java 5 was EOSL.

[vincent himpe]

Re: how many times has your boss asked you

ah ,maintaining slovakias prime lens manufacturers codebase. Father Jack will drink to that !

[Anonymous Coward]

Re: how many times has your boss asked you

> the answer : NEVER

Speak for yourself.

While beyond my own skills, I've had two colleagues troubleshoot software by solving least squares and building Kálmán filters with nothing but pen and paper. A sight to behold that was and it taught me a lot too.

[Anonymous Coward]

In the HEI where I work, computer science assessment is stuffed anyway. Everything appears on Chegg or CourseHero the day its issued and so many students use the answers given in those places that the academic misconduct system has effectively collapsed. Meanwhile the course tutors flatly refuse to change their assessment methods. Not their problem.

What will be their problem is a plummet in student numbers when our CS degrees are completely discredited. Which I expect in two or three years.

[Anonymous Coward]

By using Copilot in CompSci exercises

students will have an easy ride to getting a diploma but they will be a big help in rendering their career irrelevant. The Copilot will become so proficient at coding that even an administrative assistant will be able to input specifications for the code. No longer need for a CompSci diploma.

As for the evaluations, all educators will have to do is to ask students to explain the algorithms they used and to comment the code. By doing this, even if Copilot wrote the code the student will have the opportunity to prove his knowledge.

[Anonymous Coward]

Is this a trig question?

Once upon a time professors required their students to memorize tables so that they could be ready to do linear interpolation on the fly in calc class, but then electronic pocket databases for looking up these tables got to be so ubiquitous that it did become pointless to spend more than a brief history lesson's worth of class time on it anymore.

The knowledge isn't in knowing how to press the tab/tan key it's in knowing where the starting point is, and what to do with potential results once you have them. Maybe the focus should be more on why the copilot's code runs in O(N^3), or how basic string operations these days can cascade into the system going up in a glorious virtual memory pyre when you throw a few billion objects down the pipe?