GitHub Copilot may be perfect for cheating CompSci programming exercises Microsoft's AI code-suggestion tool GitHub Copilot is showing itself to be so capable that educators may have to rethink how they teach computer science. University of Massachusetts Amherst computer science professor Emery Berger earlier this month published a blog post warning educators that "students armed with [Copilot] …
But I wonder if not grading those tests would be (part of) a solution?
There seem to be two issues at play:
* People putting themselves at a disadvantage by not learning/ practising the stuff (their problem)
* People putting others at a disadvantage by getting comparable or better scores without having put in the effort
Not grading would address the second issue.
(Edit: nah, this is a daft idea. Still, I'll leave it here for discussion)
I don't think it is a daft idea. This is a problem for introductory assignments sent out as homework where you can't add complexity because the students don't know enough yet. It's the kind of thing that we can write in two minutes today, so there's only so much you can do to prevent someone getting a simple solution. Even plagiarism without being detected would be pretty easy with this kind of question.
My answer to this problem is like yours: put out those homeworks, perhaps give students a small grade for having turned something in, and give them feedback on how well it worked, but don't grade it. People who want to learn will do them anyway and get good results. People who cheat will get only a few points. Then, give everyone a test on restricted computers or even on paper (though I don't much like coding on paper, it can be done), and grade that. Someone who learned with the homework will do it again and score well. Someone who cheated will probably look at the questions and panic because their cheating tool isn't available. This also gives any student who wants a dry run in case they don't understand a concept, so they can learn it in time. The homework can be treated as a practice with feedback, and the test can be secured from cheating more easily.
Or you'll have people with poor memories like me who's old but trying to learn the small assignments. Completes one but then for the next one still needs to refer to the last one due to already forgetting some of the formatting for some functions.
Another idea is telling the student "I want you to comment your code. Tell us what each bit is doing. Then when you've done one whole section. I'll ask you to come into my office and explain your code." Could even do it remotely if needed. Problem is, Uni classes are probably too large to do that.
I don't think that will be a problem. In most courses, the first exam is at least a month if not two into the term, so you'd have lots of practice before it gets to the test. Referring to the old homework when doing the next one wouldn't be a problem under any course conditions I've experienced. By the time you get to the test, hopefully the experience of having done what would probably be ten to twenty of those tasks has helped you to understand the structure of a chunk of code so you can reproduce it.
If it's a locked down machine you could provide the language spec and API documentation. You could perhaps print them out if it's a paper test. (Hey ring binders! Now you're talking!)
Lets face it, 50% of programming time* is googling the spec. There is just so much stuff we have to remember that none of us can hold onto it all down to the tiniest detail.
* Humorous exaggeration. Except when I get sucked down some rabbit hole.
If the task is to compute Fibonacci numbers using recursive function calls, AI can fill in the comments, too.
Ask more complex (to understand, not to implement) questions and the AI cannot fill the response anymore.
If you want to test for understanding of revisitation, ask the student to write some function that demonstrates that.
In pretty sure that AI cannot yet write code to demonstrate understanding of recursion without giving actual algorithm to implement.
Maybe the task could be "write a function that demonstrates use of recursive function to compute something else but Fibonacci numbers or factorial".
That would guide human students to look for examples for those two cases and then decide about their own function.
My teacher used to call me a keyboard junkie. He always said to write out your project on paper in steps first. However I always ended up writing my project steps in the code as comments so my code always ended up documenting itself. Kind of like
# I want to read in x,y,z variables and verify
<then I would proceed to write my code snippet>
# I want a function to log
<then I would write my logging function>
# I want a function to handle failures
<then I would write my failure function>
You get the idea. It works for me, and I can usually go back to old code and quite quickly work out what I had done and even the reason behind my thinking.
> or even on paper (though I don't much like coding on paper, it can be done)
I know it can be done, and it's great. I was only allowed to touch the computer (at my dad's office, back then computers were extremely rare not to mention expensive) after I showed my dad that I could program. We went through an introductory LISP programming book together first and I had to do all the exercises with pen and paper. I was about eight.
The world could be very different today if every child was competent at programming by age ten or twelve. But I digress.
"The world could be very different today if every child was competent at programming by age ten or twelve. But I digress."
Right, good luck getting that going. While you're at it, I have several other skills that could be handy if everyone knew it by age 12 instead of some people knowing it in adulthood. I learned it relatively early as well, but not everyone has the interest to do the work required to have that experience, and some people would prefer to spend their time learning something else by then. If you have a miraculous invention that enables everyone to know a lot more than they already do, by all means let's train everyone to write code. If not, let's not force people to do it if they don't want to and are planning to use their time doing something different.
You can make a case for a lot of things being critical knowledge that everyone should learn, but for most things, it's not true. We would probably get many benefits from teaching everyone medicine. I'm not talking first aid and biology. I'm talking about proper medical school training including basic surgery, including practical application as early residents. We also live in a society where laws and contracts are important, so let's give everyone at least a year or two worth of legal training. We live in a world of many cultures, so you'll have to be fluent in at least four languages from different language families. We don't have an extra few years to have everyone do that, so we don't. I'm guessing that, like me, you don't have a medical or legal degree.
We live in a world with a lot of computers. That doesn't mean everyone has to program them. Programming is one of the things that's easier to self-train, and I know many good self-taught programmers out there. By all means we should have resources available for those who want to learn it. I'd even be happy with a mandatory small chunk in the curriculum that introduces everyone to the idea, and they can decide from that taste whether they want to continue on to more advanced work. Training everyone to be good programmers by the age of twelve, on the other hand, is elevating what we like over a lot of more important things for little benefit.
If you don't grade anything and only have like a final exam that actually counts, some people won't learn shit.
I say that as personal experience, I had a teacher that did that, I ended quitting after six months since not only I did really bad in the "Final exam" I didn't actually learn anything.
> I say that as personal experience
That doesn't prove anything. Were the exercises corrected, with relevant comments and such, or was it fire and forget?
I did part of my studies in France and at certain points during the year we would count the homework in kilogrammes per week (no joke). It wasn't scored as such but we had to correct and comment on each other's homework so you could be failed if you didn't turn yours in. It was a character building experience that was.
"Were the exercises corrected, with relevant comments and such, or was it fire and forget?"
It was the early 2000s, the teacher didn't grade or correct the homework, just told us of the password to his website that had the correct answers. Yeah he changed the password every two weeks but that's not the point.
So not only we were supposed to do the homework, we were supposed to not cheat and to correct it ourselves.
Do note the class was only three times a week and the teacher just left after each class and he liived in another city far away and no, not making this up.
So while yes we could ask some questions if you got lost in the beginning that was it.
It was also programing and math, and we weren't told of any books we could buy and or read so for me at least it was a complete disaster.
We are talking academia, not basic school. As long as the tests are corrected and commented, there is no need for grading tests. The comments should tell the students where their deficits are, so they know what to focus on to pass the exam later. This would make cheating in tests totally pointless and failures in the tests are not a disadvantage but an asset which give students information on their learning requirements.
To be honest: I don't want people, who are not able to learn in such a way to be able to pass any exam.
Graded tests lure students into cheating, especially if those test grades also influence the final grades.
Any task solved with cheating will not yield anything useful for the student.
A Uni lecturer friend and I discussed a similar issue which was how to produce an exam that could be taken at home during the pandemic with no supervision? One option we came up with was to include a program that contained mistakes and ask the student to explain what was wrong and why.
For now, at least, Google and copilot can't do that.
The problem with the approach is that the student learns how to debug a program (not a bad thing in itself) but not how to write it in the first place.
Level the playing field in both teaching/coaching and exam situations by giving them a controlled development & test environment such as a standardised VM to work in?
No direct Internet access or remote copy/paste, and if they type in a complete program from start to finish, they're copying from somewhere.
You could record the sessions to review their technique. Though that would still need a lot of supervision/coaching.
tfewster,
Like the idea but it would be quite labourious to create and to review the sessions for every student.
Using a paper based system is not such a bad idea.
I know that it would be a disadvantage to people who work better on-screen with all the 'toys' *BUT* ultimately you are trying to teach something which they should be able to explain, in extremis, on a piece of paper/whiteboard etc.
Simply, ignore the obvious typos etc, which should not be there if you are being strict, and check if the basic approach is correct and the techniques make sense.
Problem is some of us learn by looking at others code. Currently trying to learn Python doing the free Harvard course. But struggle at times with the problem sets and makes me, at least, want to give up and move on. Looking at others code really helps when totally stuck. However I make sure I understand exactly what is happening for every line. And I've even changed the code in mine and done the solutions different ways.
My logic has never been great so when some take an input and convert to an int at the same time. I know I wouldn't of thought of that so do it the longer way of taking the input then on the next line converting it to an int. I never totally copy, I always make sure I understand what the code is doing.
I like the restricted dev environment idea! They take the tests (do the programming) on ethernet-connected uni lab computers, with no general internet access, and proctors making sure nobody's using a personal device to access the Internet (or plugging a wireless USB dongle into the lab PC they're using).
> The test consisted of an uncommented program, with lines deleted. We were supposed to "fix" it, but we weren't told the program was supposed to do!
That's the kind of tests that used to get me in trouble.
I'd probably have deleted the rest of the lines and left a comment to the effect of "your code was shite but not to worry, I got rid of the evidence".
One thing is "fix the program". That indeed should develop debug skills.
Another is to change or modify the program. Then you need to understand the current form and be able to write new code. One trick can be that most of the existing code must be reused (as part of the assignment).
> The problem with the approach is
How about giving them the program with errors as you describe and asking the student to reimplement / refactor all or parts of it according to a new specification?
For some people, most of their programming career consists of maintaining someone else's code anyway.
No, you wouldn't, at least not until you could prove they did so well. If, for example, they had you fixing bugs that were all a bit obvious, all revolved around incorrect behavior, etc, you're not competent for most programming tasks. One of the frequent requests of a programmer to modify existing code isn't to fix a bug, but have it do this stuff but faster than it's doing it now. That takes understanding of how the code interacts with the systems and how performance is affected. Many others are about adding features. Some even need functionality started from scratch. They want to test that you know how to do all of that, and if you only know how to fix someone else's bug, they might be able to find better people. If they can't even prove you know that, you're not a good candidate.
In a perfect world, people wouldn't be allowed to write their own code until they had spent at least five years fixing up other people's poorly designed, unnecessarily complicated and badly written code.
It really is the only way to learn how to write maintainable code.
Only if you have good solutions. Otherwise, it teaches people a lot of bad habits. If you learn from a lot of people who weren't able to write good code in the first place, then you'll end up basing your knowledge on what's by definition the wrong way. Yes, some of that would be remembered as the bad way to do things and you wouldn't reproduce it, but other bad practices would probably still get through.
Going way way back here, but when I went in for compsci in the 80s, they taught us with a turing complete custom language which nobody else used. Don't even remember what it was called, but it leaned heavily on Pascal with a bit of "verbosity for teaching purposes" so some COBOL thrown in.
Of course you didn't have a home setup or Internet anyway, so this comparison only goes so far.
But even if, we sure could not have used CoPilot. Unless we already knew other languages and could cross-port back and forth. Anybody able to do that is already qualified to probably teach the unit.
In my college freshman Pascal class, for a while they had us writing/testing/debugging programs written in a synthetic language called "Gear's Assembly Language." The emulator ran on our CDC 3300 (which in turn ran a nifty, developed-in-house time-sharing operating system called "OS-3").
I'd love to get hold of a copy of OS-3 and/or G.A.L.
It's not problem .. you have a chat with the student, let them know you're impressed with their solution, and ask them to explain it.
You are testing comprehension, not necessarily implementation.
So what if you are using someone/something else's code, does it answer the question that's been presented?
I don't get the problem. Having tests/exams set that require people to remember syntax and particular function calls is lazy. Testing should be about problem solving.Copilot is really just a more advanced version of code suggestion in IDEs. This is like the idea that programming on paper is a useful test, why? Testing problem solving and giving people motivation is what exams and testing should be about, whether or not I can remember the peculiar syntax of a specific language is not a real test of whether or not someone is learning. Memory tests are pretty much useless. Just my 2 cents as I am a current student after having worked in computing environments for many years.
For most courses, you're right. For introductory courses, the first step probably should be teaching the inexperienced student how to use a tool. Their solution to each problem will be built from language concepts that they have to understand, so it makes sense for the first introductory class to test their ability to use a language. If you don't understand how a loop works, your solution to a problem, if you have one at all, will be hideously unmaintainable, so it's important that everyone learn quickly what loops are and how they're used.
> it's important that everyone learn quickly what loops are and how they're used.
Coming from a functional background acquired at an early age I've always been slightly confused by loops. I only started to grasp them when we did an introductory electronics course in school. Sometimes it helps to think of programmes dealing with mutable data in terms of electronic circuits.
How about just fixing the tasks? Don't create so easy assignments that AI can directly fill the answer.
If your intent is to train future software developers, don't deny access to tools that would be used in real work. Instead, ask questions that require understanding instead of formatting known algorithm to a given simple question.
I think this is most useful in the introductory courses. When you're teaching someone how to write really basic functions because they don't understand all the concepts they'll need to implement a solution, you can't just keep adding complexity. You can find simple examples that aren't in common use yet, but eventually you'll run out of good ones. There's a reason that Fibonacci and factorial are always the first two examples of a simple recursive algorithm, because they're easy to understand and write without needing extra knowledge.
If you're making good software developers, you need more than understanding someone else's code. At some point, they'll have to write their own code and it will be in the next class when the problems are harder. They should start writing stuff now, even if the answers are kind of basic.
Spin it up and give it a try. I haven't tested it myself, but having seen a few examples, I'm pretty sure it won't be able to read a spec and spit out a program any time soon. It probably will choke pretty quickly on most assignments after introductory courses. We have to train people to write programs because this tool isn't an AI programmer with perfect knowledge, and we need to start by training them on the basics that this one can do because they need the easy building blocks to get started. If we want people to self-train the introductory steps, this is no problem. If we want to use the traditional get-taught-and-tested method, we'll need at least a few occasions where they don't have this to make sure they have a good starting position.
I'm sorry, but every Uni educated junior I come across is basically worthless until they get 2 years of actual software engineering experience anyways, unless they are actually self-motivated, self-taught and "have it", which is rare because Uni grads tend to be the most entitled insufferable hype-powered-technology-chasers of the bunch. So the worry about co-pilot making it harder to enforce making them do useless busy work drills to meet arbitrary syllabus goals made up by a 10 years out of touch professor so that they can be given a "degree" in knowledge that becomes outdated in matter of a few years is not exactly of concern to the real world at large.
TL;DR: If co-pilot is invalidating your CS course, it's probably not worth much anyways.
I think it odd that you can get into engineering or programming without presenting a portfolio or performance, like fine arts and music require.
But then again I always thought of engineering and programming as part of the Arts & Crafts more than the Sciences. Like pottery with vectors.
If a CS course has a problem with co-pilot then there is already a problem with the design of the CS course.
Lets take the "find the nth element of a Fibonacci series"
I did a search on that exact text and found multiple examples here is one web page https://prepinsta.com/c-program/fibonacci-series-up-to-n-using-c/. So copy one of those examples make some superficial changes (formatting, variable names) job done. If the code is going to fail a plagiarism detector then so will Co-pilot generated or a students own code as it will look the same, there are only so many ways that that code can be written.
I think Co-Pilot may not be the best way to cheat on that problem. Normally these sort of assignments are to show that you understand what you have been taught, for example Recursion. So for the Fibonacci problem the solution would have to use recursion. There is no guarantee that Co-Pilot generated code will use recursion, the webpage above has four examples one of which is recursive. The last thing you want is for Co-pilot to produce some novel solution. "Copilot is different, he said, "It actually generates novel solutions."
So if your CS course awards marks for assignments that only require code then you have the problem that a student may not have produced the code themselves. Be that web search or Co-pilot. Even before the internet if one student could produce working code for an assignment they all will. Just think about it. ;)
- to write a recursive Fibonacci sieve ?
- hand calculate the eigenvector of a matrix ?
- solve an integral ?
- make a perfect 1inchx1cinhx1inch cube using nothing but a hacksaw , a file and a ruler. ?
the answer : NEVER
Industry spends billions on dollar to have cutting edge tools to do that work. Draft it in solidworks, send it to CNC.
Copilot is nothing but a tool that does the tedious repetitive stuff, so you can focus on the real work. So yeah, allow it in the classroom. Teach them how to use it to their advantage.
Too much time in education is spent in memorizing stuff that can easily be looked up, and solving little meaningless puzzles. You might as well fill out the sudoku and crosswords.
Teach the APPLICATION of things and show how it is used in the real world.
Drop a bunch of abstract formula and the students can solve them. Give them a real world problem and they don't even know what formula apply.
It's a common joke I've seen in the comments of YouTube videos about 'coding interviews be like' and so on.
Interview: use an online whiteboard to show us how to balance a Red-Black Splay Tree in O(n/3) amortized time. Then, assuming we have a circular buffer queue of n elements and you want to find how many times the letter "A" appears on the heap, how many apples were sold in Grantham last year?
Actual day job: Updating the test cases for org.FeckarseIndustries.Java.JavaClasses.JavaPackages.EnterpriseStuff.IEnterpriseFactory.IEnterpriseFactoryFactory.JankyRubbish.OldStuff - this was of course written in Java 5 back in 2010, long after Java 5 was EOSL.
> the answer : NEVER
Speak for yourself.
While beyond my own skills, I've had two colleagues troubleshoot software by solving least squares and building Kálmán filters with nothing but pen and paper. A sight to behold that was and it taught me a lot too.
In the HEI where I work, computer science assessment is stuffed anyway. Everything appears on Chegg or CourseHero the day its issued and so many students use the answers given in those places that the academic misconduct system has effectively collapsed. Meanwhile the course tutors flatly refuse to change their assessment methods. Not their problem.
What will be their problem is a plummet in student numbers when our CS degrees are completely discredited. Which I expect in two or three years.
students will have an easy ride to getting a diploma but they will be a big help in rendering their career irrelevant. The Copilot will become so proficient at coding that even an administrative assistant will be able to input specifications for the code. No longer need for a CompSci diploma.
As for the evaluations, all educators will have to do is to ask students to explain the algorithms they used and to comment the code. By doing this, even if Copilot wrote the code the student will have the opportunity to prove his knowledge.
Once upon a time professors required their students to memorize tables so that they could be ready to do linear interpolation on the fly in calc class, but then electronic pocket databases for looking up these tables got to be so ubiquitous that it did become pointless to spend more than a brief history lesson's worth of class time on it anymore.
The knowledge isn't in knowing how to press the tab/tan key it's in knowing where the starting point is, and what to do with potential results once you have them. Maybe the focus should be more on why the copilot's code runs in O(N^3), or how basic string operations these days can cascade into the system going up in a glorious virtual memory pyre when you throw a few billion objects down the pipe?
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
