Gentoo and NetBSD ban 'AI' code, but Debian doesn't – yet The Debian project has decided against joining Gentoo Linux and NetBSD in rejecting program code generated with the assistance of LLM tools, such as Github's Copilot. The first FOSS OS project to ban code generated by LLM bots was Gentoo, which issued a council policy forbidding code generated with "AI" tools in mid-April. …
In a goldrush, responsibility of the participants is a future consideration, if it's a consideration at all. However, like toxic spoil, the responsibility gets foisted on everyone else. Anyone with an infringing copy is, at least in principle, legally responsible. They may in turn have a claim against whoever supplied it to them. There may ultimately be a claim that the original authors consented in some way by agreeing to some vague legal terms when they made their code accessible online. It's a long thread that may unravel for decades to come.
It does seem more than a little ironic that corporations that were recently expressing shrill fears about the "viral" nature of the GPL infecting proprietary code should so suddenly have concluded that infection by their own, more deadly, virus is suddenly desirable. But the reckoning seems to be that if you stop to reflect, someone else will have legged it with all that lovely, one-in-a-lifetime, haul of lucre that could have been yours.
"If code is inadvertently copied that contains vulnerabilities, who is responsible? The programmer who contributed the code – even if they didn't write it themselves? The original author, who never contributed that code or even knew that a bot was parroting it?"
The programmer is responsible for vulnerabilities / bugs in code they contributed. Even if they didn't write it themselves and simply copied it from somewhere else they are responsible for the quality of code they contribute.
This is how it worked at my last employer when a developer copied code that was in the code base they assumed ownership of the copied code and were responsible for fixing any vulnerabilities / bugs found in the copy. This stopped the developer passing a problem back to the original developer as 9 times out of 10 original implementation did not have the problems found in the copy.
I don't see how Gentoo or NetBSD could somehow stop including LLM generated code, dependent as they are on upstream fixes.
Unless every major distro and the kernel were to do what Sony did and preemptively threaten over 700 LLM companies, there seems no way to stop it, but no distro or even the kernel has the money to do that and the GPL probably isn't even compatible with that threat (all LLM snakeoil businesses would need to do is host copies of GPL code used in training data that people could download).
Debian's policy seems to recognise that.
It is not entirely about "stopping" the problem.
When your rules specifically state that X is not allowed, then you cannot be held liable, as organisation, if you can show a) you have the right policy and b) you actively (try to) enforce it. The liability will then fall upon the person(s) or organisation(s) who offered/added the code in question.
You are missing the point. The major part is that neither Gentoo or NetBSD organisations can be held liable because they have explicit rules. They can simply say to anyone wanting to sue them to "go to the source" and remove the problematic version. They can either fork or reimplement if they need the functionality without the allegedly tainted code.
(That is besides possible code reviews. But they are not foolproof. Better and easier to blame upstream.)
It prevents the case of "willfully blind" infringement. If you know of a possible infringement, then your policy states that you remove it, which you have to do when you are informed.
Simultaneously, your policy explicitly states that you do not want code of this LLM type, which can cause infringement. Therefore, if anything in the code should fall in the forbidden category, then "you do not know, or it would have been removed". It follows that you no longer can be guilty of willfully blind infringement. In other words, the contributor (the upstream) is infringing and you are plausibly the victim.
The party accusing you of infringement has very high bar and must proof you knew, which is very hard with the policy in place (and an enforcement record of that policy).
"Debian's policy seems to recognise that."
It also requires you to be listing the copyright license used in the package for ALL the code used. It needs to be from a confirmed source. I for one fail to see how any AI code will be able be confirmed to be by an author of that code that allows the release for use by the Debian project. Just another example of their hypocrisy, though to be fair it is hardly a rare trait in the tech or any other industry for that matter.
A policy is nice, but without enforcement it brings very little to the table.
I tend to write reports in a very structured manner, especially introductions and summaries. I can't count the amount times my university has inquired about this because some kind of genAI detector matched with 80%+ confidence...
I wonder how they are ever going to efficiently detect AI generated code given the extremely structured nature of code in general.
Is this something an experienced human would be able to detect, like many of us are able to do with bodies of text?
How are you going to include this in the review process in a reliable and efficient way?
As Gentoo themselves explain, it's not so much enforcing a rule as it is about making a statement to the community.
Undoubtedly something might slip past, but that isn't the point. The point is more of a matter of respect. The team are trusting a contributor to contribute good, well-meaning code, and if they are caught not doing that, their merge request will not be accepted. It's less about "AI is banned" and more about "we're including bad AI code along with all the backdoors and painfully obvious bugs we were already rejecting". They're asking developers to be earnest about their contributions. They also explicitly state that they don't believe AI will be banned from the project forever, just that right now, LLMs are simply too notoriously bad to be trusted.
I regularly use chatGPT-4 to generate small fragments of code - python or bash.
For simple tasks and a few dozen lines of code it's for the most part correct given a clear context.
But it's not perfect and occasionally GPT-4 will get a a stupid idea into its context and keep on repeating that.
GPT-3.5 is faster but breaks very quickly.
GPT-4o Is a hybrid of 3.5 and 4 (?) and from my first experiences it gets things wrong at about the same rate as GPT-3.5
Overall, with GPT-4 I think I am ahead. It does the small-scale execution and I do the large-scale direction. I get mostly useful code and minimal 'inventiveness' by GPT-4. I won't be using 3.5 or 4o any time soon.
So banning code 'generated by AI' is actually a NIL ban. Any code that gets put into projects has to be micro-managed anyway so there is always a human in the loop and the code is human-generated with machine assistance.
It's not a particularly good analysis. It's thoroughly sophomoric when discussing any of the philosophical questions. There's no attempt to define "intelligence", and the definition of "artificial" is contentious at best, if not outright unsupportable. Liam wastes far too many words on the sort of waffling we typically see from people who have not studied the technology in depth and haven't bothered to actually grapple with any of the larger questions.
I'm dismissive of gen-AI applications myself, and I don't find the transformer architecture elegant or pleasing. But dismissing it as "not intelligent" (much less "not artificial", which frankly is a fairly inane argument) is meaningless without offering a definition of intelligence — and that definition had better be well-founded and sufficiently precise to admit to some sort of usable decision procedure — and rather misses the point.
And the term "artificial general intelligence" long predates the introduction of the transformer model (Vaswani et al, "Attention is All You Need", 2017). See for example Wang et al, Theoretical Foundations of Artificial General Intelligence, 2012.
"And the term "artificial general intelligence" long predates the introduction of the transformer model"
See also Mark Gubrud's 1997 paper titled "Nanotechnology and International Security"
https://web.archive.org/web/20110529215447/http://www.foresight.org/Conferences/MNT05/Papers/Gubrud/
Note that as far back as the 1960s we were talking about the difference between machine learning and so-called "cognitive" machines and synthetic intelligence. And of course the concept goes back much further in Science Fiction (or Speculative Fiction, if you prefer).
During the meanwhile, what we have now is nothing more than ML, or as some researchers prefer to call it "Weak AI". It doesn't matter how many megawatts of electricity, how many acres of computers or how many thousands of hours of powerpoint slides full of hype you throw at it in marketing meetings, it's still nothing more than weak AI, at best. The basics haven't changed appreciably since the 1960s.
Yep. Intelligence is nowhere to be found, apart from in the head of the programmer, not the program.
*cough* Eliza *cough*
This was supposed to be a general intelligence program. It still fools lots of people today.
What a good article this is. AI is neither artificial nor intelligent.
(Just like, on a tangent, the S in IoT stands for Security. :-)
T1: ¬ artificial ∧ ¬ intelligent ≡ ¬ hing_useful
T2: ∀X, programming_language(X) ∧ (interpreted(X) ∨ JIT-compiled(X)) ⇒ egregiously_inefficient(X)
(but damn you for the "actual game" link ... just spent an hour going to level7, and stuck at Bonus! Grrr! eh-eh-eh! Most fun since last Saturday!)
then there is the risk of license violations
There, fixed that for you. Pretty much every bit of public code that is neither abandoned nor released into the "public domain" (in quotes as various jurisdictions have differing definitions of this) comes with some sort of licence, and pretty often a term in the licence states that the licence header must be retained (so everybody knows how it is licenced). This is particularly important in the case of GPL that doesn't play nice with anything else. By reading this code, tossing away the licence, potentially jumbling up bits of code under different licences, and then regurgitating it as an answer, it's not a risk of a violation, it's a straight up licence violation.
And no, it isn't sufficient to say "but the licence doesn't forbid it" because this is something new and until recently was pretty much unthinkable; but this doesn't really matter as not only may it potentially fall into the scope of derivative works, but as said above, licences have language about retaining the copyright line and header - even the dead simple 2 line bsd says Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Failing to respect that means the licence isn't valid which means you can't use the code covered by that licence - that's how copyright works, it's a set of conditions which grant the right to a copy.
And, sure, you could argue whether taking a chunk of code is the same as taking the entire thing. I would argue yes, based upon the fact that people have received copyright strikes for including a few seconds of identifiable music in a video. Why should it be different for one but not the other?
The primary difference here is that when it comes to media, there are big multinationals with scary legal teams ensuring that their rights are protected, to the point of lobbying lawmakers to make rules that are incompatible with common sense. With open source, there isn't really any protection (GPL violations are rife, and that's one of the more restricted open licences) and instead it's the multinationals doing the thieving, and using their scary legal teams to find creative ways to justify helping themselves to content that they know will be stripped of context and spat out devoid of any credit to the person that took the time to create the content in the first place. Plus lobbying lawmakers to allow this mass theft using vague threats like "if we don't do this then China will and they'll have a technological advantage over us".
To my mind, it's not so different to borrowing a dozen CDs from the library, copying a single song from each, then making and selling a new CD of the compilation and calling it "my mix tape". How well do you think that would work out?
“To my mind, it's not so different to borrowing a dozen CDs from the library, copying a single song from each, then making and selling a new CD of the compilation and calling it "my mix tape". How well do you think that would work out?” - In that particular case, very badly indeed. You will get sued, successfully, because that’s specifically covered by the master recording copyright that attaches to performed works.
But that is absolutely not what LLMs are doing. They are producing strings of tokens where each token is *statistically likely* to follow on from the previous. Note *tokens*. It’s a basic of copyright law that “you can’t copyright an idea”. Individual sentences are in general not copyrightable, and courts have upheld that statement hundreds of times. To claim copyright on something as short as a written sentence you need to specifically show that the sentence is unique, recognisable and creative. “ It was a bright cold day in April, and the clocks were striking thirteen” is probably copyright. But “It was a bright cold day in April, and the clocks were striking twelve” would *not* be, even if another author had used exactly that phrase in a copyrighted book. By definition, an LLM is using the most likely completion word, ie the average verdict of several different sources for the next word. If there’s just a single recognisable source, the LLM isn’t going to choose that token unless you’ve explicitly told it to, ie “write a dystopian novel in the style of George Orwell”. Therefore, LLMs really *can’t* be violating copyright unless it is reproducing more than a couple of paragraphs identically. You may like the law to be different, but that is the law for the written word as it is.
In the past, the courts have ruled even whole repeated paragraphs word-for-word as not-copyright- violation, because they’ve been too commonplace and non-distinctive. For code, you are going to need to call an expert witness who will say, with a straight face, “yes I read the class and Method “AddColourToBlock”, and I instantly recognised it as Heyrick’s because their style is so distinctive”. Really? Who are you going to call who will say that in court? It’s a ridiculously high bar of proof. But it’s the same bar of proof as applies to books. The class and method being only objectively the same, character-by-character, is not enough to win a copyright claim.
Secondly. The law that applies to performed music is different to text. It’s actually different sections of the statute book, which seem similar on the surface being called “copyright”, but there are some subtly different concepts. Important one: “what is a derivative work”. For music, derivative means “what is the causal link”. If I use the phrase “We all live in a Purple Brigantine”, and I have heard the song “Yellow Submarine”, that is derivative. But if I can somehow prove that I personally never heard the Beatles song, then it is *not* derivative however similar the words are. If the copyright holder can prove that I am very familiar with the song, then I can get caught even when the words are quite different. That’s why music copyright generates so many court cases, because it’s all arguable about intention and mental state not objective fact.
Whereas for the written word, derivative means that the words have to match. Doesn’t matter what the causal derivation is. For the previous example, it’s not derivative work copyright violation, 100%, and no argument about it. Why? Because “we all live in a” is a common phrase, not copyrightable, while the distinctive part was “Yellow Submarine”, and I didn’t use those words. Now if instead I use the exact same words in an advertising jingle, even with a *different tune*, it becomes a performed work, and the Music Copyright law is what determines it.
People’s expectations have been shaped by Music Copyright law…..but that’s not the law that covers code copyright.
>> To my mind, it's not so different to borrowing a dozen CDs from the library, copying a single song from each, then making and selling a new CD of the compilation and calling it "my mix tape". How well do you think that would work out?
> In that particular case, very badly indeed. You will get sued, successfully, because that’s specifically covered by the master recording copyright that attaches to performed works.
Just to put the cat among the pigeons, dirty the waters and Cloud The Minds Of Men:
There are lots of audio recordings that are out of copyright (all of the applicable copyrights) and can be quite legally copied into your "my mix tape". Your only problem would be to make sure that it is, indeed, your mix - because you would not own the copyright to the tracks but you *would* copyright to the compilation (caveat: maybe not even that in some jurisdictions). But if your mix ended up the same compilation as someone elses...
Not that that would stop the music industry from going after you if your mix tape hit the big time.
Copyright in music has even more moving parts than other committed-to-paper works: the original question about "my mix tape", as interpreted in my answer above, is all about the copyright in a specific performance. Your discussion of copyright in music was about the actual music itself - the score and libretto - and distinguishing copies of those, which is a whole other kettle of fish (right up to the "performed work", when the static written parts collide with the artists' interpretations).
And all of which really helps to confuse the question of copyright in software, because we have such an awful mess, whenever we look to other areas that have copyright law, that we can not point to "*Here* is THE distinguishing characteristic that describes how to tell what a copyright infringement is", therefore applied to *this* software...". Without that, we can all have many happy (!?) hours arguing here about what this means for LLMs and have absolutely no effect on the final, irrecoverable result of this whole debacle: some lawyers are going to get rich.
One of the tricks of the AI flingers -"They are producing strings of tokens where each token is *statistically likely* to follow on from the previous". It does not matter how you encode the text, if the system can reproduce the source text in some form, it has "stored" it.
@heyrick "And, sure, you could argue whether taking a chunk of code is the same as taking the entire thing. I would argue yes"
No matter how small and generic the chunk of code is? Of course not. Would you argue yes if it was :-
A single line of code to opened a file. or a small function serializing out to file. How about a file declaring APIs?
There are limits to copyright., Copy a function from a GPL project that implements a Binary Search Tree and rename variables and function. License violation? Of course not, sure the logic would be the same, but the logics no different to any other Binary Search Tree.
Gentoo and NetBSD banning AI code makes sense there are enough examples of co pilot producing copyright code, that using co pilot it is more the case of when there is a license violation.
"No matter how small and generic the chunk of code is?"
Personally I think copyright is broken and not really suitable for modern uses of content, but that's an argument for another day.
I was drawing a distinction between those who want to claim rights over a phrase in a song (or quoting a line from a book elsewhere) and those wanting to take lines from source code. It has been pointed out that music (etc) has special copyright rules. Why? Human created content is human created content, should it matter if it is incomprehensible to the man on the street? I'm sure many men on many streets might say the same about Sunn O))).
Yes, I'm quite aware that some functions really have only a few ways they can be written. But, then, there are only so many useful chord progressions...
What's good for one copyright protected entity should be good for another, otherwise, what's the point?
And how exactly can this policy be (independently) verified or policed?
If a contributor passes off a piece of code as their own original work, how can anyone determine if that was true, or not?
Worse is if a rival makes a charge, then is there any way to judge the merit of that claim.
I can see that making a policy like this might provide the distros with some degree of indemnification, but I doubt that would stand up if a case was ever brought against them
The LWN article that was mentioned and linked to in the OA provides some context to the Gentoo project's discussions. The following direct quote (enbolding mine) suggests that enforcement and detection were not the main focus of the policy statement.
"Sam James agreed with the proposal but worried that it was ""slightly performative [...] given that we can't really enforce it"." Górny wrote that it was unlikely that the project could detect these contributions, or that it would want to actively pursue finding them. The point, he said, is to make a statement that they are undesirable."
When I pointed out this obvious drawback then, a lot of people didn't like the implication
This is no different than the existing policy of "must be your own work"
I'm not allowed to copypaste someone else's code and tell Gentoo or BSD - or my employer - that it's my own work.
If I do so and Gentoo notices, I'll be told to remove it or add the relevant attribution notice.
But will Gentoo notice? Probably not.
If someone spots my passing off much later, Gentoo will go back and remove the offending code, or add the relevant attribution.
Using an LLM is simply another way to copy someone else's work without attribution even being possible.
Thank you Liam for a great analysis and explanation, and it more or less confirms what I already suspected.
Extra kudos too for banning the data collection robots at your place that idiots and sheeple willingly install in their bedrooms and sitting rooms. They are banished at my establishment too :)
For you =>
> Large language models are, by nature, orders of magnitude bigger than that, and they are not human-readable code. They are vast tables of billions or trillions of numerical values, calculated by huge numbers of machines. They cannot be checked or verified or tweaked: it would take cities full of people working for millennia to read them, let alone understand and amend them.
Most articles, including those from The Register, keep usin the word "parameters" to describe all of those numerical values. As nicely described by this article, they are not now, never have been and (barring a miracle of computer science) never will be "parameters"!
A parameter is something you feed into a function to generate a known effect: box(x, y, height, width) has four parameters and - here is the important bit - *you*, the human, the programmer, KNOW what they mean! You can set them - and if you don't get the (usually blindingly) obvious results then you can declare that fact as having found a bug[1].
You sometimes come across a "parameter" that isn't well-described, may not even be labelled and then you are either in an "artistic" situation[2] or are considering revenge on the idiot who created this. Consider the annoyance at coming across "I passed in (q + 17.63) here 'cos that made it work, don't know why"!
But when you are faced with billions of values, where you can not say *why* they are set at that value[3], how they may differ in meaning or effect from any of the other values in the array, not even the one in an adjacent cell, then they are "understood". Unless you make an extraordinary effort - and get very lucky - you can not even say "look, I can't put it into words, but if you add one to this specific value you can see the effect on the output" ![4]
Do not use the word "parameter" to describe these values.
The numbers inside an LLM are, at best, "weights", as in a "weighted graph". But even that starts to lose meaning when you can't pretend to comprehend what the resulting graph represents. They are, individually, little more than near as damnit arbitrary numbers.
[1] ok, the bug *can* be in either the code itself *or* in the documentation, the thing that provided you with the knowledge in the first place. For example, you may be upset that making 'y' larger moved the rectangle in "the wrong direction", but the error may not be in the code but in the docs, which forgot to remind you this is a screen-graphics function, not a proper maths function, so y points down, not up. Or it is even weirder and you are looking at an (x, z) projection so y is ignored - until you switch on perspective. But all of those interpretations are still explainable, understandable and decently parametric.
[2] e.g. plenty of knobs on synths, especially circuit-bent ones, are very unhelpfully labelled and you aren't even supposed to be able predict what they'll do, just twiddle, have fun ad maybe get lucky with the next Top Ten hit. But even then, *someone* exists who actually knows what is going on: "Of course changing that resistor makes it sound like that, I spent days getting rid of that cross-coupling through that power line!" Or even "bloody singers, if they bothered to read the manual for Autotune it is *obvious* that setting those values will turn this $2000 precise audio analysis and resynthesis tool into a $15 over-clipped crappy diode ring modulator! Why look so surprised?!"
[3] logically, you could, of course, log whenever the value was changed and derive some chain from that, but it would itself be so ludicrously huge, dwarving the existing table of numbers that it could never be done. A shame, as then we could start to work on figuring out what percentage of this bit of output we can attribute to Fred's writings, what percentage came from Jim's copyright work and pay them accordingly.
[4] yes, there was the report where a group found where an LLM "stored the word 'Paris'" and by changing that they affected all the outputs to now say things like "the Eiffel Tower in Margate" - except they put in lots of efffort, figured out *one* number (and not, say, all of the entries in this block or row), they didn't report this gave them a mechanism to find, say, Moscow, London and other capital cities (or anything similar) so basically got lucky. Now, about the remaining billions of "parameters"...
[Author here]
Oh, you noticed that? What an unaccountable coincidence that was, eh?
;-)
They _do_ have uses. Not good ones in this jaded old hack's opinion, but they exist.
I have marvelled for years now over many people's utter inability to simply drive a search engine. This is an answer.
My own octogenarian mum can't use Siri or Google Assistant on her iPads. She gets stage fright, basically: she says the invocation keyword, then she freezes. Entering into a conversation where the device gradually draws out of her what she wants and needs would be a huge help.
Circa *1965* Joseph Weizenbaum's `Eliza` proved that people would enter into conversations with computers if they talked back, and were easily fooled that this was a person chatting with them.
Custom tutorials for complex programs or commands. When people can't be bothered to learn, or find it hard, or won't need it again.
It remains an open question whether intelligent autocomplete in program code will prove useful enough to overcome its own issues -- lack of understanding, erroneous results that people lack the skills to notice, etc. But it probably will. Personally I had encountered people who didn't have the basic numeracy to spot that their pocket calculator had given a wildly wrong answer because they missed a decimal point, or pressed some other wrong button. This is nothing new.
The first accounts of how reading and writing will destroy people's ability to remember stuff are several millennia old. Every new tool has a price. It doesn't stop it being adopted.
Cars exacted a terrible cost on society. Our roads are lethal traps and I am currently training a tiny human to avoid them and cross them with great care. They are a cause of the obesity plague: convenience is literally killing us in the millions, and the fumes from all combustion-powered engines will likely kill most of humanity within the century.
This post has been deleted by its author
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
