Open weights are not open source: Why AI's favorite label is under dispute

The AI industry likes to abuse the word "open." It appears in product releases, research papers, policy debates, and investor presentations. A company publishes model files to Hugging Face, developers run them on their own GPUs, and the release is quickly described as an "open source model." Not necessarily. It may only be open- …

[tiggity]

But everyone has been aware of this flaw for ages - inability to know, in detail, what a model was trained on has always been an issue (especially for those who care about copyright).

[Clausewitz4.1]

>inability to know, in detail, what a model was trained

That would drive a lot of lawsuits, no? We all know Chinese/Anthropic/OpenAI/etc.. are trained on proprietary data, even copyrighted books

[katrinab]

I would argue that the weights are the compiled binary, and the Ollama or whatever you use to interface with them is your runtime environment.

[that one in the corner]

Well, they are just a bunch of near-as-damnit-arbitrary-numbers[1] that are processed by a program to generate output: "running" them is much the same as running bytecode through a VM for Java/Lua/Python/BCPL CINTCODE/whatever language you wrote this week, so - yes.

In other words, providing them is just allowing you to run the program yourself, as opposed to paying to run it on Someone Else's Computer. Well, not quite, as you can more easily[2] "do things" to those weights, like add in your own super-secret-special-sauce training data than you can (generally) tweak a dot-pyc fioe's contents to any useful effect.

[1] nadans

[2] ok, the things the tools do are not easy, or cheap, but using those tools is doable by more people than can hack bytecodes.

[VoiceOfTruth]

The 'open source' world does not own the word 'open'

As far as I can see, there is no confusion here. Open weights are described as such. They don't claim to be open source.

If the open source world feels like building an open source LLM, all they have to do is build it.

[that one in the corner]

Re: The 'open source' world does not own the word 'open'

It isn't "the open source world", it isn't *any* "world" doing the talking.

This is just the OSI desperately trying to be relevant and continue the pretence that they are of any use or significance (as a reminder, no matter what they tell you, the OSI did not come up with the term "open source software").

As you say, "open weights" is straight forwards and obvious in its meaning: nobody (who has the faintest idea about how any of this works) is being misled by the term.

That isn't to say that there aren't other parts of an LLM that it might be good to see, even though none of those actually make the LLM itself: all the items mentioned (in particular the bulk inputs) are part of the process that generates the LLM, not the actual beast itself.

Oh, and don't get hung up on any belief that, if you had all of those inputs, you could recreate the LLM: even leaving aside the absurd amount of resources that would burn, the whole process is non-repeatable anyway (well, unless you also managed to get the identical sequence of every bit of randomness in the training process, including the precise order inputs were fed in, how much it was left to chug away until the next inputs were dumped into the hopper - or the same ones again! - let alone obvious things like the numbers returned by call to rand(). Oh, and any oddities in how the hardware is quantising everything; is your implementation precisely the same as the original? We all know of stories about differences in IEEE floats and those have had a well-propagated spec for ages.

[VoiceOfTruth]

Re: The 'open source' world does not own the word 'open'

Thank you for your considered response. I realise that I was conflating the OSI with open source in general - something no doubt the OSI would enjoy. That's my mistake.

The OSI claims for itself far too much. On its web site it states: "As steward of the Open Source Definition, we set the foundation for the open source software ecosystem." That's a bold claim. OSI was founded in 1998, long after the GPL 1 and 2, BSD, MIT, and Apache 1 licences were already a thing.

I agree that OSI is trying to stay relevant. It appears to be trying to take ownership of the word 'open' in the term 'open weights'. What next, the screenplays for Open Sesame should be open source?

[doublelayer]

Re: The 'open source' world does not own the word 'open'

They might not claim to be open source now, but that's only because when they did, most notably Facebook with its early LLAMA models, people got really angry about it. There are two other complaints you haven't considered, both of which I find somewhat plausible. One is that stuff calling itself open weights doesn't always come with the same freedoms that open source provided even in the where and how you can run it sense, but I think it is intended to suggest that it does. We don't have the ability to enforce that definition of "open", but it doesn't mean we have to like something if we think it's intentionally misleading.

But in this consideration of the OSI's definitions, we have an even simpler reason to be unhappy with them. They're trying to define "open source AI", not "open weights", and their definition of it is insufficient.

[Catkin]

I'm not sure the two marry up enough for a direct comparison. I can understand why open source advocates would view full transparency on training data as advantageous but, at the same time, one can take an unpruned open weight and refine it further.

If I release a jumbled, convoluted, completely undocumented source code, that still counts as open source. While it would be good form to document it and explain what I was thinking (/drinking) when I wrote it, I'm not technically under any obligation to do so, even if it's written in brainfuck, in order to qualify for the label.

[doublelayer]

I'm not convinced by that argument since the weights are not code by any human definition. It can be executed and wrapped by code that modifies its behavior, but people don't modify model behavior by editing weights directly. One of the key points of open source is that the binaries are not enough to qualify, even though you can do all the same things and more if you only have a binary for the software. You couldn't use the argument that you have the assembled instructions for this software, so I don't have to release the code that was compiled to generate them, and still expect to successfully call it open source.

[Catkin]

I'm not saying they're directly comparable, I'm pointing out that providing the entire process that went into writing source code is not a feature of open source either. In closed, compiled code, it's also not possible to readily take what is being run and modify it in the way one can do with an open weight. It falls, in my view, somewhere in the middle, so I view the open weight label as reasonable.

[Pete Sdev]

Precedent

The smart opensource licences stipulate that not only is the source code freely available, but also instructions and any necessary config files to enable someone to compile said source code.

Thus a opensource LLM would need to include the training data (or at least a link to it) and training instructions that would also someone to replicate the model.

[Catkin]

Re: Precedent

Compilers and config files just let you compile and run the source code, so it seems like the equivalent in a model would be something like Ollama.

[doublelayer]

Re: Precedent

Ollama would just be the runner, not the compiler, and certainly not the things needed to compile this model specifically. In fact, they probably could get away with withholding the compiler and still calling it open source because compilers under different licenses don't prevent something being open source.

[Tron]

Don't be so dismissive of 'hollow tech marketing terms'.

The industry is built on them.

[ecofeco]

Re: Don't be so dismissive of 'hollow tech marketing terms'.

Since the 1990s.

It's hollow marketing terms all the way down. To dismiss or minimize them is a literal crime. Fraud is fraud and still illegal no mater what criminals say.

[Anonymous Coward]

What's the Difference?

Training data is important and possible poisoning of models even more so. At the same time, open weights still offer something better than inference as a service when integrating AI into mission critical tasks. There is a difference between foolish and stupid.

[Groo The Wanderer - A Canuck]

The fact that you need specialized skills, tools, and technology infrastructure to make the changes to a model that you describe is all on you as the developer, not the distributor of an open source licensed product.

[Anonymous Coward]

Just to muddy the waters…

…what if I use an open weight LLM and use that to train an open source LLM :)

[Clausewitz4.1]

Re: Just to muddy the waters…

You do have a point, but more hardware will be required

[Anonymous Coward]

UIUO

unstabe input. unstable output.

[Taliesinawen]

AI inference models and training

The people who build and train these systems do not possess a complete causal/mechanistic explanation of what the trained networks are doing.

“we do not understand how our own AI creations work”

“we don’t understand how models do most of the things they do.”

“AI scientists and AI developers only have a minimal ability to explain why these models made a given decision over another one.”

--

20:06: 28092026: stuck in moderator hell :(