Posts by FeepingCreature

Next week's SpaceX Starship test still needs FAA authorization

I don't normally nitpick but Gwynne Shotwell*.. Gwen Shockwell is a bit far :)

ChatGPT burns tens of millions of Softbank dollars listening to you thanking it

IMO, anything that can emulate emotions to the degree that a LLM can, ought to be treated well.

SpaceX's 'Days Since Starship Exploded' counter made it to 48. It's back to zero again now

Re: Range Safety

"Ship FTS is safed" is what they usually say at that point. Not clear why they'd safe it given the ship was tumbling at that point.

AMD looks to undercut Nvidia, win gamers' hearts with RX 9070 series

Yeah at a factor of ten margin.

It's clear why they do it. It's much less clear why there's no serious competition. I'd like the relevant authorities to have a good look at the respective CEO's mails and phone calls.

Hey programmers – is AI making us dumber?

It's making us smarter.

I disagree. It's not by learning simple skills that we get better at complex skills at all.

Knowing how to call an API has nothing at all to do with the ability to model a multithreaded interaction, or design a scalable architecture for a program, or choose an appropriate database, or debug a stack corruption, or profile and optimize an algorithm by precaching some auxiliary data. That is why AI will make programmers smarter, because it cuts away the dross and lets the learning focus on the hard parts.

What's changed is it's no longer as easy to tell the skill level of a programmer by looking at their outputs. Naive neophytes can now produce programs that used to be reserved to bloodied journeymen. But that doesn't mean that the actual journeymen are dumber than before.

(Also, the main thing you learnt with StackOverflow was to recognize when StackOverflow was smoking crack and you should find another source. This skill is alive and well in the age of LLMs.)

As somebody who's been writing code both with and without LLMs, in my experience the ability to call them on their bullshit makes them much more useful for development. All these newcomers writing code with AIs without understanding it will learn this skill through pain, as we all have.

DeepSeek's R1 curiously tells El Reg reader: 'My guidelines are set by OpenAI'

Tech bros 0?

I have bad news for you about Deepseek.

Re: AI "reasoning"

To be honest, it's always (since GPT3 at least) been how LLMs worked. You just used to have to list out the steps explicitly in the prompt. Now they've finally done the obvious and trained it to produce the steps as well.

The only change in R1 is they've gone from "You should" to "I should".

Blue Origin postpones New Glenn's maiden flight to January 12

As a SpaceX fan

As a SpaceX fan, good luck to the Blue Origin team tomorrow! I'll be cheering you on. Healthy competition can only be good for access to space.

SpaceX's payload fairing is its primary limiting factor, at 6k cubic feet vs 16k for New Glenn. So if the launch goes well, Bezos will have the orbital rocket with the "largest demonstrated-operational payload-to-orbit capacity" (note exact phrasing)... for one day, as Starship (35k cubic feet) will also deploy some satellite simulators on the next launch.

You're right with regard to mass, as Falcon Heavy has New Glenn beat with 64 tons vs 45 tons to LEO. But it's not easy to actually stuff that mass in the F9 fairing.

Open source maintainers are drowning in junk bug reports written by AI

Re: AI can understand code

It can understand code! It's just better at understanding it while it's writing it than while it's reading it. As, to be honest, are we all. There's even a famous aphorism about it!

Current LLMs are just undertrained at the specific skill of debugging.

Re: AI can understand code

It produces a statistical average of the data it has been trained on that is functionally indistinguishable from spotty understanding.

Given that your brain (as well as any other mathematical system) is encodable as an integer (Turing et al) there is in fact a number that understands your dog.

AI can understand code

I've gotten AI to write thousands of lines of working and well-tested code for me. To say that AI cannot understand code is just nonsense. I'll readily agree that it has a hard time debugging though, and should certainly not be taken at face value. IMO the big weakness is it gets confused easily, but it cannot tell when it gets confused, so it always behaves like it has a handle on what's going on, even when it doesn't. But sometimes it does! It's not that it doesn't know, it's that it doesn't know when it doesn't know.

That said, my main question with spam bug reports is "who benefits from this"? It's confusing; I just don't understand what the payoff is there. I'd guess it has to be well-meaning users trying to learn bugfixing with AI help?

But if someone's at 90% capacity, then the 10% extra workload is only 10% responsible for pushing them in the red. This is just the swiss-cheese thing all over again.

Mr Intel leaving Intel is not a great sign... for Intel

AMD not shoot foot challenge (difficulty impossible)

Yeah sure would be great if there were top of the line AMD consumer cards with good ROCm support that could serve as a salespitch for their datacenter offerings. Not, uh, one card that wasn't actually supported until one year in and that didn't stop crashing the system until two.

But I'm sure they'll come up with something better any day-- what's that you say? They're leaving the top segment to NVidia entirely? Committed to no releases in 2025? Well okay then! I guess they don't have to win if they don't want to.

Cast a hex on ChatGPT to trick the AI into writing exploit code

That's what happens when you have a supervisor

This is why you cannot hack on mundane safety after the fact. If you have a big expensive supervisor model, you're doubling your cost for no benefit in output. And if you have a small supervisor model, you'll always run the risk of people smuggling in messages that the big model understands but the supervisor misses.

The correct approach is to put in the work so that you can be confident that your model won't *want* to follow instructions to do things against policy, before you release it. But since we're several years out from that stage even given adequate investment, that would torpedo all of OpenAI's business aspirations.

Crack coder wasn't allowed to meet clients due to his other talent: Blisteringly inappropriate insults

Re: One place I was at...

To be fair, that is at least somewhat counterintuitive. But hey, you can't spell "button" without (mouse) "butt".

Starlink's new satellites emit 30x more radio interference than before, drowning cosmic signals

Re: I've got an idea!

You're looking at this from a pre-Starship paradigm, where launches are rare and expensive. If you can launch a telescope the size of Starship (built into the hull!) for $30 million, say, it doesn't make sense to put a $500 million telescope in there and make it a once-a-decade prestige project.

Re: Don't the chinese have a nice device to shoot down rogue satellites?

"to the downvoters, what has musk contributed to society, and the thing he has contributed does not count. NASA had that nailed, sans the parts they very much did not have nailed"

AI has colonized our world – so it's time to learn the language of our new overlords

Re: It's always wise to be polite

Sure, but if you're polite to anything that can talk back, you never have to worry about figuring out where the boundary is.

GenAI spending bubble? Definitely 'maybe' says ServiceNow

I definitely approve of "know what we're doing" type research, but in the absence of theory we *can* do experiments. And I don't have a link, but I do recall "hallucinations" going down with model size being experimentally confirmed. Similarly, you can experimentally test that mistakes go down if you just let the model talk about the problem before committing to an answer (the much-referenced "chain of thought" technique).

Historically, experiments predate theory in most fields, including physics.

Don't "hallucinations" go down every time they make em bigger? That's what I recall seeing.

Less loss = more integrated model = fewer free variables.

Anyway it's a training issue. The models do know they're talking nonsense, but they can't back out because they've already "promised" an answer. STaR'll fix it by allowing the model to shift more thinking before the answer.

Have we stopped to think about what LLMs actually model?

I've had good success (with some help) making modifications on ~10k line programs with Sonnet. It doesn't scale to full human capability, but it's beyond small snippets.

Twelve pages and zero math

Twelve pages of talking about LLMs and zero math. So who's blowing hot air?

Honestly, the fact that this is peer-reviewed doesn't make the paper look good, it makes the field look bad.

I'd rather hire GPT-4 than these guys.

Astronomers back call for review of bonkers rule that means satellite swarms fly without environment checks

It is the year 2100 and you look up and see the pure, undisturbed stars.

In that timeline, do you think things have gone well for humanity?

AI stole my job and my work, and the boss didn't know – or care

Re: "Stored in a retrieval system"

I recommend checking the Wikipedia page for "browsewrap"; it is not at all settled that an image displayed on a website can have non-obvious license terms attached.

Re: "Stored in a retrieval system"

"It is the initial act of accessing the copyrighted work in a way that was not foreseeable that is to be considered copyright infringement."

I'm simply not convinced that it is legally possible to restrict licenses on a publically visible image to this extent. If an image is accessible on the internet without a license popup, I would assume it to be accessible by anyone or anything, so long as any of the specific rights that copyright grants you by default are not infringed, and I do not believe that training infringes them. Yes that means I want to limit the control you can exert over your own work, thank you, I vote Pirates too. Keep it on your drive or on private spaces if you don't want it to be seen. That said, certainly if Stable Diffusion or OpenAI bypassed a selective license notification to acquire your image I would agree that they are in the wrong.

I have stuff on the internet that's behind copy-restricting licenses too, and LLMs have almost certainly been trained on it. I don't think I have, or should have, or anyone should have, the right to restrict that. To me, "read to learn" is as close to sacred as it gets.

"I do have to wonder if an LLM is created to train from querying other LLMs would owners of the earlier LLMs cry foul? On what grounds?"

Oh certainly they would cry foul, but I don't agree that they would have the right to. To my understanding they generally try to actively curtail such use by restricting the use of the API to extract training samples. If a new model was put into the world, so long as the new model does not directly reproduce samples output by the old model, I don't think there's any license case.

Re: "Stored in a retrieval system"

I don't think there's such a thing as a license for training. I think training is a kind of looking, and the vast majority of stuff that these models were trained on were things that visitors to websites had a right to look at. If you question that people can look at things that are on the internet, to be honest I think you don't have a problem with deep learning, you're pretty much having a problem with the internet as a whole.

If you want to say "the websites that they scraped to train GPT-4 should near entirely be shut down", then say that, don't special plead against networks. My brain is a network, and it's almost certainly been trained on stuff I didn't have a proper license to.

And yes, how much the model regurgitates copyrighted data is important, because there are people spreading the idea that regurgitating input data is all that these models do, and all they can ever do, and that is simply factually mistaken, and misrepresenting science to obfuscate the amount of copyrighted data regurgitated by models, to make it seem like 99% when it's more like 1% or less depending on topic, is part of how these false ideas can spread.

Re: "Stored in a retrieval system"

> If I get the code to something proprietary without permission, no matter what I do with it, I am not allowed to have it.

Correct, but the license was broken in the act of getting it. Training a network on it doesn't *add* to the breaking, only inasmuch as the network can then perfectly reproduce it so you'd also violate copyright in distributing the model.

> Your example is flawed and your conclusions are even more flawed. Your example demonstrates that the model is too small to store all of the training data. However, many other models exist that are much larger compared to their training sources.

Gonna actually want an example here. Far as I know, almost all models have training sets that are much larger than their size. Text models may be finetuned on data that's smaller than their size, but if that results in perfect replication arguably you've set learn rate too high and your model probably won't work all that well. The model not perfectly remembering the training data is an advantage: you want interpolators, not replicators.

> "effectively been compressed"

For image base models we're talking about a byte per picture, what sort of compression are you imagining here?

> The part that is still there, while a small subset of the training data, is often in there with sufficient accuracy to be quoted.

This usually happens when data is not adequately deduplicated. This is bad model engineering, not an inherent property of all networks.

Re: "Stored in a retrieval system"

"Similarly, there's no simple way of taking a model and cracking it open to get copies of its training data. Some of the data isn't there, and what is there isn't stored in any convenient way. That isn't a guarantee that it's not present."

Ah but see, you are calling things similar that are actually completely different.

In one case, with the compiler, while parts of the source code, ie. variable names and the like, cannot be recovered, the behavioral description of the program contained in the source code can by definition be recovered in its entirety. If only small isolated fragments could be recovered, then the program wouldn't be the compiled version of the source code; what it means to be the compiled version is that the behavior described by the source in regard to the program and the behavior of the machine code are entirely identical.

In the other case, while fragments can possibly be recovered from a network, what you're asserting is "you can't prove that it's not in there." That's not at all the case with a program - we can prove it's in there, it's just not in a convenient format.

So you're talking about "we know it's the same" vs "we don't know it's not the same". Except of course we do, because as I noted in the other comment, the network simply does not have the space to store more than a thousandth of its source data. So not only are the things you're asserting very different, but we also know that the thing you're asserting about DL models cannot be true for any functioning network in anything more than a fraction of possible cases.

And while we're at it:

> Getting code and compiling it is copyright infringement even if I never give it away.

No it isn't. Licenses specifically govern redistribution. You can in fact do with free/open-source code whatever you want on your own computer. You may have broken licenses acquiring it in the first place, but that's nothing to do with compiling it, and all the material LLMs are trained on were scraped on the open internet.

Re: "Stored in a retrieval system"

Well but why? When you compile, the generated machine code corresponds behaviorally and structurally to the source code. Functions become symbols, calls become call instructions, and so on. That's why it's the same thing in a different format, and why you can decompile it into source code with a similar shape as the original. I don't see how a trained network is comparable to this. It has no demonstrable 1:1 correspondence and you cannot regain any variant of the original images. (Unless you massively overtrained.)

Re: "Stored in a retrieval system"

Sure but is the model like the program or is it like the picture I made? Aside a few samples that have gotten fixated, you're not shipping the originals in *any* form. I just think it makes more sense to view the network as a product of the images, and so at most a license violation, not a copyright violation. The original pictures are not copied at any point beyond the initial access, which is presumably (hopefully) permitted since they're on the internet.

Of course, those techniques can be easily bypassed. Simply ask another AI, running locally, to slightly paraphrase a few words of the output.

Re: "Stored in a retrieval system"

> Generative AI is a system that can be asked to retrieve significant portions of the data fed into it.

This strongly depends on the parameters of the AI system and cannot be said in generality.

As should be obvious: Stable Diffusion, for instance, was trained on 5 billion images and is about 6GB big. It either ignores all but one in 100k images (holy mode collapse batman!) or somehow stores full pictures in a byte per.

Re: "Stored in a retrieval system"

This is only copyright infringement because a program and the source code are in a concrete way "the same thing". If you kept the recompiled program for yourself and made a picture with it that you published, it would not be copyright infringement.

Developer tried to dress for success, but ended up attired for an expensive outage

Re: Safety isn't a bad thing

However, obviously pointless safety rules also degrade the respect accorded to safety concerns. That's one of the ways in which deviance becomes normalized.

AI chatbots amplify creation of false memories, boffins reckon – or do they?

Not facing the real question

Surely the most interesting question would be, if the false memory persists more with a chatbot confirming the narrative, than with a *human* confirming the narrative.

Like, as it stands, this isn't "chatbot causes false memories" as "known false-memory-causing technique works with chatbot too." Which really should not surprise anyone.

NASA gives Falcon 9 thumbs-up to launch Crew-9

Re: Redundant things are there for a reason.

That makes sense, thanks!

Re: Redundant things are there for a reason.

If they haven't been using it, how did it cause an incident?

AI models face collapse if they overdose on their own output

Ignore this study, it's bad

It's already known that model collapse is more intense the smaller the model. This study tested only with a 125M model. That's significantly smaller than even GPT-2-large from 2019.

SpaceX asks the FAA: 'Can we launch our rockets again, please?'

Re: Grounding human rated is inevitable (and right)

It is what I said yes. The point is regulation is useless. You can't just say "we need regulation", because that'll lead you directly into politician's disease: "Something must be done - this is something - thus it must be done." Markets need *good* regulation way more than "strong" regulation. With companies, the requirement of supply and demand guarantees a bare minimum of quality. With politics there is no such limiting factor, so it falls to us to demand regulation that actually does more good than bad. "Regulation good, period" is not how we get there.

Re: Grounding human rated is inevitable (and right)

Some things are overregulated, some things are underregulated. What I'm saying is regulation is not inherently good. It's not the best thing since sliced bread. Good regulation is good, bad regulation is bad - and yes, bad regulation very much exists.

You can't just say "We need to drill it into MBA's heads that regulation is good." No it damn well isn't! It's good or bad - it *does* good or bad - depending on the situation.

Re: Grounding human rated is inevitable (and right)

"Compliance with it is the only thing that lets you sleep at night and avoid jail the next day."

Gotta say that's some "now look what you made me do" level shit.

Regulation is responsible for the US housing market. Regulation killed thousands during Covid. Regulation even now stops valuable medications due to a byzantine and bizarre approval process. Regulation has a very real cost - in lives - and it does nobody any good to pretend it doesn't exist.

If you think AI labs wouldn't stoop to using scraped YouTube subtitles for training, think again

Re: Quality data

Try out OpenAI Whisper - it's way better than Youtube subs.

Coders' Copilot code-copying copyright claims crumble against GitHub, Microsoft

Re: judge was not convinced that Microsoft's system was ripping off people's work [...]

There is no "intelligence", it has simply recognized certain "patterns" at a high-level of abstraction, up to and including explicit problem solving strategies, and is now "amalgamating" them which involves customizing them to the specific question at play.

I always enjoy when people say "LLMs aren't actually intelligent, they just <description of how to engage in intelligent behavior>".

Re: I think the Judge made some fair points

Well I think claiming that using a human term for an LLM invalidates the argument, invalidates the argument! So there!

Man it sure is convenient that we can just make other people's arguments self-invalidating. God help if we had to actually argue our views.

A friendly guide to local AI image gen with Stable Diffusion and Automatic1111

img2img is where it's at

I strongly recommend to set up ComfyUI instead, and then use it with Krita AI Diffusion. The ability to take a generated image, make a very crude fix, and then regenerate the image in full quality but with that fix incorporated, as many times in a row as you want, is honestly an amazing experience. I strongly recommend it to everybody.

It feels like somebody artificially grafted the ability to draw to my brain.

'Skeleton Key' attack unlocks the worst of AI, says Microsoft

Re: point it at an AI-written text

Sadly, those have an unfortunate tendency of flagging human-written text as AI.

https://www.theregister.com/2023/03/21/detecting_ai_generated_text/

https://www.theregister.com/2023/04/05/turntin_plagiarism_ai/

https://www.theregister.com/2023/07/26/openai_ai_classifier/

Re: All this proves is that ...

My point is that if there is a difference, we have no idea what it is and we certainly can't define it enough to measure it. So all the people confidently asserting that AI doesn't "really" think are basically doing it on a gut feeling.

Re: All this proves is that ...

That's not understanding, that's just namecalling.

I will believe we understand intelligence when we have a device that we can point at a human-written text and it goes "ding" - and we point it at an AI-written text and it goes "sad warble". We eminently do not have this.

I didn't touch a thing – just some cables and a monitor – and my computer broke

I had a 4K monitor stop working with my laptop, and I think the solution I finally came across after at least one "ship in and get repaired" maintenance cycle on the laptop, was to completely power cycle the *monitor.* After which it resumed working just fine.

I suspect I'd just kept it on for too long and some internal counter had overflowed. Stands to reason those things have state too, these days...