"It might seem purest hype to compare large language model AI to atomic technology"
It does, because it is. This whole article could easily have been written by chatGPT, with a prompt instructing it to compare llms to nuclear weapons. Hmmm, maybe I'll give that a try later - might be able to pitch the output to a tech site :)
As others have pointed out, these things are not going away. The genie is out of the bottle, and is not going back.
They are tools, nothing more. They haven't worked out how to make a nuclear bomb for themselves - if they can provide that information, it is because it's somewhere in their training data and could just as easily be found via a quick Google search (once you get past the zillion sponsored results for Geiger counters etc that will then stalk you around the net for weeks).
Like any tool, you need to know how to use it. In this case, that means giving it clear instructions, and being prepared to reject the result if it's full of crap. Using it as an all-knowing oracle is going to lead to tears before bedtime.
They do not need regulation, any more than development of the computer needed regulation. Given how ubiquitous they have become, how easily they can be turned to any task - including the design, construction and deployment of nuclear weapons, btw - and how they facilitate access to just about any information, I'm sure there are plenty of politicians who dearly wish they had been!
Let the technology progress at it's own pace (it will anyway). It's already useful, and may become more so. I'm somewhat sceptical of the advantages of cramming it into every nook and cranny of things like office at this point, but if it isn't helpful it won't be used and will go the way of clippy.
I don't think there's much advantage in restricting access to just the boffinry community either. Let the public play with it, as they are now. Indeed, encourage them to do so- with full disclosure that it's an experiment and might sprout utter cobblers, of course. Let them get used to it and work out for themselves how heavily they should rely on it
Then maybe we can get away from hype and get on with the important stuff until the next new shiny comes along.