back to article Grok-1 chatbot model released – open source or open Pandora's box?

As promised, Elon Musk has released the model behind the xAI chatbot, Grok-1. Released under the Apache 2.0 license, the base model weights and network architecture are now available. The model has 314 billion parameters and needs hardware with enough GPU memory to run it. It is fine-tuned for applications such as natural …

  1. mostly average
    Facepalm

    No, it isn't fine tuned.

    It is the raw checkpoint. The release clearly states it is not fine tuned for any purpose. I know you all hate Musk, but at least read the first paragraph!

  2. The Oncoming Scorn Silver badge
    Coat

    Curiously Enough

    my edition of the The Hitchhiker's Guide to the Galaxy radio scripts that had the good fortune to fall through a time warp from a thousand years in the future into the clearance bin at WH Smiths & wound up in my possession around 1985 - 87 isn't dog eared at all.

    Icon - What thieving bastard has taken my towel off the towel rail?

    1. I ain't Spartacus Gold badge

      Re: Curiously Enough

      Do you mean to tell us, you don't know where your towel is?

      That is seriously unhoopy man!

  3. Badbob

    Hand me a shovel

    I’ve buried my copy of Hitchhikers in warm compost. I shall retrieve it forthwith.

  4. Anonymous Coward
    Anonymous Coward

    > needs hardware with enough GPU memory to run it.

    How much? I can't find the requirements anywhere. I'm not even sure what the article is referring to specifically.

    I don't see much use in it given we have the venerable LLaMA derivatives, but it might be fun to try out. I'm sure it can be useful somewhere.

    1. may_i

      If it has 314 billion parameters and you use 4 bit quantisation, you'll need around 150GB. With 16 bit quantisation, you're looking at something around 628GB.

      Better get some shares in nVidia along with the required number crunching hardware!

  5. aerogems Silver badge
    Boffin

    I've always kind of wondered.. if you could somehow quantitate the storage capacity of the human brain, how much would it be. Then, assuming you could ever create a truly sentient AI, or do like the show Upload and digitize your consciousness, it seems like storage would be the problem. If you could exist essentially forever, and continue racking up new experiences and knowledge, that all has to be stored somewhere. If something that is basically just repeating back to us our own questions in the form of a statement, and maybe pulling in some random shit from who knows where, nothing even remotely close to what the average human brain is capable of, and that already takes somewhere around a TB of memory... I guess I won't be living long enough to ever see a sentient AI that approaches even the level of a squirrel or cow unless someone comes up with a major breakthrough in storage density.

    1. Dostoevsky

      Memory vs. Recognition

      I think you'd be surprised at how little people really *remember.* We humans are much better at recognizing experiences as familiar than actually recalling them without prompting.

      And even given that fact, our best digital attempts still fall flat on their face for most levels of reasoning. So I agree, no General AI any time soon.

    2. katrinab Silver badge
      Meh

      The brain isn’t digital and I don’t think it stores data in anything like the same way as a computer does, so I don’t think you can make meaningful comparisons.

  6. Snowy Silver badge
    Coat

    Does?

    Having nearly 6 time the parameters make it nearly six time better, or just need a massive amount more resources?

    1. Michael Wojcik Silver badge

      Re: Does?

      It absolutely is "nearly six times better". If your metric is "how many parameters does this model have?", at least.

      The quick answer, of course, is that the question is underspecified. You can't quantitatively compare one model to another without specifying a computable quantitative metric.

      More parameters does mean certain things. If quantization is the same across two models, then the one with more parameters can capture more information entropy during training and tuning; that doesn't mean it will, but the possibility is there, because of the pigeonhole principle. If the models have similar information density then more parameters means more information. Unfortunately it's impossible to prove in the general case how close to optimum information density a model is (by algorithmic information theory), so there's a lot of handwaving here.

      More information and more degrees of freedom mean, again in principle, less tendency to overfit and more power to discriminate between adjacent vectors. That might be important in a given application; it might not be.

      More parameters gives you more margin to screw around with temperature without driving the model off the rails. It may make better use of a large context window, since it effectively gives you more possible starting positions for generation. And it has more directions for gradient descent, which potentially means more surprise, and "surprise" is just another word for "information".

      One way to look at the effect of more parameters is seeing how mechanistic interpretive techniques such as sparse autoencoders, linear probes, and AtP* perform against small models versus large ones. If the interpreters become less competent, that suggests there may be more going on "behind the scenes" — that the larger model is actually achieving additional function through scaling.

  7. Anonymous Coward
    Anonymous Coward

    I asked it when the Muskian colony on Mars would be ready

    It said: “Five to ten years, go fuck yourself”

  8. Groo The Wanderer Silver badge

    Gee, Musk has cash to throw at GenAI? Does that mean he was able to pay severance to the fired employees and rent to the stiffed landlords?

  9. Mockup1974 Bronze badge

    Always good to see this stuff open sourced. It's a hedge against companies like OpenAI turning evil or becoming a puppet of an authoritarian government.

    1. Michael Wojcik Silver badge

      It'll sure save time in the race to extinction.

  10. Neoc

    While I am a BIG fan of Douglas Adam's HHGttG, isn't "Grok" from Heinlein's "Stranger in a Strange Land"?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like