DGX Spark, Nvidia’s tiniest supercomputer, tackles large models at solid speeds Nvidia bills its long-anticipated DGX Spark as the "world's smallest AI supercomputer," and, at $3,000 to $4,000 (depending on config and OEM), you might be expecting the Arm-based mini-PC to outperform its less-expensive siblings. But the machine is far from the fastest GPU in Nvidia's lineup. It's not going to beat out an …
I reserved a DGX Spark and should be getting an invitation to buy soon, but... eh
I think I'll decline the invitation for a few reasons:
> M5 Max is right around the corner with double rate FP16 and "Neural Accelerators" (Tensor Cores). The advantage this thing had over an Apple Silicon machine is almost certainly null and void now. A Mac Studio with an M5 Max is far more useful to me than this very focused device.
> The Elon PR stunt is massively off-putting, yuck.
The machine looks really nice but the Ubuntu image is riced and gross, full of bizarre bloat but at the same time, lacking the docs/manpages. If its anything like the Jetson Nano, it just needs to be completely wiped and a clean (glibc based Linux) image prepared using the raw BSP (Board Support Packages).
Due to some weirdness with the bootloader, kernel modules and user-land tools, this will take time (took me about a week to get the Nano productive). And at the end, you have something clean, functional but completely non-standard from what Nvidia's support probably expects.
This post has been deleted by its author
Very thorough and complete. It'd be great if the usual suspects would send the ElReg Review Bureau a set of their Thor, M5 Max and Strix Halo units for a further comparative analysis imho!
It looks like the 128GB of RAM and 500 TF/s of dense oomph at FP4 (1 PF/s sparse) are the key features of this very cute desk machine, about the size of a Mac Mini of some generation. The Thor might be twice as fast on models of the same size (if MEM bandwidth is ok) but it is grey-ugly by comparison. And it seems there's no real purpose running LLMs of 3 billion parameters and smaller locally on Core i3 Whiskey Lakes with no usable GPU, so this Spark's ability to fine-tune 70B models, and run 120B models, could prove useful to some folks (eg. workers who's boss insist they use AI on the job, retirees endeavoring to stay in touch with tech, etc ...). But of course, they'll need to fix the Firefox out-of-memory issue first!
This unit reminds me that AMD should have put out a ½-, ⅓-, or ¼-scale MI300A, with added FP4 support, for exactly this space. Seems to me it could be solid competition to the GB10/Spark, and maybe help hook kids in early and often, cementing their favoring of the ROCm SOCm ecosystem and suchlikes! The competition should also help make such devices more affordable, and hopefully they'd sustain solid FP64 perf so that proper grad students involved in HPC might make good use of them too!
This is a lot less exciting than it sounded on paper. It's a bit overpriced for that sort of performance...and for that reason, ah'm oot.
If it was half the price, even $2000 it would be an instant no-brainer. At $3,000, it's not even competitive...for $3,000 you could get 4x RTX 3090 ti cards and achieve better performance...you'd have less vram...you'd have 96GB instead of 128GB...but for certain types of model, I don't think that's much of a trade off...particularly for image diffusion. It's possible to get extremely high quality results out of a 30GB model...especially if you can achieve higher iterations per second...because there is a point where more iterations becomes more important than a bigger model for image diffusion for an individual user or a small team.
I might wait a year before I pick one up used, they will probably be dirt cheap in a years time.
