A closer look at Dynamo, Nvidia's 'operating system' for AI inference

Nvidia's Blackwell Ultra and upcoming Vera and Rubin CPUs and GPUs dominated the conversation at the corp's GPU Technology Conference this week. But arguably one of the most important announcements of the annual developer event wasn't a chip at all but rather a software framework called Dynamo, designed to tackle the challenges …

[steviebuk]

The label maker?

I'll get my coat.

[HuBo]

A pitch to sell more Blackwells?

Yeah, I guess Dynamo helps one scale his/her operation smoothly ... say from NVL8 FP8 Hopper to NVL72 FP4 Blackwell which should give at least an 18x perf increase (9x for the bigger NVL, 2x for FP4). The dashed-line operating points at eyeball approx 50 TPS with 1M TPS/MW for Hopper go to 200 TPS and 6M TPS/MW for Blackwell, or 24x (4x on TPS and 6x on TPS/MW) which kinda matches the 18x (i.e. perf & eff scale nicely).

Good job iiuc (I think)!

[LybsterRoy]

-- As we've previously discussed, the faster and cheaper you can turn out token after token from a model, the better the experience for users. --

I would disagree. The best experience for users is to leave the damned thing turned off.

[myhandler]

Shouldn't the headline be:

A closer look at Dynamo, Nvidia's 'operating system' for AI interference

[Irongut]

The steam engine

Created the industrial revolution. It needed neither a dynamo nor electricity and didn't lie whenever it felt like it.

[markr555]

Re: The steam engine

Yeah but the dynamo came along only 100 years later, so give him a little 'interpretational license'

[John Smith 19]

I smell hardware lockin

Let me guess that this will work "Just a little" better with their hardware than their competitors.

Oh great.

The OS Wars (GPU edition).

Yeay.