Wanted: A handy metric for gauging if GPUs are being used optimally

GPU accelerators used in AI processing are costly items, so making sure you get the best usage out of them ought to be a priority, yet the industry lacks an effective way of measuring this, says the Uptime Institute. According to some sources, the cost of an Nvidia H100 can be anywhere from $27,000 to $40,000, while renting …

[m4r35n357]

So . . .

* It does not work.

* It is destroying the internet, artists, the environment & society.

* It is hugely expensive.

* You cannot even tell if you are getting what you pay for.

Shut up and take my money - where do I sign up? (I can't help reading that as "gouging")

[An_Old_Dog]

The New Old Thing

Utilization evaluation features could be built-in to CPUs and GPUs, and the results made available to software reading a port.

Chip manufacturers see a negative benefit to that.

IBM's OS/360 had an API for user-supplied utilization-accounting software, but that API changed frequently and capriciously.

I can't imagine why. /sarcasm

[Korev]

Re: The New Old Thing

One thing that irritates me hugely about AWS Batch is that you can't find out memory utilisation very easily so you over-provision to avoid jobs crashing and therefore spend more money...

[Korev]

How could we be sure that the GPU manufacturers can't game the system?

[boris9k3]

Monitor with a third party offering... of course it will cost you more.

Modern tech will always offer a solution to a problem you never knew you had.

[PCScreenOnly]

Tencent

Perhaps that is what Tencent have done and them made adjustments to fully utilise and not be bothered by the latest and greatest - which they cannot get

[45RPM]

Instead of handy metrics, how about architecting software properly in the first place? I’ve seen use-cases for AI which don’t need AI, at least, not as a first resort. They could, for example, use an expert system to refine the request for the AI (and possibly even provide an answer without requiring the AI, and all the compute that that entails, at all) before passing it to the AI.

But efficiency isn’t really a watchword of computing these days. For all their other benefits, modern languages (pretty much anything (not quite everything though) including and since Java) aren’t efficient. They’re safe, yes, they’re easier to develop for. But they aren’t sympathetic to the available resources of the computer.

So yes. It’s an important consideration. But it’s not the only consideration.

[phuzz]

Just look out of the window, if the seas are boiling, then you know your AI is online.

[Sorry that handle is already taken.]

I'll have a go

Is your GPU being used to train generative AI models or mine cryptocurrencies?

If yes, then it is not being used optimally.