The Register Home Page

back to article Tens of thousands of Nvidia GPUs sit in Meta's Prometheus supercluster, waiting for a failure

More than 11 percent of the GPUs at Meta’s Prometheus supercluster in New Albany, Ohio, are either dead or sitting idle waiting to step in when the next drops out, Zuckercorp revealed this week. The massive pool of redundant GPUs is just one of several measures the hyperscaler has taken to keep its AI training runs on track. …

  1. Anonymous Coward
    Anonymous Coward

    Why am I reading this?

    This is the sort of faffle that makes CIO types think they're relevant. And need fresh underpants. It's all very impressive, I'm sure, but most of the words are there for the impressiveness and if I wanted to be duly impressed by facebook's marketeering department's token techie writer, I'd read their blog instead. Why take the trouble to regurgitate their numbers of impressiveness (a kilowatt per accelerator, what's that, a card, a rackmount chassis, what?) and then forget to tell us what they mean, add some editorial insight, or something that makes this actually worth reading? In idiot cee-row speak, what's the value-add here? Are you fishing for taking ciodotcom's crown? Do you really want to, once you know it's made out of underpants?

    1. LogicGate Silver badge

      Re: Why am I reading this?

      Did we read the same article?

      I read: AI companies buy up all the GPUs, so that consumers can not get any, and then leave them sitting idle and depreciationg."

      At this point I went to the comments to do some venting.

      1. cyberdemon Silver badge
        Mushroom

        Re: Why am I reading this?

        What I am reading into this is that the slop-mongers are not being entirely truthful about their GPU failure rates. If they are spending say, 10 billion on a Gigawatt of GPUs, but see the need to have 11 percent ($1.1 billion worth) sitting idle as hot spares ready to pick up the load from dead GPUs, then that would appear to support something that Ed Zitron claimed almost a year ago, that the slop-slingers were cooking the books when they wrote down their GPU assets as having a six year depreciation period when that seemed rather unrealistic, especially when accounting for hardware failures, (e.g. electromigration - the slow degradation of a copper conductor that has an excessive current per cross sectional area passed through it) That allowed them to pretend that they were a better investment than they actually were.

    2. Anonymous Coward
      Anonymous Coward

      Re: Why am I reading this?

      Couldn’t not do something ‘low value/low priority ’ at Meta like learn about fake news, learn about CSAM, learn about the fraud on Facebook Marketplace…..

      So it can be eradicated, and when the resources needed pause that work so they can do work of value??

      1. VicMortimer Silver badge

        Re: Why am I reading this?

        Why would they do that?

        That's all stuff that makes them money. That's not "low value/low priority". That's the stuff they WANT.

        1. Anonymous Coward
          Anonymous Coward

          Re: Why am I reading this?

          Cheaper than Cambridge Analytica, 2026 US States settlement… I suppose.

          1. DS999 Silver badge

            Re: Why am I reading this?

            New Mexico just won a $25 billion judgment against Meta. If that was upheld (but no way the Supreme Court's bootlickers will allow billionaires to have to suffer the indignity of being held responsible for wrongdoing) then extrapolated to the whole USA that would mean a nearly $4 trillion judgment. Goodbye and good riddance, Meta! If only...

  2. Throatwarbler Mangrove Silver badge
    Terminator

    Pay to play?

    I gotta ask . . . how many of these news articles are unlabeled sponsored material? That's how this article reads to me.

  3. b1k3rdude

    I have zero sympathy and fcks to give for anything that costs facebollox serious money that that equivalent to a kick to the gonads.

  4. Sorry that handle is already taken. Silver badge
    Mushroom

    Idle GPUs...

    ...are the reason IT hardware is so fecking expensive right now. Ed Zitron's sources indicated that Microsoft and Amazon had tens of billions of dollars worth of Nvidia GPUs sitting in warehouses, because they had nothing to plug them into. All that wasted manufacturing capacity.

    I wish only the worst for everyone involved in this.

    1. MachDiamond Silver badge

      Re: Idle GPUs...

      I get the reason why they want to have the kit so when the building is ready, they aren't waiting on delivery. The counter to that is if it sits too long, it grows a green fuzz and isn't good to eat anymore. The building should be the easiest part of the project and simple to put on a calendar. I seem to recall M$ having some project management software (never used it, there's better). I've never had an issue with issuing a PO to a vendor for a future delivery. That puts delivery on them (big missed delivery clause) and also an out if the thing ordered goes out of production, gets superseded or there's a new product with better specs.

      Sure, construction schedules slip, but not by a year. There's also a cost of holding inventory for too long. There's the physical space to hold it and the capital that's tied up that could at least be in an accounted earning interest. If it wasn't purchased with investor money, but with a loan, there would be interest costs accumulating that investors should point out.

  5. Mostly Irrelevant Bronze badge

    Why does Meta think they need to burn all this money at all? Why not just wait for Anthropic or OpenAI to fail and pick them up for pennies on the dollar. I swear, Zuck used to be a much more cutthroat businessman, now he just seems to tilt at windmills. Not only that it's the same windmills everyone else has already tilted at!

    1. Anonymous Coward
      Anonymous Coward

      > Why does Meta think they need to burn all this money at all?

      - MetaVerse

      - VR Glasses

      - AI

      Get your popcorn

    2. Charlie Clark Silver badge

      Part of the strategy of cashflow rich players is to drive up the costs for those dependent upon credit, whether from investors or from the bond market.

      The real cost of these data centres won't be the chips but the power.

      1. Jellied Eel Silver badge

        Part of the strategy of cashflow rich players is to drive up the costs for those dependent upon credit, whether from investors or from the bond market.

        The real cost of these data centres won't be the chips but the power.

        Or power + credit. All to train models to do.. what? And hopefully make some money, some day. But this caught my eye-

        https://www.zerohedge.com/ai/807-billion-red-ink-and-600-billion-tab-inside-anthropics-updated-ipo-prospectus

        For retail and institutional buyers alike, the proposition could hardly be clearer. Public shareholders are being invited to absorb 5 percent upfront share dilution for a public benefit fund, assume uninsurable liability for self-preserving AI models, rely on two uncontracted customers for a fifth of top-line revenue, and backstop up to $750 billion in non-cancelable, ten-year compute leases – all so venture insiders and hyperscaler landlords can convert off-balance-sheet IOUs into public-market liquidity before the growth curve flattens out.

        With Anthropic preparing to IPO. Or take the money & run before the AI bubble bursts. Which from Anthropic's filing seems both certain, and contagious-

        Anthropic’s updated S-1 is the ultimate case study in that transition. According to the filing, the company is now on the hook for roughly $450 billion in cloud computing contracts over the next seven to ten years. Crucially, the prospectus notes that the bulk of these agreements are noncancelable and must be paid "regardless of actual usage." Those commitments include:

        $111 billion contracted with Google Cloud

        $110 billion contracted with Amazon Web Services (AWS)

        $31 billion contracted with Microsoft

        Up to $86 billion to rent Nvidia chips from smaller AI cloud providers Nscale and Lambda through 2033 and 2034, respectively

        And sitting at ground zero is Nvidia, currently churning out chips as fast as it can but whether it'll ever get paid for them seems rather uncertain. Nscale already took a hit with Microsoft pulling out from their Loughton DC and FaceMelta's insatiable appetite for other people's data. Massive capex & opex to chase the AI fairy and do.. what? Target ads at users that don't want them, and users leaving FacePalm's services because of their change to T&Cs that would allow their AIs to roam freely and slup up business & individuals private data.

        Anthropic's IPO may prove a bit of a canary for the AI bubble bursting, and then there may be a flood of Nvidia's hitting the market. Then everyone may be able to run Crysis again.

        1. Charlie Clark Silver badge

          I think the approach is a companion to that of the "network effect" so beloved of VCs: play beggar thy neighbour until the costs of doing business force the competition out of business because they have weaker balance sheets. Excess inventory can be written down if necessary – watch tax revenues go on holiday for a few years – and then prices can be increased to provide the returns they've become used to.

          But it seems no-one told the Chinese or the Indians… I'm not heavily into developing with AI but it looks like we're approaching the moment when the free, open source models are "good enough". At the same time, clever market participants are putting inference systems on a diet and looking for the cheapest provider for a particular workload. Nothing Silicon Valley hates more than a free market!

        2. Anonymous Coward
          Anonymous Coward

          > Nvidia, … but whether it'll ever get paid for them seems

          >rather uncertain

          NVidia was already paid for them. Stocks.

      2. Roland6 Silver badge

        >” Part of the strategy of cashflow rich players is to drive up the costs for those dependent upon credit, whether from investors or from the bond market.”

        “Those that depend upon credit” includes governments, such as the UKs which has a large exposure to the bond markets and is having trouble, because of the AI bubble, of raising funds (ie. Selling bonds) at the low rates of interest necessary to ensure there is some tax revenue not consumed by interest payments available to fund the sorts of things governments typically fund…

        1. Charlie Clark Silver badge

          Yeah, I'm not convinced by the argument that global demand for capital is what's pushing up bond yields. If that were the case, governments would just need to wait for the AI bubble to collapse and watch yields fall. The reaction of the bond markets to Bessent's hamfisted interventions also suggest markets are looking at more than just macro signals. Many have bought Google, et al. never-never bonds knowing that these probably won't reach term. They know which companies have the necessary cashflow to make payments and eventually redeem them, and are have started circling round those who can't, starting with the "neoclouds", because bonds buy them leverage…

          1. Roland6 Silver badge

            I think it’s not do much demand for capital, but “investor” expectation. The AI bubble and the never-never bonds have enabled “investors” to demand/expect better rates of return, even though they most probably know these bonds aren’t really worth the paper they are printed on(*).

            (*) “Worth the paper they are printed on”: The direct digital equivalents "not worth the bytes it's written in" or "not worth the pixels on the screen" don’t seem to convey the same sentiment, in this context, given we are used to seeing (in films) stacks of paper bonds - for dramatic effect.

  6. david 12 Silver badge

    "gigawatt of capacity"

    To think that we wasted all that time benchmarking processors and graphics, when the simple measure was already available.

    1. MachDiamond Silver badge

      Re: "gigawatt of capacity"

      The only metric the industry has come up with is the power input. They have no way to measure the outputs and efficiency.

      I was raised (as an engineer) better than that.

      1. Geoff Campbell
        Terminator

        Re: "gigawatt of capacity"

        Tokens per gigawatt might make some sense? Folds hardware and software efficiency into one measure, and measures the only thing the end user really cares about.

        GJC

        1. Geoff Campbell
          Terminator

          Re: "gigawatt of capacity"

          Ahem. That should of course be tokens per GWh. Deplorable laxness.

          GJC

      2. Charlie Clark Silver badge

        Re: "gigawatt of capacity"

        It's also a signal that this is the marginal price for the service they hope to provide.

    2. Korev Silver badge
      Coat

      Re: "gigawatt of capacity"

      > To think that we wasted all that time benchmarking processors and graphics, when the simple measure was already available.

      Yeah, the other measures were just flops

    3. MarkMLl

      Re: "gigawatt of capacity"

      I agree, it's like working out how many person-miles your car has done this week by the number of insects on the windscreen.

      However it's not just Meta which is guilty of this, Musk used the same metric when he announced his GigaFab.

      I'm disappointed that neither journalists nor editors are sufficiently clued up to flag this idiocy.

      1. MachDiamond Silver badge

        Re: "gigawatt of capacity"

        "I agree, it's like working out how many person-miles your car has done this week by the number of insects on the windscreen."

        To measure what? How good of a consumer you are? Something about your parenting skills? If you always send out "thank you" cards when somebody gives you a gift?

    4. Aladdin Sane Silver badge

      Re: "gigawatt of capacity"

      Up that by 21% and you'll see some serious shit.

  7. Claptrap314 Silver badge

    What exactly is new here?

    The ORIGINAL electric computers, with their tens of thousands of vacuum tubes had a failed star rate >10%. The solution? Once, running DO NOT TURN THEM OFF.

    This problem is literally as old as electric computers, and so is the solution.

    It's not my fault that the folks over in Zuckland either had to figure this out themselves and then thought they were so special that they should do a press release about it.

    Sure, after all these decades, there are changes in the application, but no one who has worked, or, as in my case, simply been in the room, with big compute jobs is going to find anything new here.

    1. Anonymous Coward
      Anonymous Coward

      Re: What exactly is new here?

      When I was in the military we had "hot standby" and "hangar queens"

      1. khjohansen
        Black Helicopters

        Re: What exactly is new here?

        309th AMARG??

    2. Christoph

      Re: What exactly is new here?

      I once heard a story about an early computer where the service guys sat playing cards *inside* the computer. When the pattern of flashing lights around them changed, they went and got a replacement valve for the failed one.

    3. lglethal Silver badge
      Joke

      Re: What exactly is new here?

      But hang on, we all know the answer to any modern computing problem is "Have you tried switching it on and off again?"

      I mean dont you watch that documentary called the IT Crowd?

      1. Claptrap314 Silver badge

        Re: What exactly is new here?

        Key term: "modern computing environment".

      2. Anonymous Coward
        Anonymous Coward

        Re: What exactly is new here?

        I do appreciate the joke, but "Have you tried switching it on and off again", wouldn't work, as it then remains in the off state.

    4. retiredFool Silver badge

      Re: What exactly is new here?

      What is a bit surprising to me is that they need 11% redundancy. In today's world of semi, that seems inordinately high. Do gpu's fail that often, or are these ai ones run so hot that they do regularly fail. I'm trying to remember the last time I had a cpu fail, and I don't think I ever had, sun's, x86, x86-64's, arms, all have been fault free. Probably used over 50 in my years computing. Not a huge number granted. At 11%, 5 would have failed on average. And some of those 50 got cycled once a day. I've got a few running that are looking at 10 years in service. Disk drives are a different story, those(plural) have failed on me.

      1. Claptrap314 Silver badge

        Re: What exactly is new here?

        As I read this press release, it sounded like they deployed "compute" a rack at a time. So we're not talking about replacing individual processors, but hundreds (or thousands) at a time. That could bring MTBF way down, but I suspect there is even more.

        I doubt I would have thought about this if not for my SRE training, but it seems quite possible to me that they are deliberately running the parts hot. The chips are doing a lot of processing, and even if the market value of a token does not pay the bills TODAY, the accountants have certainly priced an internal value of a token where they are making book. Depending on that value, and quite a few other things, one might be able to crank out extra tokens at a rate sufficient to absorb the cost of pulling a rack out, replacing it, and repairing it. They don't expect to be running these CPUs in three years, after all.

      2. andy the pessimist Bronze badge

        Re: What exactly is new here?

        For a brand new process node a failure rate of 200dppm is (grudgingly) accepted. For an automotive device on a mature process node 15-20dppm is acceptable.

        Running a device hot will accelerate the device through the bathtub. Generally 100 c is dodgy. 70-100 c should work. Above 70 its out of spec.

        No comment about software crashing

  8. sabroni Silver badge
    Happy

    more chips inevitably means more problems

    I got 99 problems but my chips aint one!

  9. Anonymous Coward
    Anonymous Coward

    Impressive Engineering to be sure with an obvious solution to faults.

    My main issue is all of these proprietary LLM's running on Gucci hardware. The Chinese come along and say we are doing similar for 10% of the investment. The Chinese are also not betting the entire economy on this. If the Chinese are 6 months behind these companies I'm happy to wait so people can eat.

    Something doesn't add up.

  10. Anonymous Coward
    Anonymous Coward

    Odd choice of name

    Prometheus stole fire from the Gods and gave it to the people. Accessible, convenient energy used for cooking, cleaning, creating real tools or things of value, even healthcare.

    This one's taking several billion watts of that energy and putting it in control of the remote, disconnected, and powerful. Seems like the opposite thing.

  11. david willis

    This big question ?

    When its all up and running will it be able to run cyberpunk 2077 at full chat ?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon