Atos will be paid $29m over $1b UK Met Office supercomputer dispute

The UK government agreed to pay Atos £24 million ($29 million) in an out-of-court settlement following a challenge to its decision to award an £854 million ($1 billion) Met Office supercomputer contract to Microsoft. The the Department for Business Energy and Industrial Strategy (BEIS) and the Met Office entered into a …

[hoola]

This is Public Sector Procurement

Having been involved in procurement as part of the public sector everything is a merry-go-round of arse covering.

Raynor ranting that the Government is not looking after public money is a minor piece of detail.

Everything in creating tenders is about contracts and legal teams protecting the organisation against unsuccessful responses suing them. Whether the successful bid actually delivers what is required is of lower priority.

What is equally poor is these companies believe that it is acceptable to sue because they were not accepted. There are two sides to the problem and in my view it is the attitude that these providers of services and solutions believe that they can sue just because it is public sector.

Given the absolutely abysmal services and solutions they do provide yet they do everything possible to one can only feel that a reset in the entire system is needed. What goes on in public sector procurement is not tolerated in the private sector because it is so inefficient.

[Anonymous Coward]

Re: This is Public Sector Procurement

Can confirm and probably from an entirely different sector and part of the country than you.

Irony is we have specialists, teams of them involved in procurements which can take years to complete, requirements, bidding etc all run very professionally but there is ultimately nobody "in charge" who would ever be held accountable should it not be successful and by the time it is unsuccessful - they've left, retired etc.

[Anonymous Coward]

Re: This is Public Sector Procurement

When working for the NHS, we put a tender out for a new system. One of the requirements was context awareness as part of interoperability - it needed to talk to other systems and retain context when moving between systems.

One vender offered their solution which failed entirely to meet core criteria including interoperability - it was a silo system with no data connectors available. For whatever reason. Their offering was rejected. They kicked off a fuss and sure enough, threatened legal action if we did not include their non-compliant offering in the second round of testing. So... they were included, and they failed to make the grade. So they threatened legal action once more.

They got told to jog off. Their system met 20% of the requirements, and failed to meet the key elements entirely, but we'd let the clinicians take a look and they had flatly rejected it.

Some companies believe in vexatious litigation to bully companies into paying them money. Seems the Government still hasn't learned this (or to play fair in how they award contracts)

[Anonymous Coward]

Re: This is Public Sector Procurement

I’m experiencing it now. It’s a joke. The test/survey/bid document questions are skewed to ensure one supplier remains favourite throughout the process; fundamentally, the winner has been decided before the bidding begins. And yes, carefully there is no single person to whom decisions can be attributed so no deputy heads ever roll.

[Anonymous Coward]

Re: This is Public Sector Procurement

Yep, seen that so many times before. The equipment is chosen and then the requirements are written around that specific device so it precludes other vendors.

Then they they go through the tender stage, which is simply ticking the boxes. They often already know who they are going to buy from. The requirements are not questioned because how can they be. There will not be clinically trained individuals in procurement with sufficient knowledge to question the requirements, they just have to accept the requirements.

Not every purchase is like that but probably many more than you might imagine.

In most cases you just have to accept it. I suppose it's called building a relationship with the customer before they buy the equipment or already having your foot in the door. The big companies are good at the wineing and dining....

Yes, it's the NHS I'm talking about.

[Roland6]

Buy a supercomputer from Microsoft?

Didn't know MS had any experience in high-performance computing. Suspect someone has been misled into believing cloud (and thus Azure) is equivalent to a supercomputer...

[R Soul]

Re: Buy a supercomputer from Microsoft?

And you thought Atos did?

[Anonymous Coward]

Re: Buy a supercomputer from Microsoft?

Atos is literally one of the main providers of Supercomptuters in the world…so yes

[Anonymous Coward]

Re: Buy a supercomputer from Microsoft?

The HPC isn't being supplied by Microsoft, it's actually Cray machines in a Microsoft datacentre. One of the main reasons for not going for another self-hosted supercomputer at the Met Office was the power requirement on already stretched local infrastructure.

Please do not assume that the local expertise is stupid enough to think that Cloud == Supercomputer.

Whilst Azure is being used, it's only being used for the close coupled post-processing and the archiving on model runs used for research into model improvements and running climate simulations.

AC for what should be bleeding obvious reasons.

[Anonymous Coward]

Re: Buy a supercomputer from Microsoft?

One of the main reasons for not going for another self-hosted supercomputer at the Met Office was the power requirement on already stretched local infrastructure.

And exactly how many MWs are required? 10, 40?

Paying somebody else is not going to be cheaper than doing it in-house despite the infrastructure costs.

I suspect you have the wrong people in charge at the MetOffice with insufficient knowledge that just like an easy life and like to play the blame shifting game. Contracting out everything costs the tax payer a lot of money but hey at least we have other people to blame for the screwups and I can sit behind my desk in the knowledge somebody else is doing my job and taking the risk.

[Anonymous Coward]

Re: Buy a supercomputer from Microsoft?

You seem to think the technical teams in government support outsourcing. I've worked with central and local government and the NHS over 30 years, most teams would love the option of running systems in-house but they are not given the option, internal technical teams end up managing security and systems integration across a huge number of externally managed services. Believe me the frustration you feel when a supplier repeatedly makes basic mistakes ignoring your advice gets to be soul destroying.

[Jellied Eel]

More questions than answers.

The linked pdf makes for some interesting reading. It's mostly around questions for expert witnesses, and one of the core claims was around 'architectural equivalence'. Some of that seemed to revolve around CPUs. So the production system using Intel's Saphire Rapids, aka Gen4 Xeon and dev/test systems using Ice Lake Gen3 Xeons. I'm no supercomputer expert*, but there seems to be a lot of differences in the microarchitectures between platforms.

Strangely, the doc seemed to rule out some questions on architectural equivalence saying those would rely on opinions from expert witnesses. I'm not a lawyer either, so don't understand the subtle distinction between experts offering opinions vs statemenst of fact. Having done a bunch of government bids in the past though, I probably would have questioned the bid question to get clarification. If I'd been evaluating the bid though, I probably would have scored it low based on the difference in architectures. I guess the devil would be in the detail around how exactly Atos answered that question. But to me, it seems a bit like saying production would be on Gen4 Xeons, the rest on some old 486DX50s, and that's fine because they're both x86 compatible. Shame it didn't go to trial because it would have been interesting to see the filings, assuming they were published.

*But my newest funbox does run Crysis..

[Anonymous Coward]

Re: More questions than answers.

"The linked pdf makes for some interesting reading. It's mostly around questions for expert witnesses, and one of the core claims was around 'architectural equivalence'."

Someone putting "architectural equivalence" in the requirements is screaming out loud to the tenders: "we can choose or dismiss you based on any hidden politics and nothing factual".

That's probably why they had to give money to Atos.

Whenever I have that kind of requirements in an RFP, I'm thinking of running away ...

[Jellied Eel]

Re: More questions than answers.

Whenever I have that kind of requirements in an RFP, I'm thinking of running away ...

Often the best thing to do with public sector bids anyway. If the Met Office had scored them with a couple of points, I wonder if Atos would still have sued. The doc also had questions about compilers, licences and moving code between architectures. If they're different microarchitectures, that would seem more challenging than if all environments used the same. Seems like Atos successfully litigated their bid costs, and being neither the Met Office nor the government's money, both seemed fine to waste cash to make it go away.

[hoola]

Re: More questions than answers.

In HPC the CPU, architecture, links memory etc are all connected and part of the solution. You don't usually design supercomputers around chips and chipsets that are at the wrong end of the lifecycle.

[Jellied Eel]

Re: More questions than answers.

You don't usually design supercomputers around chips and chipsets that are at the wrong end of the lifecycle.

That's what I don't really get. The Judge seemed to exclude some of the architecural differences, which doesn't make sense to me. There looks to be quite a few differences between Gen3 and 4 Xeons proposed for the test and live systems. So I'd have binned the Atos response as well as I'd want my test environment to be a cut-down, but architecturally equivalent verson of the live system. Especially given the economic importance of the Met Office's forecast products.

[Roo]

Re: More questions than answers.

Getting two Xeons of the same stepping and firmware performing identically is a challenge in itself. That said I wouldn't necessarily bin a bid on the basis of using two different generations of Xeon, in practice supply can be constrained, I very much doubt any supplier can guarantee you a ready supply of replacement motherboards/CPUs with identical firmware and steppings of all the major components a couple of years down the line. In practice, in a distributed system of any non-trivial size, you will have to swallow differences in the nodes, and that will likely manifest itself before many systems of size are even fully commissioned given how long it can take to get all your code and data transferred & validated. BTW: this has been the case for at least 30 years, it's not a new thing, time for folks to grow a pair and get used to it.

[Roland6]

Re: More questions than answers.

>You don't usually design supercomputers around chips and chipsets that are at the wrong end of the lifecycle.

Which raises the question as to why the supercomputer designers are sticking with Intel and not switching to the new AMD architecture which is delivering a much higher core count etc..

[Anonymous Coward]

Forecasting

So gone are the days when a barometer, a look out of the window and a wet finger, were all that was needed to decide whether to take a macintosh (raincoat)

[Ian Mason]

Re: Forecasting

You should always take a macintosh because the alternative would be PC (Pissed on Copiously).

[Claverhouse]

Arcades Ambo

No matter how vile both Microsoft and Atos may be, I can never understand how people can sue for not being chosen, any more than suing because an umpire gave one out. Just as when Amazon sued the Trump administration for contracting with Microsoft for some Space nonsense; and then whined that Trump was prejudiced against the creepy Bezos.

Both companies could crash and self-destruct for all I care, but it's unsporting not to acknowledge it's for the buyer to choose who to do business with.

.

And if the corrupt people's assemblies have passed procurement regulations and constitutions that enable this stupidity, then they should be abolished.

[ragnar]

Re: Arcades Ambo

"I can never understand how people can sue for not being chosen"

They're suing because they believe the government broke the law and that they have enough chance of proving it in court that they can get a payout by threatening to do so.

[LybsterRoy]

Re: Arcades Ambo

All you need is this bit "they can get a payout by threatening" the rest is superfluous

[Jellied Eel]

Re: Arcades Ambo

And if the corrupt people's assemblies have passed procurement regulations and constitutions that enable this stupidity, then they should be abolished.

Those do actually make sense. So procurement rules theoretically make the process fair and transparent. Anyone who is capable of supplying X can tender to supply X instead of the deal being done between public sector and one of their mates. There was, and probably still is a lot of corruption. It can also add a lot of cost both to bidders and the procurers given consultancies often insert themselves into the process to 'manage the bid' and collect fees. Scoring can also create problems, ie "You mus supply a fully compliant response", even when RFPs sometimes asked for the impossible. That creates additional pressure for DA (Design Authorities) from bidders, because there's usually also pressure from sales to win the bid. They stand to get commission, even if the company will end up losing big.

[gandalfcn]

Brexit is obviously the cause.

[Anonymous Coward]

atos

We use an Atos system and they have not covered themselves in glory supporting it. To the extent we would all very much prefer them to skip the next procurement. Unfortunately our Gov procurement rules explicitly exclude past performance, market reputation, and staff preferences, so they may well get some more millions putting in a system then breaking it for us.

We also do the primary system + architecturely similar test platform thang, and also would not accept a different cpu generation on the test platform. Seems cut and dried, and should really have had its day in court.

Atos, the Oracle of the HPC world!

[Jellied Eel]

Re: atos

Unfortunately our Gov procurement rules explicitly exclude past performance, market reputation, and staff preferences, so they may well get some more millions putting in a system then breaking it for us.

I really don't get why that policy isn't changed, especially as post-Brexit, HMG has more flexibility when it comes to procurement. It seems crazy that you can't exclude bidders with a long track record of spectacular, and publicly embarassing screw-ups. I know from speaking to some public sector folks, it's a very frustrating policy. They can be pretty confident they'll get a great looking bid response, but know the implementation won't be as slick.

[dubious]

Re: atos

Not to forget Atos lobbied to get UK firms excluded from post Brexit EU HPC contracts which should automatically put them on the UK Gov's shit list.

[Roo]

Re: atos

AFAICT the government's shitlist comprises of everyone who doesn't donate to the Conservative party.