Atos, UK government reach settlement on $1 billion Met Office supercomputer dispute A court case which would have seen Atos take on the UK government over a £854 million (c $1 billion) supercomputer contract for the Meteorological Office has ended before it began. The case, Atos Services UK Ltd v Secretary of State for Business, Energy, and Industrial Strategy and The Meteorological Office, concerns an …
> an active archive system capable of storing nearly four exabytes of data.
Crikey! Just under half a day's internet traffic in 2021.
I wonder how many years worth that is when you take out advertising, streaming and pr0n?
A meteorologist once told me that statistically the best weather forecast for 'tomorrow' was....
I think that's a fairly well known story from the time of the first Met Office computer forecasting system.
When their first forecasting model was up and running, they ran a trial, pitting the new system against their existing 'traditional' forecasting methods and, as a control, threw in the assumption that "tomorrow will be much the same as today". The test results showed the computer and 'traditional' forecasting to both be correct around 50% of the time while the 'about the same as today' control case was right 66% of the time.
IOW, UK weather changes roughly every three days. I got that story from a friend whose brother was a Met Office man, but the computer models will have changed quite a lot since then.
The last comparisons I know about, made a few years ago, were that the Met Office was sufficiently far ahead of the US equivalent for the USAF to be using it rather than their equivalent and that the European Centre for Medium-Range Weather Forecasts was pretty good as well.
Here is an interesting piece on how Americans living in hurricane prone areas started using the ECMWF forecasts years ago. Ars Technica: 'The European forecast model already kicking America’s butt just Improved'.
For most of us, myself included, probably the most important thing to get out of a a weather forecast is knowing whether to take a coat because it may rain on the way back. If, however, you're in shipping or aviation you want to look a bit further ahead. If you work in agriculture knowing if it's going to rain next week can be the difference between a wasted crop and a bumper harvest. If you're in a potential path of a hurricane it can be a matter of life and death.
What I understand from analysis is that the gains in meteorological accuracy the last three decades have been huge, particularly for a few days out. And old man looking at the sky might be able to tell you it's going to rain overnight, he definitely can't tell you if it's going to rain next week.
A lot of the computing power of the ECMWF has moved from Reading in the UK to Bologna in Italy and once their new supercomputer is fully operational some of the expected improvements are in the two to four weeks range. Predicting the likelihood of a storm in a certain area four weeks before it happens! One of the ways this is achieved is a higher resolution using much smaller 'tiles' and therefore much more data to work with.
One of the ways this is achieved is a higher resolution using much smaller 'tiles' and therefore much more data to work with.
Presumably that means collecting a lot more data and so a lot more monitoring stations. Back in the 90s I did some work with the guys at Reading on the systems preparing the collected data before feeding it onto the Cray for analysis. All the other customers using these machines who were also using Crays were mad keen on extracting the last possible bits of performance out of the boxes. The ECMWF team didn't seem to interested in the performance, "Why not" I asked. "Coz this box is processing every incoming record in plenty of time and could easily cope with 5 times our most optimistic request of monitoring stations so it doesn't need to go any quicker. The analysis system is another thing."
I guess a lot of the increase in data now compared with 25 years back is the amount of satellite data available to them.
Interesting bunch, I enjoyed working with them and also the Met office guys when they were in Bracknell.
Back in the '80s, ECMWF used IBM mainframes as the systems to collect the data. In 1978, I was at a presentation where I was told that NUMAC's IBM 370-168 was the same as the system they used as a FEP to the Cray 1 at ECMWF.
For the seven years I worked at the Met Office in Exeter, they upgraded their mainframes from a sysplexed z990 to a sysplexed z196 to a cluster of two z14s.
They use mainframes as data gathering and forecast distribution systems, not because of the computing power, but because they don't crash.
During a critical power problem while I was there, we had to do a load shedding operation in order to prolong the critical environment as the UPS ran out of power (can't remember why the diesel generators wouldn't start up, but we were just on the UPS). Surprisingly, the supercomputers were the first to be shutdown, in the sequence test, collaboration, non-forecasting and then the forecasting computer (they used to keep two systems, one running the live forecasts, and the other able to pick up the live forecast, but normally running research work - it's a bit different now since they installed the current Crays). The last systems shutdown was to be the mainframe.
The reason for keeping the mainframe, storage and comms systems running was that some data cannot be gathered later if it is missed.
I found more details on the resolution improvements.
The new supercomputer capacity will enable a number of important changes for the ECMWF’s operations, including crucial progress towards its goal of improving the horizontal resolution of its forecast from 18km to 10km and increasing the number of vertical resolution layers from 91 to 137. The ECMWF also has an ambitious goal of a 5km ensemble forecast set for 2025.
ECMWF Opens Bologna Datacenter in Preparation for Atos Supercomputer
A lot of the computing power of the ECMWF has moved from Reading in the UK to Bologna in Italy
Yay! Another Brexit dividend. Still, the UK then gets to buy its own meteo supercomputer and also pay the unsuccessful bidders (and lawyers) for any procurement fur cups. Anyone who can count: How are we doing net from the pot of 350 million for the NHS, I wonder?
Perhaps the European model works OK for northern hemisphere. During recent Oz eastern coast floods the local Met office was uncertain because none of the models, USA, Oz, Japanese or European were close. Japanese and Oz ones were usually closest, but some days weather was highly unpredictable. European model was the most inaccurate ITIRC. Any improvement in code and models has to be A Good Thing.
the best weather forecast for 'tomorrow' was 'same as today'
In terms of being more right than not it feels correct and what most people anticipate. The limitation of that is when there's a drastic change coming over the horizon or when one wants to look more than a couple of days ahead.
The other rule of thumb is; if it's been gloriously hot all week it will rain at the weekend. In the UK anyway.
Unless “tomorrow” is a bank holiday, in which case, it is going to rain.
Usually it rains on the last Monday in May, because it is a bank holiday. It didn’t this year, because it wasn’t.
Usually the weather is very good at the beginning of June, and I know that to be the case because I have two family birthdays then. It wasn’t this year, because it was a bank holiday.
I interviewed Jack Scott for my school magazine in the late 1970s/early 1980s and first hand his comment was two-fold - remember what it was like yesterday, and it'll be mostly like that, or look out of the window and it'll be mostly like that. According to Jack, covers the vast majority of cases.
Farmers find it pretty helpful, especially around harvest time.
Aircraft operators find it quite useful to know when to use wing de-icers.
Armed forces think it is useful to be able to predict and use atmospheric lensing to fool the baddies into thinking gun fire is coming from a different direction.
Good weather forecasting is really quite useful.
It was "forced" to put the service out to tender.
What I've not identified the exact circumstances that "forced" the service to be put out to tender in 2014. However, I suspect there was (Conservative) government interference, given the wording of the tender announcement:
"Provision of a service and system to enable the creation of high quality weather forecasts for UK (both national and regional) and global output."
[https://www.government-online.net/bbc-weather-services-weather-production-system/ ]
I suspect the Met Office would have to partner (with MeteoGroup?) to satisfy this requirement (doubt a Conservative government would permit a publicly funded agency to fully compete with (foreign) commercial entities as that just wouldn't be cricket.
Although this raises the question as to who was shortlisted, as suspect the tender was biased towards one company...
However, as MeteoGroup are a customer of the Met Office, the BBC are effectively still using Met Office data for their UK weather forecasts.
So in English, BEIS and The Meteorological Office screwed up and more tax payers money used to pay off ATOS with a nice little earner for no creating a tender document and before any gets on the bash Boris bandwagon, this will have been done by faceless bureaucrats and civil service mandarins who can barely claim to breathe and cash their pay cheques at the same time.
Talking to some Cray engineers over the last few months and they seem to have a few systems running MS Azure Cloud SW but the instances inside the cloud are running Linux.
Didn't MS add a whole lot of code the Linux kernel a year or so ago to allow Linux to run the back end parts of Azure as well as running as nodes within the cloud (oh and doing various bits of the networking too).
As an ATOS uk customer I can tell you that they are very short of money. So I expect they settled for not that much and it was an offer too good to refuse for the met office to make them go away.
They’re constantly invoicing us for work not yet complete and we waste a load of time and money arguing so after a while people less resolute than me just cave in and cough up. Otherwise they just threaten to down tools and kill already overdue projects stone dead
Even without their proposed split they don’t seem to be long for this world given their constant cash flow issues and chronic slowness to complete billable work.
I guess the lesson they can take from this is that in future it’s better to submit feeble bids and sue for compo than to actually win and do the work.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
