They were not the cloud you were looking for, insists Amazon Web Services in unsealed JEDI protest A heavily redacted version of Amazon Web Services' latest protest against Microsoft getting the lucrative Joint Enterprise Defense Infrastructure (JEDI) cloud computing contract has been unsealed. Unsurprisingly, AWS reckons the decision is total ███. The decade-long $10bn single-vendor contract was awarded to Microsoft in …
Well, the most obvious one would be the lack of 3 each data centres in two regions to ensure adequate resilience.
As Microsoft had 1.25 DC's in their western GovCloud region and 2.25 in the eastern GovCloud region when the contract was awarded, I'm pretty sure that's less than the required 3 per region. As far as I'm aware, Microsoft are still working to complete the final western GovCloud DC more than a year later (it was due to open H1 2021) and the second has only been up for a few months. Still, better than IBM and Oracle I guess who haven't even got around to purchasing land for their proposed
No big deal? The major savings for the contract come from consolidating around one vendor so the US1.25-$1.5bn annual projected cost in 2019-2021 for DoD cloud services that were supposed to be reduced to under $1bn/year under JEDI ($10bn was the expected 10 year lifetime cost if the all 3 renewals occur but is likely to rise as the DoD consolidates 100+ of the 225 "offical" data centres versus the estimated 2500 locations hosting multiple servers).
So awarding it to Microsoft will likely delay those savings at the likely cost of US$1bn. On top of the majority of the existing cloud migrated infrastructure sitting in AWS so that instead of tidying up the marginal DoD DC's (i.e. those where security barely scrapes above the required level and costs are stupidly high) you get to migrate all the stuff you moved in the last 3-5 years when Microsoft had no GovCloud Azure offering (they were busy getting their O365 infrastructure up to the required level given they were 5-8 years behind AWS/Google on GovCloud).
And this ignores any of the more technical "which cloud is better arguments".
This was a contract to save costs that likely doubles the cost of DoD DC's over the next 10 years which isn't easy to do - even sales people eventually have a nagging doubt that charging $1k for a cage nut with 10% annual maintenance is a little over the top...
Who benefits from the delay? Pretty much every vendor other than AWS and Microsoft as this goes through the courts. And there's an awful lot of big names in the US military industrial complex making 100's of millions in the process of "legacy" IT service contracts.
Why insist on 3 DCs for each region, and then go into vendor lock-in?
From a purely resilience perspective, it would have been more beneficial to demand inter-operability of the JEDI cloud services, and then award the contracts per region.
This would ensure
* that company X with a strong footing only on the west coast can take part in the bidding
* that the DoD can change their cloud service provider without the cost of "inventing the wheel all-over again"
* that the bidding process is less prone to corruption, since a "the winner takes the whole cake" bidding process might lead to a "use the whole arsenal including shady and illegal practices" bidding approach. If there is also the option of a partial win, the bidding may remain a bit more civilised
A cynical man might assume that "3 DCs for each region" was written into the contract specifically to make it awardable only to AWS.
A 10 year contract isn't that long, in the grand scheme of things. More important than inter-operability during those 10 years might be to make sure that at the end of the contract the data is portable to avoid lock-in around year 7 when the renewal bids go out...
> A cynical man might assume that "3 DCs for each region" was written into the contract specifically to make it awardable only to AWS.
I thought it was. The initial award was challenged by Oracle because Deap Ubhi, one of the people that worked on defining the contract, previously worked at AWS. He was then again offered a job at AWS, just as the contract was being written and awarded to... AWS.
"The initial award was challenged by Oracle because Deap Ubhi"
The initial proposals were (figures are estimates):
- AWS will charge $100m for services to help migrate out of existing DC's, discounts for existing services and will begin work on the migration within 12 months so you get savings in the first year.
- Azure will charge $100m for services to migrate your existing DC's to Azure and offer discounted rates for those services. As you are not using Azure GovCloud services because we haven't finished building it yet, there will be minimal savings until these services are fully stood up in 1-2 years.
- IBM/Oracle. The US government will pay to acquire land for the new DC's. The US government will pay for the DC's. The US government will pay to operate the DC's. At the end of the contract, the US government will hand over the facilities to the vendor and pay the vendor to decommission them although the US government may choose to extend the contracts, forfeit the decommissioning fees, pay the vendor for the land and DC build again and continue to operate the facilities at an increased fee. The facilities will be ready in 3-5 years depending how quickly the land can be acquired. There is minimal exaggeration in this summary - this was the way the DoD had acquired many of its existing facilities.
As you can see, the only reason Oracle could have lost is because of Deap Ubhi.
Why 3 DC's per region? Resilience - the DoD is trying to consolidate 220 data centres (actual data centres from 60 different vendors, not just 2-20 racks in a room with insufficient AC/power. The DoD have so many of those "computer rooms" they estimate the total as 2500+ because they can't get around them all to verify them before they relocate due to facilities closing/moving/invasions ending/etc...) Plus these are defence systems so having one or two targets seems a little short sighted. Also, 3 DC's was a minimum, not a maximum - AWS GovCloud East services (and the services provided by Google i the tri-state area) are delivered over approximately a dozen separate locations.
Is it lock-in? Yes but... The DoD has been taken for a ride by so many of its existing vendors they (the DoD) was trying to get rid of the worst of them. Building large data centres takes time (2-3 years if you have land and provisional planning permission) so giving business to vendors who are not up to the task dramatically increases cost. The previous 20 years of lax security and overcharging to bring services upto FEDRAMP standards meant the DoD was happy to ditch vendors for those that were capable.
AWS and Google worked closely with the DoD to demonstrate their government facilities were suitable for the DoD from around 2010-2016 and in the process revealed a long list of shortcomings with previous providers. AWS and Google were so good that the DoD started to use them for a lot of thigns BUT the other vendors were still getting contracts and screwing the DoD in the same ways they always had. In 2017 JEDI was announced with the intention of being awarded in 2018. Oracle/IBM challenged their exclusion for not having suitable data centres (they were going to build them
JEDI is split into 4 separate contracts over the 10 years - they can award them to other vendors if needed. Yes there will be migration costs but with the amount of consolidation they have, splitting the work over two or more providers with facilities is always possible. And its still better than the 60-odd DC providers they currently use with associated network interconnects.
In terms of corruption, existing vendors typically charge 10 times as much for "military-grade kit". AWS expected to save the DoD $100's of millions per year by allowing the DoD to stop using 20+ year old DC's that were being charged at ultra-premium rates. Even in the worst case that AWS is massively overcharging, you're still likely to reduce the overall level that existed previously.
Seriously, if you look at a cost-saving contract and one vendor will save you a bucket load of money (likely $5bn-$8bn over 10 years on an expected IT spend of $250bn in that 10 year period) AND do everything you need today AND all of the competitors are 2+ years behind AND you have so much work that you will likely award more contracts to spread it over multiple vendors IF they develop their facilities, which one would you choose?
Oh...and every month you wait to make a decision, you're continuing to pay existing vendors at non-JEDI rates while you wait for the Azure GovCloud services that aren't ready... The delays so far have likely cost the US taxpayer $500m+ in savings/legal fees/rolling over end of life contracts
Finally, for those worried about lack of diversity with AWS - AWS would have had "back office application services" while the existing contract with MS for O365 "front office services" had already been awarded. Now everything is with MS.
Does that cover all of your questions?
What an unending whinge about life not being fair that case is. Has Amazon always been "fair" to its employees, suppliers, etc? Have a nice taste of being on the other end of the stick for once. As no fan of anybody involved in this affair, I hope they all continue this muck-slinging for as long as possible in order to look as undignified and crude as possible to as many people as possible.
At least half the cost of mud slinging will be at tax pay expense. The rest will be divided between AWS investors and employees. That split may well land mostly on the investors because it is difficult to take anything more from the employees but I am sure the best minds at AWS are working on it.
Its hard to say whether this was Trumps interference on behalf of Republican party donors or a wider Republican party initiated delay for the benefit of vendors. In all likelihood there were Democratic senators that also backed this approach for their military donors but it was the Republican senators who were the public face of this.
Hypothetically, if AWS were awarded the contract in 2018 as expected, the likes of Oracle had some shiny new DC's that they would have to tell the markets were no longer in use because the DoD moved al their services out of Oracle DC's and onto AWS DC's. And if that wasn't bad enough, AWS were going to help review Oracle DB licencing and get it to a more proportionate level.
And it would also hurt all the other DoD vendors too (IBM, Lockheed Martin, Metron, General Dynamics, and Raytheon) who own existing facilities that are uncompetitive in the modern data centre world where 20 year old facilities with little maintenance and in obscure places are difficult to sell services from. At the very least, they wouldn't get the premium's the DoD had been paying.
These are questions are raised multiple times when JEDI articles are published:
- these services will be hosted in US GovCloud facilities which are separate to the vendors public cloud offerings
- JEDI is the first big "cloud" contract where there were multiple vendors tendering. Previous contracts had been awarded to Google, Microsoft and AWS but they were smaller ($100's of millions over 5-10 years versus what is expected to be a gradual ramp up to around $1.5bn/year in year 10 assuming no issues).
- for perspective, JEDI would amount to between 2.55%-10% of annual DoD IT spend over the life of the contract assuming the minimum and maximum projections are hit.
- security is tightly controlled at the facilities, including Internet access via FedRAMP requirements. Retrofitting existing vendors data centres to meet FEDRAMP uncovered a lot of significant security holes in some vendors and significant costs were incurred by the DoD to address these.
- while Google showed initial interest in JEDI, its facilities did not meet all requirements for a FedRAMP Authorized Cloud Service Provider so pulled out citing employee disapproval. Google have since obtained a lower level of FedRAMP clearance for a limited number of their products with DoD usage approved on a case by case basis versus pre-approved for Authorized Cloud Service Providers.
- there is the option to exit the JEDI contract at 3/5/8 years if another vendor is able t offer the same or better services
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
