NASA to launch 247 petabytes of data into AWS – but forgot about eye-watering cloudy egress costs before lift-off NASA needs 215 more petabytes of storage by the year 2025, and expects Amazon Web Services to provide the bulk of that capacity. However, the space agency's budget didn't include download charges, an omission that has left the project and NASA's cloud strategy in peril. The data in question will come from NASA’s Earth Science …
I wonder if anybody at NASA made a comparative cost evaluation of how much it would be for NASA to upgrade its DAACs to meet the 240+ PB storage mark vs properly costing AWS to get the job done.
With the download costs added to the mix, I really wonder if it wouldn't be better to go and upgrade the DAACs.
>I've seen numerous reports which say that cloud is great for bursty/on-demand workloads, but for a constant load, on-prem usually works out cheaper.
This is largely correct. The reality is the boundary between "bursty" and "constant" is much lower than people realise. Start breaking 20-30% utilisation and you're almost certainly better doing something CAPEX-y rather than OPEX-y. Of course AWS are now attempting to serve that market with large pre-pay deals, but that's basically just outsourcing at that point.
The one true exception to this rule is raw storage. S3 is comparable to a respectable performance SAN with dedicated networking and guaranteed QoS to every host in the environment. Object storage on public cloud is overwhelmingly cheaper than pretty much any comparable alternative option, and if you don't need your data accessible then you can make it even cheaper with offline storage like iceberg.
Which identifies the other category of workload the cloud is a good fit for - anything using a small but varying subset of a large, mostly cold data archive. Scientific computing tends to meet this criteria. They're not doing daily batches, they're doing on-demand experiments on small subsets. That's a perfect fit for object storage.
Yep, if you want something 24hrs a day, then it's likely to be a magnitude more. First thought, "great, I can get rid of my techs - it'll manage itself! - and for less", which 6 months after you have, becomes "I need some techs to manage this thing", followed by 6 months later again ...... "I've been sold a pup, we need to bring back inhouse into a traditional datacentre"
I've seen this cycle in customers already. Existing CIO had things humming nicely, gets lured away by better offer. New CIO hired with a "cloud first" mentality. Dumps on-prem gear because "it's really expensive to run at $1.5m per year" and moves all data and computing to AWS. 6-9 months pass and the economics of the AWS deal become apparent and the CIO flees in disgrace. Next CIO buys new on-prem gear and gets everything humming.
Lather, rinse, repeat..
"6-9 months pass and the economics of the AWS deal become apparent and the CIO flees in disgrace"
But he got paid, got a bonus when he reduced costs and no penalties ensued when this was proven to be wrong.
In all liklihood he got a _GLOWING_ reference to get rid of him and some sucker will hire him for the cycle to repeat.
Such individuals need to be named and shamed, but they have misuse of Employment and Data Protection laws to hide their misdeeds down to a fine art
They will have a "cloud first" policy like every other muppet. It took me months of persuasion to get our strategy to be "Cloud where appropriate".
I'm sure people in high places are trying to make Amazon realise the PR opportunity from cutting NASA a break for this particular SNAFU.
Pascal, I expect you are correct. NASA is generating AND using so much data, that outsourcing the management doesn't make much sense. Perhaps having AWS as a back up could be useful for files that aren't super sensitive.
NASA would obviously be a major customer and would get priority support, but an in-house team and storage array means even more direct service. All of this information IS the value of NASA. You don't outsource your key asset.
Actually, in classical Latin, it's a fourth declension noun, not the usual second declension, so the plural would have been 'vīrūs'. But, in checking my reply, I did find that there's a Neo-Latin form which does use standard second declension inflexions outside of the nominative/accusative, so then it's 'vīra', as you said. OK, I'll allow it, with that Neo-Latin caveat.
Actually, in classical Latin, it's a fourth declension noun, not the usual second declension, so the plural would have been 'vīrūs'.
It appears not: In usage by classical writers (Vergil, Cicero, and that crowd) virus is seen to be second declension. There is no known plural form in classical writing but, because its gender is neuter, its plural -- had it one -- would be vira.
At least according to Dr. Smith's dictionary and Kennedy's primer.
Cloud-costing seems to be designed to produce the loudest exclamation of dismay from your company's CFO every month. Since download quantity cannot easily be estimated (nobody in the Financial office ever having to have thought this way before) there is a tendency to under-estimate it wildly.
"there is a tendency to under-estimate it wildly."
There is also a massive vulnerability if your competitors want to play dirty by arranging massive levels of low-level access
I'm not suggesting that would be a way of taking out dodgy marketing outfits, That might be illegal
I may well be missing something here but:
* If scientists are going to download so much data locally that went to the cloud first, then surely that data should have gone directly to a NASA data centre first anyway (either for use or for pre-processing prior to AWS ingress)
* They still have to store that data locally (and back it up since you don't want to incur the egress charges again if it gets lost), so you still need some kind of significant storage capability within NASA
* This could be mitigated by pre-processing/sifting data sets in AWS (incurring CPU/GPU compute charges of course) - assuming that the workloads don't require specialised hardware unavailable in AWS - so that there is less to download. You still have to store that reduced data set locally though
* Assuming that there is no specialised hardware required then can they sidestep this by working on the data in the cloud entirely (i.e. no need to download most of the data) using cloud compute/virtual workstations? I imagine that the CPU/GPU compute charges would be eye-watering BUT they would have burst capacity on demand (although they would soon learn about marginal costs of instances spinning all year long...)
"surely that data should have gone directly to a NASA data centre first anyway"
That's the rub. NASA didn't want to run its own data centers: it opted to upload all the stuff gradually to the cloud. According to the audit report, it didn't realize that people can't download this stuff "for free" from the cloud – someone has to pay for the bandwidth. NASA, in this case.
C.
"“Collectively, this presents potential risks that scientific data may become less available to end users if NASA imposes limitations on the amount of data egress for cost control reasons.”"
And that's why cloud deals shouldn't be decided only by freaking accountants who don't have a clue data is going both ways in whatever pattern the business needs it to.
They failed to understand that and that's what it is gonna costs NASA.
Next time, NASA, understand IT pros are useful ...
> someone has to pay for the bandwidth. NASA, in this case.
So this is partially true. AWS supports a mode called "requester pays", which you can deploy to reverse the charges. The problem is that egress charges tend to be the dominant part of the cost of large-scale S3 storage (to encourage you to keep everything in AWS), so NASA would then be externalising the cost of downloading their datasets to the institutions that want to use them. Not impossible, but those kind of costs need to be carefully managed, and it means the institutions needs to be AWS-savvy enough to engage with requester pays.
ArXiV and a couple of other places are set up like this. Downloading something small/routine? They bear the cost. Downloading the full archive? Get your wallet out.
" so NASA would then be externalising the cost of downloading their datasets to the institutions that want to use them. "
Yup. NASA tried exactly that and was told to FUCK OFF in no uncertain terms by everybody involved - up to and including threats to remove critical instruments from the missions.
Anonymous because I have to work with these people.
I have to work with NASA (amongst other space agencies) and this kind of "oversight" isn't particularly surprising.
They also take the attitude that we should bork OUR site security for THEIR convenience when it comes to data transfers, instead of actually doing some work to fix their underlaying problems - because the same admins behind this kind of story refuse to allow anyone inside NASA to sort things out - much like it was before my dealings with "Cancer Omega"(a senior NASA data admin) in the 1990s led to the discovery that hackers had broken into - and had free rein on - the same computers being used for command-and-control of Mars Pathfinder along with a slew of orbiting satellites(*) - yes, this really happened and is documented. It led to a shakeup of NASA security and data policies which apparently has been undone again...
(*) At that point they were mostly being used for IRC denial of service attacks and running bots, but those same NASA computers participated in various large-scale late 1990s attacks on commercial network service providers - in at least two cases driving those providers out of business.
If scientists are going to download so much data locally that went to the cloud first, then surely that data should have gone directly to a NASA data centre first anyway (either for use or for pre-processing prior to AWS ingress)
I was likewise confused at first. Surely the likes of NASA could just negotiate a couple of fat peering connections with AWS and bump off egress charges (in the same way that Bandwidth Alliance deals allow customers to discount egress costs from Azure if they are using something like Cloudflare, because Azure and CF peer and it costs MS nothing to send data to CF compared with sending it over transit).
I think the point is that this is the cost derived from arbitrary users - people at academic institutions around the world. I wouldn't be surprised if this is where the oversight has occurred - NASA will likely be able to get at their data in AWS for free, and it was forgotten that most of their data users are outside NASA and EDSIS will get stung for the egress to them.
I think.
"Surely the likes of NASA could just negotiate a couple of fat peering connections with AWS"
NASA has limited egress speeds from its public archives to about 100Mb/s _total_ for YEARS in order to try and discourage use - and that bandwidth includes feeds to the institutions providing the instruments producing the data.
The agenda has been going on for a long time. The people behind this have an agenda that data storage and IT in general is "not part of NASA's core business", therefore must be farmed off to market forces. They've also been starving NASA's IT budgets on the same basis.
"then surely that data should have gone directly to a NASA data centre first anyway "
Well yes ... sort of. Back in the day, that was called a ground station. It took the data, unpacked the multiplexed telemetry data into structures that were/are a bit more conventional and added appropriate metadata. Again, back in the day, that was necessary because vehicle bandwidth was very costly and therefore data streams were as compact as possible. And data rates were often very high because the link could only be up during short time windows. So the ground station equipment was specialized, expensive, and often project unique. Then the data went to a project command center for analysis before being archived.
Assuming things are still much like that, the AWS part is presumably just the archiving.
This cloud business seems very similar to the energy ponzi scheme that brought down Enron, buy cheap and sell at extortionate rates leaving people in Calefornia with large bills and power outages.
I guess with AWS, this will be high rates to get your data back and lost data because do we really think amazon will have redundancy to keep data over multiple DC securely. Just look how they treat the staff, cheap and nasty, so why do we think the staff will care much about the data they care for.
Yes, we do really think Amazon will have redundancy to keep data over multiple DC securely.
Mainly because there's plenty of evidence that they already do.
So no, no lost data. How do Amazon pay for this remarkable service? By charging for things like data transfer out of their cloud. It's almost as though it's a transparently priced self sustaining model that's been in operation for a couple of decades.
"I seem to remember some online services losing all their data on AWS and having to close up shop because they assumed cloud meant redundant data."
Even worse than that:
Some outfits thought they'd gotten redundancy by distributing their data across multiple cloud providers.
What they discovered was that the "multiple providers" were sock puppets (or resellers) of ONE service provider with all their data resting in the same place in the same cloud which went "phutt"
You're already going to be getting most of the benefits of scale associated with being "cloudy" anyway, even if you run it in house.
What's the annual storage budget for that, I bet most of use could plan some hardware (including power costs and disk replacements) that would cope for 5 years and pay a healthy salary for a (small) team to look after it.
Once you go past a couple of hundred TB (well before that figure actually), cloud becomes wildly uneconomic
We've let academic staff rant about cloud being cheaper and let them go get quotes for equivalent services from AWS, Google, etc.
EVERY SINGLE ONE who did so has come back and apologised.
The one who ignored us and pressed on regardless discovered he was paying five times as much for his data storage/access and tried to charge that back to central IT - which didn't go down well with the "powers that be" when he tried to claim nobody had told him about the extra charges and printed documents were then pulled out of filing cabinets showing the emails giving exactly those warnings.
"Oooh; look at all the savings we can make!" - With hardly a thought given to the hidden costs and downsides.
Of course, in most cases, the downsides are borne by the users so they can usually be disregarded as they rub their hands with glee.
The one with the convincing advert in the pocket.
Government procurement rules.
We developed a bunch of analysis software for a previous mission around NAG libraries because we got a government site-licence and had no budget to pay for a developer to integrate a free alternative.
But we were then forbidden from sharing any models or source code with researchers outside the boundary of the site-licence. Collaborators would have to come to us to use the data.
But somebody's budget shows a saving from buying a site-license over using free software
Watch this go bang. Once NASA factor in the ridiculous Internet pipes they'll need, with umpteen levels of resilience and bandwidth in the 10's of GB/s range, and pay AWS bazillions forever more. I can't quite believe how they think this will save them money. In all reality, NASA engineers will be twiddling their thumbs while things upload/download, or AWS goes down, or they suddenly find they're trying to access data from a much slower region. Trust me, it won't work, and NASA will pay the price. Nobody saves money moving to the cloud!
Senior people inside NASA have been issuing exactly those warnings for years - and gagged or fired for their troubles.
The people who came up with this hair brained scam get their bonuses and skedaddle before the hammer comes down. Their successors get the shower of shit and blame.
Outside of IT ( and in it too, I guess) a good rule of thumb is that using external services is the same cost as using internal ones minus infrastructure savings, but plus suppliers costs and profits.
So the only advantage is (infrastructure savings-supplier cost).
Beancounters, who should be able to create and apply formulae, seem to struggle to manage this one. Because they distinguish between capital investment and budgeted costs. And don't seem to realise that spending on the former saves money on the latter.
i.e. short term v long term thinking.
And it's not a new problem. In my sphere it's been school buildings that were cheap to put up a few years before, but then are unaffordable to maintain. Costs kicked down the road.
"In my sphere it's been school buildings that were cheap to put up a few years before, but then are unaffordable to maintain. Costs kicked down the road."
In Scotland it was school buildings put up cheaply and then had to be emergency closed as they started falling down.
But nobody prosecuted the administrators responsible for this, at most they pointed fingers at the substandard building contractors those administrators hired.
Guess who then gets to merrily skip along to another government department or organisation, rinse and repeat?
Yeah, that's along the lines I was thinking, except racks of up to 48u are available, so you can get 11 of those 4u 60 drive enclosures per rack (you could actually fit 12, but then you'd have nowhere to mount the switch). Also, Seagate are promising to release 18TB drives in the first half of this year, which ups the potential density a bit. Power-wise, you could _just_ manage with 5KW per rack constant power, but only if you carefully manage the spin-up of the drives - if they all start at once, the initial surge would draw 15KW.
"60 x 16TB HDDs"
Try 90 to 100+:
https://www.supermicro.com/en/products/chassis/4U/946/SC946ED-R2KJBOD
http://www.chenbro.com/en-global/products/RackmountChassis/4U_Chassis/RM43596
http://www.chenbro.com/en-global/products/RackmountChassis/4U_Chassis/RM43699
https://www.westerndigital.com/products/storage-platforms/ultrastar-data102-hybrid-platform
"Up to 1.4PB raw capacity in a 4U enclosure"
I've seen 120-drive prototypes too.
If you don't feed a rack like this with 10-15kW capabilities you're not going to fare well.
The 9-14W per unit IDLE power consumption of these very large drives is just that - sitting there spinning with the heads parked. As soon as you actually start seeking the power goes way higher than that. Count on a 30% increase in power draw simply by having the heads flying, let alone seeking.
Most people are familiar with this adage. However, NASA are arguably more interested in seeing things go up, rather than coming down, so it is understandable they haven't factored this into their costings.
Civilisation has seen abandoned supermarket trolleys, abandoned e-commerce trolleys, can we soon expect to hear about abandoned datasets?
The data has to be stored somewhere and wherever that is there's going to be a code. As for access to the data, on the AWS S3 pricing page (https://aws.amazon.com/s3/pricing/?nc=sn&loc=4) on the 'Data Transfer' tab is this little nugget which may be relevant:
You pay for all bandwidth into and out of Amazon S3, except for the following:
• Data transferred in from the internet.
• Data transferred out to an Amazon Elastic Compute Cloud (Amazon EC2) instance, when the instance is in the same AWS Region as the S3 bucket.
• Data transferred out to Amazon CloudFront (CloudFront).
(my emphasis) So providing 'data scientists' are working from an EC2 instance in the same region as the NASA bucket there will be no charge to NASA. Of course there will be a charge to the 'data scientist' to run the EC2 instancein the same region and to add sufficient storage but that's not the concern of NASA. In this scenario the data transfer the costs will be $zero. Of course this assumes NASA staff configure the bucket correctly and only allow access from IP addresses in the same region.
This story sounds a bit odd. I design AWS architectures for a living and whilst 250PB is a lot of data I see very little reason for large downloads that would incur these charges. The one reason to donwload the data would be to access on-premise compute resources, which would make little sense to do for organization doing research. AWS compute offering is super cost-effective and incredibly scalable if designed correctly. It would be interesting see more about their use case and the need to download so much data. For analytics, AWS supports all the major languages, ML notebooking, and more.
1) I found this case study for NASA that build this library (https://images.nasa.gov/) basesd on a architecture for delivering image content to the public. (https://aws.amazon.com/partners/success/nasa-image-library/)
This CDN implementation would incur cost with content getting stored at edge locations.
2) Open Data on AWS stored in S3 Object Store, which is are available in us-west-2
A collection of Earth science datasets maintained by NASA, including climate change projections and satellite images of the Earth's surface.
https://docs.opendata.aws/nasa-nex/readme.html
It would be interesting to dig into this one in detail and figure out how they have things configured....
225 PB of storage is not a super large amount, especially by 2025. Today individual spinning disk drives and NVMe flash drives can support sizes of 16TB for a single drive. Storage subsystems deployed in a single T42U Rack can support up to 25 PBs.. So today that could be service in ~10 racks. By 2025 much better density should be expected. NASA should store its 225 PBs on and off prem. Access to the data on prem would not need to be encumbered by the high AWS (or ASURE) charges for access by all the offsite customers. NASA should also negotiate with the cloud providers for nearly free access from the worlds scientific communities, or choose another cloud.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
