A lot of datacenter networks are run by absolute clowns. Not Amazon's

Last month I sat in a networking lab with AWS VP of Global Network Engineering Matt Rehder and cracked a joke about cage nuts cutting your hands to ribbons. I wasn't prepared for the blank stare I got in response, but I really should have been. Mind you, not because the joke wasn't funny (my jokes are hilarious), but rather …

[Tom Womack]

I mean, they've had their big data centre outage, about six months ago. The strategy to protect data centres from ballistic missile attack by putting them in strong American allies which also host massive military bases and trusting the US Army to provide enough Patriots didn't quite work.

[Jou (Mxyzptlk)]

> American allies which also host massive military bases

Germany has a remark for you... Allow us to be offended as easily as the US, as country, seems to feel privileged to be offended.

https://worldbeyondwar.org/military-empires?operatingcountry=unitedstates&hostcountry=germany

Highest number of US bases in a country outside US, with 118, is Germany. World War II reminder. Just like those over 1000 unexploded WW2 bombs we find each year, 80 years after. Same for the density: Number one. Next in line is, I think, South Korea 75 US bases.

Not that there were bad choices with several other countries, most notably former Persia, where the US had done and still do everything possible to keep them as enemies. Starting with kicking/killing the emperor, which allowed way too much freedom and openness in his late years, and establish an "Islamic Republic" under Ruhollah Mostafavi Khomeini. Reason: Oil. Mohammad Reza Pahlavi was not without problems, but at least he modernized Persia a lot, over and over again.

[sedregj]

"Germany has a remark for you"

I grew up in West Germany as an Army brat (the other army - UK). Oh and Daddy was an ATO who (amongst other jobs) cut his teeth finding and sorting out German freefall munitions in Scotland.

Your views on the times post WWII and now may not match mine.

My family still keep in touch with several German families - life long friends. One of those, the man was ex Luftwaffe (post war). "the sound of freedom" was the general quote whenever a fighter plane or flight flew over and there were a lot. At another, earlier posting, the depot caretaker etc was an ex Uhlan (Prussian cavalry, WWI) Herr Masker was a lovely bloke who let me and my brother ride his ponies and anyone who dared to try and ride "Max" who was a stallion that was quite murderous! Herr M was captured by the British in the '14-'18 show, escaped and walked back home and went back to the front to face modern machine guns and tanks whilst mounted on a horse (the tactics did change and I have no idea why or how he survived hell ... twice).

I could go on at length. Whilst (nominally) the Brits and Yanks were an occupying force in West Germany (East Germany did not have a very pleasant time and I did visit via Check Point Charlie - we were allowed a day), things are just not so simple. I also remember how the West Germans looked forwards to unification before it happened - it always seemed to them to be inevitable, which it turns out was a given. I do know how hard that unification really was.

I'm sure we don't need to go over why the Brits, Yanks and that were hanging around West Germany in the first place.

[Jou (Mxyzptlk)]

You assumed one bit wrong here: Every US I ever meet/met here in Germany, most of them affiliated or in the military, were/are great people. Even tourists, which are easy to hear since USA-ians groups talk much louder than typical Germans. All of them very positive and open minded people. Two apartments in this house here are rented to them, great tenants which expose all the the good sides of Americans.

But there one or two filters I get:

1. Military stationed in Germany, especially non-GIs. Military training changes the mindset, for example soldiers DO get deescalation and behaviour training compared to US-cops, and not just everyone gets stationed here. You don't send H. fans here.

2. Being outside the US bubble for a week or better a month twists most of their views around in a shaker, but actually getting outside USA (and not just Canada/Mexico) shows the difference of what they learned from the news and what they see in reality. Especially when they have an accident and don't have to be afraid to get an ambulance due to the non-costs. Or that a bang from a fire cracker does not set us in "shooting, duck and cover!" mode.

I did not meet many Brits here though in Southern Germany, but was in Brit a few times before Brexit.

Kudos to your Dad, who had so sort the fallout from German bombs. Oh, and I finished season two of "Dads Army", including the three reenacted / remade lost episodes. And it shows that they really wanted to reenact as good as possible.

[Bebu sa Ware]

Why don't they…

When you search for SIDR not much that is relevant pops up but a talk from Callahan in 2023 (Seems Javascript has to be enabled.)

I don't know why they don't write this stuff up and publish it like the chaps at Bell Labs did 50 years ago. I can read considerably faster than these chaps can waffle on and is much easier to skip the padding in text and "cut to the chase."

Routing - from the end system perspective - in a highly connected network with typically very short paths between the nodes is interesting even in abstract.

More the concerns you might find in SAN design than conventional networking but presumably the two have converged within AWS.

[Not Yb]

Re: Why don't they…

"I don't know why they don't write this stuff up and publish it like the chaps at Bell Labs did 50 years ago."

They clearly feel that keeping this stuff relatively unknown brings competitive advantage.

[SVD_NL]

Re: Why don't they…

I don't get it either. On the other end of the spectrum, there's Cloudflare. They make a blog post about every fart released by their engineering team, and i love reading up on those. It's great marketing for their products too!

[coredump]

cage nuts

"They politely chuckled at my joke about cage nuts once they got the reference"

I shouldn't let this bug me, but it kinda does. I mean, *somebody* is still installing their servers in racks, and doing the (often OCD) cabling job. Labels/tags, inventory, etc.

Just because AWS folks have (I'm guessing) outsourced it or some similar move, doesn't mean the work happens by magic or telekinesis. Or AI android bots. Though I imagine somebody is working on that....

[Quinnypig]

Re: cage nuts

Author here. Interestingly their gear all appeared to be on quickrails. So there are no cage nuts left in their setup. I feel old.

[Roland6]

Re: cage nuts

> Interestingly their gear all appeared to be on quickrails.

Assume these are click on/slot in and so aren’t themselves bolted in place…

I’ve used quick rails in server racks, but not that often, in data comms racks. Although I hate how vendors (eg. Dell) make their quick rails server model/generation specific - hence I’ve got a box of old quick rails removed when servers got upgraded…

[Lusty]

Re: cage nuts

I always clipped the rails back on to the server in case it was reused.

[coredump]

Re: cage nuts

Sure, I dealt with quick rails towards the end as well. Most of them were pretty nice, but you can ding up your hands and fingers on those too. :-)

Not really the point, though. AWS's cloud architects may not be touching the metal, but somebody is doing it for them. Not quite "clouds all the way down", eh.

[cmac]

Re: cage nuts

Arista have been doing tool-less rack rails for over 10 years now

[Lusty]

Re: cage nuts

Bugs me too for similar reasons.

That, and I still have two of the free cagenut tools NetApp used to give away with every device. Nobody sensible ever bled from these things!

[coredump]

Re: cage nuts

I loved those tools back then too.

I watched other folks try to finagle and wrestle cage nuts with various things like flat screwdrivers and PCI slot covers, which work in a pinch. Nothing worked as well as those flat little pieces of spring metal.

[Jou (Mxyzptlk)]

Re: cage nuts

You know, for some LACK-rack is king!

What, you don'tr believe me? Proof!

[John Miles]

Re: cage nuts

Often used for 3d printers - lack 3d printer enclosure

[tin 2]

Re: cage nuts

Bugs me too, cos even if everything is on quickrails, it's not THAT long ago driving a cagenut up a fingernail was a regular occurrence in the physical domain of networking. To not be aware of it at all, but be amazing at networking cos of all this experience... I can't reconcile the two.

[ecofeco]

I've worked at a server factory

Yes, one that builds the servers and ships premade rack. Many moon ago.

The sheer volume we shipped each week was mind boggling. My estimate is 1200 servers per day. 5 days a week. Every week of the year. Minimum.

And this was, god, what, 18 years ago?

[Anonymous Coward]

Re: I've worked at a server factory

When I were a lad around 30 years ago I was building 1,2 and 4U servers. Not as many as that to be fair. I do remember a few big jobs and having to build massive (at the time) raid arrays and set up parallel computing systems. Also doing on site repairs (hardware swapping really, no point trying to troubleshoot, just swap it out like for like. In case of catastrophic failure it would come back to us) and whatnot going to Telehouse in London and one in Nottingham. Them racks were fiddly bastards though on most occasions I would find the server on the floor ready for me to sort as they didn't want me dicking about with the infrastructure that had other peoples servers in them which is fair enough.

[Anonymous Coward]

If you are putting your data on somebody else's server...

You are either the victim of a scam or the perpetrator.

Pretending Amazon is somehow BETTER than the rest of the scammers? Absolutely CRIMINAL.

You want your servers to be more cost effective?

MAKE THEM YOUR SERVERS. NOT SOMEBODY ELSE'S.

[Anonymous Coward]

Re: If you are putting your data on somebody else's server...

I don't disagree with your sentiment about cost effectiveness but that's taking it a bit too far. "Cloud for everything" is not for me either, but what makes it a scam?

Personally, I wouldn't choose to put data belonging to myself or my clients on a server that I have no genuine control over, that could theoretically be raided by another nation state government in the name of 'national security' and that ultimately costs me more than a self-owned, -located and -managed server.

I do speak with companies who have different priorities to mine though. They can't justify the cost of getting someone really good to manage their infra as they don't need a lot, but they do need to be able to offer guarantees of resiliency and physical security to their clients on what they do host, which are included (on paper at least) when they go with a big cloud provider. The cost per server is outrageous in my view, but it's actually the best option FOR THEM when looked at holistically as making a difficult problem someone else's responsibility in a way that even the client accepts is worth its weight in gold for how they operate. Maybe that changes as the company matures and scales, but everyone has their own growth and operating strategies.

I personally don't like hyperscalers or what they represent but economies of scale are a thing and clearly there is a lot of good engineering that sits behind it. I think it would be crazy to act as though there is nothing that I could learn from them when they operate orders of magnitude larger than anything I need to plan for.

Their service isn't something that I would choose based on my requirements but what makes that a "scam" or "CRIMINAL"?

[Heya]

Re: If you are putting your data on somebody else's server...

Hold your horses.

When you start running workloads which have vCPU requirements which exceed the ability to count using 10 fingers and 10 toes, regardless of whether you count in binary or decimal, you can come back to me with your opinions.

Let me know how you manage owning all the hardware and software to manage those racks, and the personnel identifying failed hardware doing physical hardware replacements. Oh yeah, and how you sourced multiple redundant setups dispersed geographically as well as multiple redundant fibre hookups in each geo. I assume you're going to be purchasing your own real estate to manage this too, because if it's hosted in someone else's data center then your participating in criminality, right?

How long does it take to get am electrical grid hookup these days, and also source generators for redundancy? Remember, the uptime target is three and a half nines. Your downtime budget is 2.6.seconds per week.

[Anonymous Coward]

Re: If you are putting your data on somebody else's server...

"Remember, the uptime target is three and a half nines. Your downtime budget is 2.6.seconds per week."

99.95% is ~5min / week

AWS provide an "SLA" of 99.5% (~50min/week) at the instance level and 99.99 (~1min/week) at the region level.

I put SLA in quotes as they only provide a service credit to be used in your next monthly cycle, not really an SLA when they still get paid and you do not get compensated for the loss you incurred.

With the questions about how it is done, we (not the same AC), do it quite easily, we spend in the region of 1.8m/yr, including IT personnel (not including all other employees and hardware to run the business, this is about IT), have provided 100% uptime on 93% of our services over the last 5 years, with the worst individual service being 99.5%, this includes scheduled downtime and internal services which have no affect on customer facing production systems. With this we generated ~500m/yr in revenue.

All hosted on prem, redundant internet connectivity, redundant fibre and power coming in from different sides of the buildings to different suppliers, with building UPS and generator. The buildings built over 10yrs ago to our spec..

The costs for hardware replacements will be going up, but, we will be holding off on renewals for a few years now. See how the costs pan out, as what we have runs, covers the needs we have and will be supported for 5+ more years by the vendors.

[Heya]

Re: If you are putting your data on somebody else's server...

Sorry, 99.9995%.

[Horsie]

Re: If you are putting your data on somebody else's server...

Then by that measure you're not going on any of the public clouds.

[Heya]

Re: If you are putting your data on somebody else's server...

Why not? The numbers you quoted for AWS don't preclude using AWS as a primary platform to deliver 99.9995% uptime. Many companies do.

The point i was redirecting the conversation to was primarily that there's a world of differences between a mom and pop website hosted on your own hardware in a non air-conditioned room up to running a few million vCPUs with redundant power/fibre/security and the staff to maintain it. Cloud providers are neither criminal nor a crazy choice, neither is colo. Buying colo space is still owning hardware not under your full control. You're hoping physical security in the colo is trustworthy, and the networking is secure, etc etc

[Anonymous Coward]

Re: If you are putting your data on somebody else's server...

You can, but you would need to have at least 3 regions in play per service to get the 99.9995% up time from their SLAs, With the ideal being instances running in different 'cloud' environments, as single cloud operator can have its control planes affected, impacting multiple regions (AWS for example has had this happen).

[Jou (Mxyzptlk)]

Re: If you are putting your data on somebody else's server...

That is still two and a half minutes per year... Powershell:

((get-date "2026-01-01")-(get-date "2025-01-01")).TotalSeconds*0.000005

157.68

OK, that IS managable. Still the question: What kind of CLOUD service this is which needs that level, and whether the client software can handle failover transparently. And how much it costs.

Lets take something I know which can beat that availability: Server 2012 or higher as file server cluster (all three types of HA file server variants...), with multiple links and IPs, since Windows 8.0 as client (and higher) with SMB3 can handle all that stuff in the background transparently. Including automatic link aggregation.

[Heya]

Re: If you are putting your data on somebody else's server...

Agreed, that is the core of a system which could theoretically achieve that level of uptime.

Tbh, this is my day job and i still find it fascinating to scale systems to that level of reliability, and yes it does include multi-region and multi-cloud. And yes, it is more expensive than fully self hosting (effectively) three small data centers across the US and Europe.

However is the premium worth the physical security, data security, flexibility, scalability, and guarantees for uptime? Yup.

[FIA]

Re: If you are putting your data on somebody else's server...

The point i was redirecting the conversation to was primarily that there's a world of differences between a mom and pop website hosted on your own hardware in a non air-conditioned room up to running a few million vCPUs with redundant power/fibre/security and the staff to maintain it.

I think your analogy works the other way too though.

I like F1, in the UK this requires a VPN or a Sky subscription. I pick the VPN route. Because all the major VPN providers are blacklisted I spin up my own in Azure.

This is because the cloud is 'computing as a utility', nothing more, nothing less.

There is no way in the world I could do that on my own, I couldn't afford to rent a machine in a co-lo that I use for a few hours every other week. (especially if that colo's IP range was to get blacklisted).

However by using infrastructure as a utility I can. As I'm only using it occasionally I can easily afford the few pence to rent the infrastructure when I need; and because of economies of scale that infrastructure is cheap to rent when I do.

On the other hand, as I don't like giving my data to other people my cloud backups are on servers I rent in a colo. This gives me the control I want and is cheaper than spinning up a cloud storage instance to do the same job. (Thank you Hetzner for your bountiful server auctions...)

The cloud just gives you more options, it's not right, or wrong. It's just another choice. It's definitely not a scam.

[l8gravely]

Re: If you are putting your data on somebody else's server...

The problem is "Economy of Scale", it's a killer in terms of driving down costs.

Why don't you smelt your own Iron Ore for your servers? Or Refine your own silicon to make your own chips?

Because it's too damn expensive for the return!

I love running my own DC, because I'm strange and like pulling cables, racking gear, fixing hardware (i.e. swapping pieces I couldn't build myself in a lifetime of work!) etc. But I also enjoy scripting, helping my end users, dealing with the generator people and other vendors. But it costs to do it all for your self. Both in terms of people, knowledge, space, etc. AWS (who I am not a total fan of) has just managed to make economy of scale work for us, while taking their cut off the top.

running your own 1000 sq/ft datacenter with 30 racks is fun. Until one of the 10ton chillers craps out, or the power goes out and the transfer switch to the generator doesn't change over properly. Or the UPS can't take the load, or the batteries have expired and you didn't notice...

It's all stuff you can manage on your own. But economies of scale let's you spread that cost over such a large base that it becomes cheaper to use it than to not. It's a consequence of the industrial revolution that makes us all rich as kings of old today.

[Jou (Mxyzptlk)]

Re: If you are putting your data on somebody else's server...

Oh, well this is a nice text trying to convince us or the economy of scale. But why, for nearly all cases, are the customers paying more for their cloud than they did with their own servers in either a server room or a rented rack space? The "economy of scale" in these cloudy time is mostly to scale up milking money.

Most of our customers don't go cloud 'cause they want to, it is 'cause there is no choice given.

For some things "economy of scale" works though, most recent example is the scale of solar and wind farms.

[phuzz]

Re: If you are putting your data on somebody else's server...

But why, for nearly all cases, are the customers paying more for their cloud than they did with their own servers?

Citation needed.

(We're paying less)

[hri]

Re: If you are putting your data on somebody else's server...

Oh me oh my ... don't we see whose ox has been gored ! Even if ugly and expensive and un-needed and unwanted , networking/cloud is K-I-N-G ! Overdressed too I might observe, but more marketable than nekked.