The Register Home Page

* Posts by Claptrap314

3788 publicly visible posts • joined 23 Jan 2015

Microsoft blames monster email outage on software defect introduced in July fix

Claptrap314 Silver badge

Here we go again

IF I were employed, this would be hilarious to watch. As it is...

Most of what I know about HA & SRE, I learned at Google a decade ago.

This sounds like m$ is, at some level, trying to do things right. Real-world global HA means regional failover and a whole lot more. You do NOT drop customers just because a couple of regions go out. You shift them to healthy regions. (And your SREs are already working on the problem--see below).

"Monoculture" is not the problem here. It would be an absolute nightmare to try to manage parallel systems doing the same thing, both at an organizational and at an operational level. And the cost? Much, much, more than you think. Google developed an entire new language (go) that is torturous to work in because it ran fast, and at that scale, hardware costs outweigh salaries.

So, let's talk about what went wrong. Failure to test properly is the original root cause. (And HA locking sounds like a fascinating problem to work on.)

But how THAT failure turned into this?

1) Load on the affected servers rose for almost sixteen hours before failures--and no one noticed? Rule #1 of SRE: alert on yellow. Read that again. Proverbial klaxons go off if you loose half of your excess capacity, not when systems start failing. (And by "capacity", I'm talking about load availability across your fleet, not # of servers). It sounds like m$ didn't think there was a problem until the support cases were coming in.

2) Recovery from massive failures is also hard, BUT this has been a known issue for well over a decade (I would guess two, but I KNOW Google had to learn the hard way in 2015).

It turns out, the solution for mass failure recovery is the same as preventing the mass failure in the first place: quota. To quote Simon, "My job is to keep the network up. That I allow users on at all is because of my good nature." Read THAT again.

Quota servers actually are quite magical. Turns out, you really CAN drop tens of thousands of requests a second a server--if you keep the logic simple enough, but especially if you are prepared to handle 0% availability to protect your back end as a core feature. And no, your admin interface does NOT share a port with user access.

Of course, you still need to monitor your quota farm for ITS health.

Now these systems are NOT supposed to be responding to generic traffic, but to traffic generated by their own clients, which brings me to my next point. "EXPONENTIAL BACKOFF! DO YOU SPEAK IT?"

Indeed, to defend against this scenario, you can do even more than exponential backoff. As a problem develops, quota servers can signal clients to back WAY off.

Claptrap314 Silver badge

Re: "testing wasn't good enough"

Nah. Check your contract first. Or, if you're NOT in the enterprise, your Ts & Cs.

AWS distills its cloud into a local emulator that runs on your PC

Claptrap314 Silver badge
Facepalm

Let me get this straight

My business systems run on AWS. I want to test my code before deploying it. I run a dedicated test account, and test my code there.

If I use this emulator, how does my workflow change?

I test on the emulator, get things working on the emulator. BUT, the emulator differs from actual end points. So, I test my code on the dedicated test account.

Not seeing much of a win here.

Microsoft struggling to fix massive cloud outage that's taken down email and Intune

Claptrap314 Silver badge
Linux

Checks calandar

Yep. Day ends with a "y".

Attackers backdoor trusted AWS integrations to steal secrets

Claptrap314 Silver badge
Stop

You need a BOFH for these

The advice given here is wholly inadequate. This is a deep compromise of a company's AWS infra, and unless you are DEEPLY certain that your ship is 100% tight, you must AT A MINIMUM assume that enough of the AWS account it compromised that it is better just to rebuild the thing from scratch.

And it appears that the bulk of the compromises happened at orgs where the CTO should be falling on their swords because of shoddy practice.

Of course, THAT implies that you are keeping proper backups of everything & that you have everything in proper IAC.

14-year-old creates tool for AI agents to band together like a team

Claptrap314 Silver badge

Okay. Write out the spec for the First Law. I'll get it coded as you as you get the spec to me.

I, Robot was a series of thought experiments that presuppose that we can translate such vague terms as "human being" and "harm" into some sort of precise logic. I use "human being" as "vague" because the law has been rather pointedly ignorant about what exactly should be allowed to count.

Improved asteroid deflection model runs faster, reveals more doomsday worries

Claptrap314 Silver badge

I think you've been around long enough to have read the comments the last time this issue came up. If you've got fifty years to work with, then sure. Actually almost ANY method you come up with (EXCEPT a one-shot blast) is likely to be workable. But the presumption is that we are under some time constraints. Turns out, delivering the ergs directly is going to strongly beat almost any other option.

Claptrap314 Silver badge

Re: As I've said before

I'm talking about sending a sensor & computational package ahead of the erg-delivery packages. Yes, it would need to bleed energy. But, again depending on how much time we have to work with, we might want to do that to map things like gravity gradients...

Claptrap314 Silver badge

As I've said before

A single-mission solution is downright stupid unless we have several decades to work with.

As Bebu mentioned above, cows are not just not spherical. They vary in density. They also don't cook uniformly.

There are a jillion ways that computations will be wrong, and, as I have said before, this is NOT a project befitting tight design margins.

You don't send one on this mission, you send at least a couple dozen. With gaps. And maneuver rockets to adjust for what your cannot predict, but absolutely DO anticipate.

And since this IS a multi-part mission, you don't waste time trying to be optimal in the mechanical sense. You go for optimal in the probability sense. Depending on what you expect of the object, you either detonate on or slightly above or below the surface.

Each strike directly ablates a chunk of the rock, and creates a strong "kick" into a different orbit. You really do have the ability to time these strikes an hour apart or even more--enough time, if we like, to observer and correct from the ground (if we think we need it). We could even mount a precursor mission to put an orbiter on the rock to observe and report the effects of the strikes.

PC chip plunge wipes out a quarter of Intel’s value

Claptrap314 Silver badge

Re: AI is killing the sector.

The thing is, ASIC and FPGA were already very much a thing. The move by crypto to these was precisely because hashing really is so special that dedicated hardware can give you a big win. (Pun intended.)

Having 0 direct exposure to the relevant code, but with a fair amount of knowledge of some of the relevant factors, I expect that FakeI is doing a lot of sparse matrix multiplies. Or maybe not so sparse. That involves a lot more random memory access than hashing does. It's also what CPUs (and GPUs?) are made for. Since memory access has been THE bottle neck for a few decades, again, CPUs are already optimized for this class of work.

AI could break legal systems by finding previously hidden loopholes

Claptrap314 Silver badge

But wait, I'm just gettin started

As for the resilience of Common Law, the comments are correct. What I believe they are missing, however, is what actually happens when lawyers (including judges) deliberate.

What this will do is invent more outrageous and bizarre theories of law for lawyers to pretend to take seriously. It will in fact damage the law, because stability is a critical component of a healthy legal system. Probably not much, it's wafer thin...

Oracle gobbles $20 billion borrowing limit a week after setting it

Claptrap314 Silver badge

Re: Oracle's corporate bond rating

Mostly true, but that's a really REALLY low bar...

Microsoft's Canada timezone troubles leave admins fixing calendars, pretending they're in Bogota

Claptrap314 Silver badge

Re: Microsoft's stumbling way of telling the time

Certificate expiry?

License expiry?

No shame in the Domain Name game: DNS is doomed, literally

Claptrap314 Silver badge
Gimp

Re: Is this a localized DNS database?

In more ways than one, methinks.

Disgraced Indian cert issuer leaves the cert issuing business

Claptrap314 Silver badge

Re: A question about dodgy CAs

Not better than nothing. It solves a couple of important problems. The only reason that these problems seem small is that we dealt with them before the internet became big enough that organized criminals (aside from the TLAs of various countries) had not yet started attacking it.

Problems important enough to destroy commerce over the internet.

Claptrap314 Silver badge

Re: A question about dodgy CAs

I believe that the "H" in DoH stands for HTTPS. Without it, there is very limited benefit to consumers. Well, the benefit to consumers is still pretty limited, but okay.

Claptrap314 Silver badge

Re: It stinks

They were publicly exposed but not publicly used. And apparently, no one, ever dreampt that certs would be out there with such weak numbers.

My prediction: there is going to be a rush to find weak certs.

My hope: There is going to be a rush to confirm that there are no other weak certs out there.

My expectation: At least one other CA is going to get popped.

Claptrap314 Silver badge

Re: A question about dodgy CAs

They were able to substitute ANYTHING, as was anyone in the network path, and that is a problem. In particular, without HTTPs, the credit card information you send for an online purchase is visible to anyone, and can be modified.

It doesn't just protect Google's business model. It protects the business model of anyone using the internet to move money.

Yes, Google is E.V.I.L. That little lockbox helps them most of all. But everyone else is still benefiting.

Claptrap314 Silver badge

Re: A question about dodgy CAs

You might be confusing the term "traffic" slightly. First, you are definitely NOT forced to trust your ISPs DNS. Especially with DNS-over-https, your browser is going to resolve DNS where you tell it. If you are root on your box, it is not hard to change DNS resolution either.

Second, there is a big difference between knowing that you are sending traffic to my.unpopular.site.com and knowing the contents of those packets. Your ISP is going to see where you send packets. (Which might be your VPN provider.) With HTTPS, they won't know what is in them.

Claptrap314 Silver badge

Re: A question about dodgy CAs

I upvoted you, but I do have some disagreements.

1) HTTPS has never been an end-all, be-all of network security. It's purpose is both to make private the contents of packets to outside parties and to provide assurance that the server responding to www.example.com is in fact under the control of the registered owner of www.example.com. That's two important, even critical pieces of the security puzzle.

In practice, most (all?) CAs are only checking that the DNS is controlled by the entity requesting the certificate. Which, given that certificates are tied to domain names, seems about right.

2) While for individual users, this mostly aids privacy, in the corporate space, it provides a significant security benefit. Because appropriately paranoid ops teams trim down the CA list. They can do this because they know their users, while the browser manufactures cannot.

And I would be shocked if even chrome made it difficult for an individual to change the CA list.

Hackers pile on PHP2Shell as AWS reveals attacks began before disclosure

Claptrap314 Silver badge

Re: Somebody blabbed

1a) An AI "agent" was overly connected & blabbed.

'Password1' and a forgotten test account land Welsh outfits in hot water

Claptrap314 Silver badge

You ought to read up more carefully about the relationship between the public & private keys before proposing such a thing.

Claptrap314 Silver badge

Spend some time doing integrations in regulated industries and get back to me on that.

Claptrap314 Silver badge

Re: Not that fines are much better for orgs that will end up recouping the costs from the public.

Well, fines actually have to be collected before we consider if they matter or not, wouldn't they?

And just how much did UHC get fined for leaking pretty much everyone in America's private health information through a division whose api only supported ciphers no longer supported by modern OSes?

For various reasons, fines are are limited value, I agree. As mentioned in the first comment, for public entities, they don't really matter at all. Jail time, I suspect, would.

Google Cloud says a 'power event' caused today's outage

Claptrap314 Silver badge

Re: Resilience

Hydroelectric is "renewable"? Really? Maybe for a century. But not for two. It's actually a substantial problem to clean up.

Claptrap314 Silver badge

Here we go again...

I was with G 2015-2016, as an SRE (SWE side). That is, I got to see the early run-up for GCP.

First, it's important to note just how extremely different GCP is from the internal systems G built up until 2014. When I arrived, our systems were build around DC availability of about 88%. Google service availability at the time was very much 5+ 9's. Troll the archives for Google service outages 2010-2015.

But building a global HA system on such a substrate requires amazing OPs expertise backed up by some very senior SWEs. The FAANGs had that. Most enterprises don't. Let alone the SMBs. And GCP was not selling to the other FAANGs. The sales team pushed back, and rightly so. DC availability had to go up. Way up. In fact, at the DC level. the entire application of SRE had to be rethought.

So, they broke DCs up into availability zones. Now, a full DC was never supposed to go down. This meant that all of the complexity of power redundancy (previously handled by the power grid as a whole) now had to be moved into the DC. And aircon. And water supply.

As for customer-received HA? That still required multiple AZs. I don't know what G "promises" regarding AZ availability, but I would be shocked if a single AZ is supposed to be HA.

Google & AWS can give you the tools for HA. (Well, as long as us-east-1 stays up) You still have to know how to use them.

Claptrap314 Silver badge

Re: Resilience

In, I believe, 1997, US East went down. Black start required. We were back to normal function in about four hours.

Well, I guess, Spain...

Indian certification authority eMudhra’s private keys compromised – one of them was the number ‘4’

Claptrap314 Silver badge

Re: There's always an XKCD

It takes something special to earn your own XKCD, that's for certain.

Claptrap314 Silver badge

Re: Come in Number 4

Who is Number 1?

Tens of thousands of Nvidia GPUs sit in Meta's Prometheus supercluster, waiting for a failure

Claptrap314 Silver badge

Re: What exactly is new here?

As I read this press release, it sounded like they deployed "compute" a rack at a time. So we're not talking about replacing individual processors, but hundreds (or thousands) at a time. That could bring MTBF way down, but I suspect there is even more.

I doubt I would have thought about this if not for my SRE training, but it seems quite possible to me that they are deliberately running the parts hot. The chips are doing a lot of processing, and even if the market value of a token does not pay the bills TODAY, the accountants have certainly priced an internal value of a token where they are making book. Depending on that value, and quite a few other things, one might be able to crank out extra tokens at a rate sufficient to absorb the cost of pulling a rack out, replacing it, and repairing it. They don't expect to be running these CPUs in three years, after all.

Claptrap314 Silver badge

Re: What exactly is new here?

Key term: "modern computing environment".

Claptrap314 Silver badge

What exactly is new here?

The ORIGINAL electric computers, with their tens of thousands of vacuum tubes had a failed star rate >10%. The solution? Once, running DO NOT TURN THEM OFF.

This problem is literally as old as electric computers, and so is the solution.

It's not my fault that the folks over in Zuckland either had to figure this out themselves and then thought they were so special that they should do a press release about it.

Sure, after all these decades, there are changes in the application, but no one who has worked, or, as in my case, simply been in the room, with big compute jobs is going to find anything new here.

ICANN testing tech to let law enforcement access non-public WHOIS info

Claptrap314 Silver badge

That one got abused to the point that no one has monitored it, probably for decades. In fact, the fact that it is unmonitored is so widespread and so well known that I don't even get spam there now. Unless my ISP is silently dropping it for me, which I would appreciate.

Maxwell gives sub-optimal code a runtime glow-up

Claptrap314 Silver badge

After three years

We finally have ONE application of AI that I can get behind.

AI proposes a code change, which gets formally verified before implementation.

Okay, cool.

Leap seconds face judgment day as time lords prepare to vote

Claptrap314 Silver badge

Re: Computer systems designed to handle added seconds but not removed ones.

By "Eggheads", I'm not talking about the century + of scientists who have struggled with an increasingly, shall we say, "rich" description of Spacetime and developed standards.

I'm talking about the people who made the decision to push leap seconds into system time.

Claptrap314 Silver badge

Re: I never understood the fundamental need to keep the two synchronised.

Tide tables are one of the places where the stupidity of leap seconds shows up--if their precision were in seconds, which I severely doubt.

As you say, people are already messing up the conversion. Given the way the brain works, I would assume that once a 5- or 10-minute offset becomes the norm, that it would be messed up a lot less.

Claptrap314 Silver badge

Re: Computer systems designed to handle added seconds but not removed ones.

Leap seconds were NEVER a good idea for computers. This was another case of the eggheads paying attention to the wrong thing.

System time is almost entirely about relative time. "How long did this packet take to cross the network?" "How long did it take this process to run to this point?" What is more, you do NOT want to use wall time for cron or at because this is ill-defined.

Yes, when it comes to human-human interactions and the like, wall clock time is appropriate. And that is where the complexity needs to live--in libraries to do that conversion. System time should not be competing with the tax tables for complexity.

Bank of America had to bring in 1,000 contractors to patch all the bugs Mythos identified

Claptrap314 Silver badge
Facepalm

Can confirm

Source: I was contacted by a recruiter on a Sunday for "urgent" staffing request to fix holes discovered by AI. Client: BoA. Requirements included familiarity and comfort with AI. I told her I would be shorting BoA stock.

Ukrainian drone attack takes out Russian datacenter

Claptrap314 Silver badge

Re: “the source of the drone attack”

And corruption.

Claptrap314 Silver badge

In the case of Microsoft, you need to worry about all-region failures. I cannot ethically sign off on HA if everything is in Azure.

But again, multi-region does NOT ensure reliability. You have design your entire stack for reliability. And the regions CAN go down together if, for instance, a hurricane does a long track.

Claptrap314 Silver badge

If the stories here are to be believed, China is simply waiting for the correct time. At which point, pretty much every region of every cloud provider goes down, plus a whole lot more.

Attackers hijacked top-level domains, minted fake security certs for Google and other orgs

Claptrap314 Silver badge

Define "anything wrong"

“Due to the nature of the attacks, we have no reason to believe the Certification Authorities (CAs) that issued the impacted certificates did anything wrong,” the alert said.

Then what exactly is all the fuss about?

These TLDs most certainly DID do something wrong. They failed to maintain the security of their systems. The ONLY responsible response is for them to be delisted until they demonstrate that they have gotten their act together. If ICANN does not do it, the root severs should. If the root servers won't, then the browsers need to. If the browsers won't, then anyone maintaining a firewall must.

Security, people. Ever hear of it?

Register reader hit with surprise bill after Microsoft portals disagreed

Claptrap314 Silver badge

Re: false accounting still an offence...

Or Secretary of State. Or Vice President. Or former of any.

AI risks make some insurers wary of corporate liability

Claptrap314 Silver badge

Isurance company saviors, round II...

You may remember that a decade ago, some of us were holding out hope that the insurance companies would force an industry turnaround, as the quality of software was becoming so bad. That appeared to fall through when they continued to cover paying attackers for data recovery.

So I'm not holding out much hope for a better result this time. Would be nice, however.

Claptrap314 Silver badge

Re: Real World consequences .

That's actually great news. We need to press on this. They're using LLMs to read x-rays for crying out loud! I can believe that NNs could eventually get to that level, but LLMs?

Test environment let anyone access live customer data

Claptrap314 Silver badge

Re: So the lesson is

beware_of_leopard.sql

Nine-year-old runs up $118,000 bill on dad's corporate credit card advertising his Roblox YouTube channel

Claptrap314 Silver badge

Re: Why...

Yeah, that's how I remember my IBM card, twenty-five years ago. Didn't use it for anything but reimbursable expenses, of course.

Ex-FTC boss Khan urges Uncle Sam to break out the handcuffs for AI CEOs, citing 1934 precedent

Claptrap314 Silver badge

Re: Why ?

The midterms are up, and the political season starts (for the low-information voter after Labor Day). So, here we are.

Tech team stood and laughed at colleague who pressed the wrong off button

Claptrap314 Silver badge

Re: We've never tested that because of all of the problems it can cause

Actually, the TDD process required constant 100% code coverage, to include data ranging. See my other post.

Claptrap314 Silver badge

Re: We've never tested that because of all of the problems it can cause

Then you mis-learned TDD. You write ONE test. You see it fail. Then you write code so that all of your tests pass. Then, except for the first test, you review your code & refactor as appropriate.

NO code gets written except to cause a specific test to pass.

100% code coverage is thus required by the process.

I cannot over-recommend this approach. I did a one-month project where the code for the second-to-the-last test cause two surprise failures. There was a deep design error & I had to rip & replace more than 20% of my code. Took less than two hours from the surprise fails and the fix.

(The code went live with 0 bugs reported.)