Re: Fastly 'fesses up'
Then these outfits must stop claiming the whole edifice is anything other than a flaky, fingers-crossed, hackjob.
285 posts • joined 1 Jun 2010
Jeeez all the congrats on here??? WTF????
"Even though there were specific conditions that triggered this outage, we should have anticipated it," he wrote.
What utter gibberish. Show me an outage without specific conditions. Everything is a specific condition.
The shares haven't budged. There's a significant perverse incentive for an outfit like this to occasionally 'prove' just how 'critical' it is. It needs to really hurt in the wallet when this happens.
Also, where is the testing? Where is the evidence of the ridiculous uptime guarantees they make? Redundancy is a thing. Reliability engineering is a thing. Fault tree analysis is a thing. And if hardware can do it, so can software.
Slowing down, Ok. Dropping out of the air stone dead- not acceptable.
These outfits are a joke. If we're serious about crititical infrastructure uptime, they need to buck the 'uck up.
I don't get why any of this is surprising.
SolarWinds and its ilk planted the seeds. The tools were then dumped to grow the seeds. And now the 'script kiddies' can reap the whirlwind on whoever they choose.
Nation states are still discrete. It's just that now everybody else can start to take a profit. That's democracy.
As long as it's legal to pay up, this won't end.
The average 'home router' is modem + firewall + router.
One major risk is following instructions that come with your shiny new IoT garbage to configure port forwarding.
Here's a suggestion to ISPs: supply non-configurable routers.
In one easy step, we make people safer... and sadly, dumber.
This is the significant point here:
"Tesla supremo Elon Musk, however, claimed data logs from the vehicle showed neither Autopilot nor the automaker's Full Self-Driving mode was engaged. "
Wow. I can just hear the investigators, lawyers, claims managers, etc. etc. forming a queue.
As I've said before- Musk should have been satisfied with the best electric car. He'd have sold just as many.
But then.. what sort of nut job / tech disruptor reuses rockets...
For me, it's more: is the work worth doing, from a design / engineering view? Will it make the product or service better?
The larger the company, the more likely it is that people who haven't a clue, get to control people who have. And the more clueless they are, the more they feel the need to micro-manage.
The clueless can survive in large outfits.
>Valsorda points out that the vulnerability was introduced in an effort to mitigate timing side channel attacks,
Side channel mitigation is probably the hardest, most thankless and most boring job imaginable. It can destroy any structure to code, and all the old rules about how to create and maintain good code fly out the window.
We need some new science here. Sadly it's my bet that AI in some form or other will beat us to it.
>Figured out a while back as a software engineer that legacy code will eventually hit a point where F it, it's time to create something new
And I figured out that young guns always think they can do better, which is why we end up with so many wheels.
Good code does not inevitably corrode. But it does require intelligent maintenance. It's the hacking around by incompetents which does the terminal damage.
And it's the same with hardware, and absolutely with houses. Get a good builder.
Starting again is always more fun of course, as Elon knows very well...
"Two firewalls are better than one, right?"
"Remy gently explained that the DSL router had a firewall that would deal with most threats and sticking another one on the PC with everything cranked up to maximum"
By my reckoning, that makes 3 in total. And yes, as with drinking: 1 is good, so more must be better.
"Google’s underlying networking control plane consists of multiple distributed components that make up the Software Defined Networking (SDN) stack. These components run on multiple machines so that failure of a machine or even multiple machines does not impact network capacity. To achieve this, the control plane elects a leader from a pool of machines to provide configuration to the various infrastructure components. The leader election process depends on a local instance of Google’s internal lock service to read various configurations and files for determining the leader. The control plane is responsible for Border Gateway Protocol (BGP) peering sessions between physical routers connecting a cloud zone to the Google backbone."
See, we have this 'system stuff' which is incredibly reliable. But it's terribly complex. It turns out we don't really understand its full dynamic failure modes ourselves, but we don't admit that ;-)
"Google’s internal lock service provides Access Control List (ACLs) mechanisms to control reading and writing of various files stored in the service. A change to the ACLs used by the network control plane caused the tasks responsible for leader election to no longer have access to the files required for the process."
Someone changed some 'system stuff' and for some reason, it all fucked up :-O
"The production environment contained ACLs not present in the staging or canary environments due to those environments being rebuilt using updated processes during previous maintenance events. This meant that some of the ACLs removed in the change were in use in europe-west2-a, and the validation of the configuration change in testing and canary environments did not surface the issue."
Our 'system stuff' is so reliable that we don't really need to validate changes properly before rollout. So we didn't. We just validated any old configuration :-~
"Google's resilience strategy relies on the principle of defense in depth. Specifically, despite the network control infrastructure being designed to be highly resilient, the network is designed to 'fail static' and run for a period of time without the control plane being present as an additional line of defense against failure."
Our system stuff is incredibly reliable, so reliable that it'll kind of 'appear' to run normally, even when completely knackered! Isn't that just great? :-}
"The network ran normally for a short period - several minutes - after the control plane had been unable to elect a leader task. After this period, BGP routing between europe-west2-a and the rest of the Google backbone network was withdrawn, resulting in isolation of the zone and inaccessibility of resources in the zone."
Our completely knackered system stuff ran for several minutes. I know! Amazing! Unfortunately, during that time nobody actually managed to spot its complete knackerement because, well, why would they? They weren't even looking- our system stuff is incredibly reliable :-)
Very soon our system stuff fell over completely causing visible errors, which we weren't expecting AT ALL.
So why did our system, taken as a whole, fail to be resilient? Well, it's 'system stuff' and it's terribly complex. So. Hmmm... we don't... really... know... :-(
>the complexity of modern technology solutions is staggering,
1) You seem to think that's an inherently good thing. It's not.
>it always amazes me that people get so upset
2) People get upset when their heart monitor fails, electric car stalls, front door remains stubbornly locked, child's bedroom light won't switch on, etc. It's not the fault of those people. It's the complete insanity of designing every day products which require a live connection for even their most basic functionality. People get upset because they can't believe any product designer could be that much of a moron.
What you're missing is that states and big corporates with lawyers must be seen to be doing the right thing.
So they all agree with the lawyers what that means in security terms: hire one of their own to do security- a big corporate with a bunch of lawyers. And that's it. That's all that matters.
All the reactions we observe following an incident are dictated by legal. We never find out what actually happened in technical terms. We just get bullsh't. And the cover-all is: 'A completely secure system is impossible', they say.
But if some poor guy in his bedroom discovers a z day which actually matters and which fixing could actually save someone's skin, and makes the mistake of blabbing about it, he gets locked up or must go on the run.
So in other words, nothing changes. Great system.
(Nothing against lawyers........ phew.... think I got away with it...)
"Email is perhaps the nearest thing to a universal identity system for the internet, but if it is such a thing, it is much flawed."
Apart from those behind the corporate firewall, and maybe pushfraud victims, (who often don't have the insight), I didn't think anyone took an email address as any sort of ID seriously any more.
The total lack of authentication is kind of a red flag.
"That leads immediately to the question of whether it would be hosted for free"
My data has value. Without my permission and even when I don't have an account, Facebook is stealing that value via industrial scale, surreptitious slurping.
If I had an account, I would sign a contract agreeing that Facebook hosts my data in return for its value.
In neither case is it "hosted for free". Banks claimed for years that their current accounts (checking accounts) are "free" and it was always a lie. Now that interest rates are approaching zero, they may have a case.
It's early days for this, but I've signed up. Tim's been banging on about it for a while, so give it a chance.
>I'm not going to spoil my indignation by actually reading the original to check!
Yea, I'm generally going with that too...
"For asymptomatic subjects it achieves sensitivity of 100% with a specificity of 83.2%."
'asymptomatic' adjective, definition: (of a condition or a person) producing or showing no symptoms.
Anyone else see a problem here..?
I call bullsh't, (but hope to be wrong.)
> Windows 10
> Your device needs the latest security updates
> Can't download updates
> We couldn't download some updates.
> More info.
Good, because that's zero information so far.
"Opening Times (Customer Restaurant)"
Now we're talking.
How hard can it be to replace those 'helpful' Windows messages with a corporate banner, e.g. 'IKEA' on error? You'd think that would be standard behaviour for anything unexpected in embedded versions.
It would spoil Bork spotting though..
Different hardware architectures => assembly.
Thin layer abstracting most of those differences => C.
Thinnish layer abstracting common OS features => probably Rust, Go etc., but nothing 'unsafe'.
Everything above => whatever maps best to the requirement. And that's the hard part.
But it's not C++.
As always, horses for courses.
"Minns slammed the outcome as "another national joke" that ranks alongside "intercity trains that don't fit the track, or the 'Ferry McFerryface' fiasco"."
Wikipedia on a related ferry fiasco, care of The Rt Hon. Chris Grayling:
"The announcement that Seaborne Freight had been awarded the contract caused controversy after it was revealed that at the time contract was signed, the company had no ships and had never operated any. Critics also pointed out that the Port of Ramsgate would need to be dredged before services could begin and raised questions as to whether due diligence checks had been undertaken before the award of the contract. By 3 January 2019 dredging had begun in Ramsgate."
On hearing about Transport for NSW's troubles, Chris Grayling was later heard to comment:
"Look, it's not a goddamn competition."
> the possibility that the crims might just take the money and run.
Assuming they leave their calling card and have a 'reputation' to protect, that would be a fairly dumb strategy. It's no cost to them to follow through on the deal.
The victim's choice to pay is understandable, but supremely selfish.
"the attackers successfully manipulated a small number of employees and used their credentials to access Twitter's internal systems,"
This is an attack from inside the security model. This is equivalent to an Intel processor side channel attack.
*Some* employees will always have access to tools which permit account access, at the very least enabling a credential reset. *Some* can modify system code! If those employees go rogue, or stupid, then it's game over. There's no mystery to that.
>Do YOU decide what is a "jollie", and what is "essential travel"?
No, COVID-19 decides. And CO2 decides.
None of the shouty little kids like their nasty medicine, but sometimes they have to suck it up. Can't be much worse than "protein chunks in spiced slurry".
-Arm Launch Escape System
-Scrub Scrub Scrub
-Disarm Launch Escape System
That's a good sequence. I like that sequence.
It was oddly gripping, partly because it's been a while, and partly because it's such a relief from the relentless idiocy of politics. At last: real people, doing a proper job, and doing it well. (You too NHS)
Roll on Saturday.
>New versions need to be tested properly before roll out.
Ermmm... BORK?? You had me, up to that point. I think you'll find testing any new version or patch is generally... a good idea.
"but "we're about to upgrade all of our production machines to a custom Linux build to improve their stability and generally operate with more efficiency." "
Biting the hand that feeds IT © 1998–2021