Re: Enterprise backup plan?
> And of course we took the chance to secure the shit out of IT, and they readily agreed to.
Great opportunity to do this when no change control is slowing it down.
102 publicly visible posts • joined 21 May 2014
The way to respond to this is by sending a Letter before Action, threstening to take them to court.
This goes to the legal department which has very different levers.
And if this goes to small claims court, Google can't recover legal costs so they will do their very best to avoid this.
"95%" is a nice headline figure, but misinterpreted. This figure in the study specifically is about the transformational impact of specific, bespoke AI projects. The same study also shows significant benefit of generic AI solutions in something like 7 out of 9 industries but I've never seem those one quoted. It's not sensational enough.
We should think twice if the same study is quoted over and over again to support a specific conclusion, rather than being supported by a broad evidence base.
Actually, if you have a problem on prem, in the eyes of the customers and, if it's big, the press, its YourCompany that's the problem.
If the problem is AWS, then it's AWS that's the problem.
That's a huge reputational difference.
And please don't tell me that sufficiently complex on-prem systems don't have similar length outages. We just don't discuss them here.
So instead of your ISP and their upstream having access to your possibly (but hopefully not) unencrypted traffic, with a VPN you give that to the VPN provider, their hosting provider (often small outfits, rarely known and documented), and their ISP and upstream.
How exactly is that an improvement?
That may be so, but they don't need to be "human". They need to solve certain tasks and respond to requests that customers have in a way that meets theirs needs.
First line human customer support in large organisations is generally poor. GenAI based support from what I can see is generally worse.
However, it's difficult to see why it couldn't soon exceed the admittedly low standards of human support.
In an ideal world, GenAI would solve simple queries directly and assist human agents to do their jobs much easier and quicker and better. Together that would allow to employ higher-level, better human agents that can focus on the important tasks that need humans.
Why should am AI agent not be knowledgeable in local law?
You'd have different agents, trained on different material and law, for different countries. Just like human agents.
It is true that AI agents will have to be dealt with by HR as a new type of employe with similar issues - performance, bullying, discrimination, adherence to policies etc.
The value of automation isn't just about saving time for initial provisioning.
It's also to ensure consistency so that you don't experience a surprise issue when the primary switch fails and the backup is configured slightly differently. Or self-protection ACLs are accidentally missed out.
It also enables more frequent change. For example SNMP communities. Or changes in the template regarding security features.
Second sourcing cloud services, ie building your services in a cloud-agnostic way, means you only use the least common denominator of their features. So no use of fully managed higher-level services such as databases or messaging that are available without ever having to worry about resilience, building, maintaining, testing, upgrading or scaling them. Those that allow you to build complex systems in no time.
Second sourcing gives you all of the costs and few of the benefits. Not a great deal.
Not to forget the most insidious of tactics, self tests.
Ensuring that every time you switch the printer on, a significant amount of every ink cartridge is used. Ensuring that you have to replace all of them after a while, even if you print 99% black only.
Or the tactics to mix all colors to print black.
This!
To all of those downvoters: What better way is there for newcomers to learn than to sit with different people, look over the shoulders and get ad-hoc explanations?
I've seen it time and time again that the all-time-at-home people are "productive" at what they are doing, but not able to change quickly and flexibly and discover new ways.
It depends on the type of work. If there is a clearly described piece of work, then WFH can be very productive.
The moment you need to go beyond and figure it out in the context of a complex company it breaks down. Serendipity is important.
Innovation works much better if people are actually sitting together.
Bringing new people is hard if they can't look over the shoulders and ask quick questions.
I'm seeing it every day that there is a strong correlation between doing WFH almost exclusively and working in a transactional manner and being stuck in your ways.
If the content was available freely globally (which you seem to advocate), there would be no Harry Potter. No LoTR. None of the expensive-to-produce film.
Cinema wouldn't exist.
You might like that, but it's hardly a majority view that the world would be better without these.
People don't decide what they want in an empty space. It is heavily influenced by their environment and what they see. That's why we first need to create an environment without biases before people truly make choices based on their real interests and talents.
Men in nurseries are still looked at suspiciously. Of course that will turn away men who would otherwise like to do that job.
If they copied for example the exact shape and forms of the cars then this would almost certainly "piracy" (violation of intellectual property rights).
I suspect that the overlap between the potential customers of the Burmanese and the original versions is next to nil, and hence they don't consider that this a risk to sales.
It might been seen as a curiosity that enhances mystic brand status.
Just because Bugatti doesn't act against these people doesn't mean that they believe legally they could not.
"Which always makes me wonder, why did the orignal design / architect / plan not take into consideration all these points"
It's easy to put this down to incompetence or unwillingness. However I think the deeper reason is in the nature of such contracts. In order to specify fully you'd need to complete a good proportion of the design which by itself is part of the tender.
It's just almost impossible to specify precisely what you need in a reasonable sized document that takes a proportionate time to prepare.
Looking at this at reducing the RTO from 8min to 4min is a bit of a dangerous angle.
Imagine there was an earthquake that took out their SJ datacenter, and all services recovered with no data lost 8min later. Noone would complain; that would be celebrated as a success.
Imagine, on the other hand, a dirty failure that introduced instabilities which causes replication to have issues and hence data to be lost. Technically the service might have recovered in 4min but the impact and fallout would be massive.
Crucially their test was a planned failover. They FIRST drained all traffic away and THEN failed over. While this by itself is an impressive achievement, reality often doesn't allow you this luxury, and that's where things can go really pear shaped.
My experience with Santander: Made online transaction a bit larger then normal (but to account I had previously sent to). Account blocked. Online unblocking needed details of recent transaction - there were none on this account. Went to branch. Passport was not enough to prove ID. Needed debit card which I couldn't find. Alternative was someone at branch who knows me to confirm ID. I don't know anyone at branch, doing 99% online.
1.5h later, sorted. Sigh.
Infrastructure needs to be refreshed and expanded and hence its depreciation is a real cost that ends to be factored in all the time.
Additionally customers expect an exponential* increase in bandwidth consumption without cost increase. That also requires constant reinvestment.
Then there is customer service and repair which takes up a significant proportion of the fees.
At the prices of Internet access in the UK this really isn't a high-margin "cash cow".
* Bandwith consumption has for a long time been "exponential" in the true meaning ie multiplying each fixed period. Not just meaning "large" as it's normally used (a pet hate of mine)
As others have commented the problem with the "roads" argument is that is justifies any and all support for criminal activities.
Take banks for example. With the same "roads" argument you could say that banks are just providing the transport of money (legal and illegal) just like roads provide transport for goods (legal and illegal). So banks shouldn't have to do any money laundry and customer checks and should be allowed just to serve any customer.
Some may think that this is right but it's certainly not a majority view.
This does deserve serious attention but why single out AWS?
Using the 80x cost from the blog, we get 98.8% profit margin for AWS. If they did offer 27% discount quoted, that would still be 98.3%. An improvement, yes, but still atrociously high so really they should call them out together.
Add to that that GCP starts at a 22% higher cost so a 27% discount on is just a tiny amount less than AWS.
The two risk questions typically asked (and postulated in the article) are "how likely is it to go wrong" and "if how what's the impact".
These questions have a fundamental flaws. Firstly, there isn't a single dimension. A typical change has some low-likelihood change of something going a little bit wrong, and a very-low-likelihood chance of going very seriously wrong. These can't be put into a single answer.
Secondly, both of these questions are next to impossible to answer objectively and hence with a degree of repeatability. Different engineers will, perfectly legitimately and competently, come to different answers. For example one engineer might consider a typo in applying a change with impact to one particular service as the impact to focus on as it's the most common scenario. Another one would instead consider triggering an unknown software defect that impact a whole host of otherwise not directly related services, as this is the worst case impact. In either case it's difficult to see how "likelihood" could be objectively described.
Thirdly, these questions put the focus in the wrong place. It focuses on the expected outcome of the change so encourages not just wishful thinking but also focus on the "expected unexpected" outcomes.
I am advocating two different questions instead: "How quickly do you notice a problem" and "how quickly and reliably can you roll back". These two questions are considerably more objective, and they drive good behaviours such as focusing on monitoring which might otherwise be overlooked. Crucially it encourages to think about worst-case service restoration which is often most relevant for the business - think about 5min worst-case impact vs. hour of worst-case impact. So these questions focus on dealing with the biggest issues in a mature change control environment - the "unexpected unexpected".
Good plan ... except all this does is to shift the SPOF from CDN to CDN selector so doesn't solve the problem.
And it introduces significant additional complexity which experience shows is often the cause of problems. If you are serving a simple static page that's not an issue (but then you wouldn't be quoted here). Larger sites need to consider for example test coverage, troubleshooting, logging etc. which are all far more complex on multi-CDN.
Thinking you can just re-point a CNAME is, well, wishful thinking.
And that's not even touching on large services that need to do capacity planning with the CDNs and take selector decisions based on load.
4.5bn users times $20 is $90bn. Quite something created there, more than 10 times the UK music industry.
So I'm saving 15 minutes per year, on average. That assumes it takes 0 seconds to find the device which is certainly not the case for me.
Then add the time to select and buy the device. Then add the time to make this work on every single device which likely adds up to a few hours. Then the time to troubleshoot when it doesn't work. Then the efforts to take a tiny device with me anywhere because we don't already have enough in chargers, cables, adapters etc. Then add the time and cost to replace it when it gets lost.
Seriously?
In that case you will avoid the airplane that has had the most intense safety scrutiny and awareness amongst pilots. There won't be a single one who is not fully aware of MCAS and its dangers.
Avoiding all Boeing planes might make more sense on the assumption that these failures stem from systematic corporate culture and governance failures.
Avoiding all planes relying of FAA certification even more so given that they are ultimately responsible for allowing much of this to happen.
It somehow restricts your options though.
Quite so, they asked the wrong questions. Even assuming that the algorithm itself is completely black box there are plenty of questions to ask.
What training data so they use? How do they evolve the algorithms? How do they judge whether they are working well? How do they incorporate new sources of data to search? How do they counter-manipulate deliberate attempts at manipulating results (SEO)?