* Posts by ChipsforBreakfast

68 publicly visible posts • joined 20 Sep 2019


China promises its digital currency will offer 'controllable anonymity'


Which backs up exactly what |I was saying - NO digital currency or payment method is a suitable replacement for hard, physical cash.

Had those tourists had wallets full of actual money instead of plastic cards they wouldn't have had a problem, would they?


Privacy is the tip of a very nasty iceberg

I may seem something of a luddite here but bear with me. If I'm not mistaken we already have numerous & very effective ways to make cashless transactions should we so desire. Bank transfers, cards, even apps all exist for exactly that purpose and by and large function well enough that they've become accepted in society for daily use.

Cash is different. Physical cash is exactly that - a physical, immutable representation of wealth. If you hold it, you have it and it's yours to spend as you see fit. If someone wants to take it off of you then they need to physically confiscate it, a process sufficiently complex and time consuming that it's intrinsically resistant to abuse. It's not practically traceable and has no privacy risks. You can't hack my wallet from a bedroom halfway across the world. You can try and pick my pocket if you like but that carries a distinctly non-zero chance of you getting a punch in the mouth and quite possibly jailtime which again tends to limit the risk involved.

If the bank is closed (or worse, bust) then the cash in my pocket is still mine. It still has it's value, I can still spend it, at least for as long as other people agree it has value.

Now, replace that with a CBDC and a 'digital wallet'. Instantly all the advantages of cash are gone. Digital transactions are traceable, so no more privacy (anyone saying anything else really needs to go look at some modern history!). It's only 'yours' for as long as the bank say it's yours - it can be blocked or invalidated at any time, easily. Pissed off the government, no more money for you. Voted for the wrong party? Said the wrong thing? Spent more than you should on 'bad' things? Same result... what you thought you had, you don't have. That 'digital wallet' can be and undoubtedly will be hacked eventually - everything is.

For central bank transactions, perhaps it's justifiable BUT there are already plenty of solutions for that... it's not the driver for this.

In short, CBDC is not a solution to any problem faced by normal individuals. It's at best a waste of time and at worst a tool for total control & oppression on a scale never before seen.

The privacy issues are the tip of the iceberg.

Chemical plant taken offline by the best one of all: C8H10N4O2


Alcohol + CRT = fun!

I have witnessed the result of a poorly-placed CRT encountering an entire pint of lager. Spectacular light show, copious smoke and one fire-alarm induced evacuation later management were persuaded to relocate the monitor away from the end of the reception desk nearest the bar....

Nuclear power is the climate superhero too nervous to wear its cape


Re: Deaths are not the only metric

Please, dial down the angst. The exclusion zone is far from lifeless and has in fact become something of a wildlife haven. The exclusions zone for Fukushima has already been lifted.

Yes, Fukushima plans to release treated waste water into the sea - water treated to remove most radioactive contaminants leaving only a very low level of radioactivity - lower, in fact, than the naturally occurring background radiation.

All of which fails to note that modern reactor designs such as those discussed in the article are designed to prevent exactly the type of accidents that occurred in either of these incidents.

As for your 'argument' you know full well that cleanup will take decades to complete - do you seriously think we have that long to wait?


Re: Deaths are not the only metric

I don't disagree with you - there is never a perfect solution. The point really was about the relative lifespan of the two - turbines have planned lives of 20 year, solar panels roughly the same or less, depending on the conditions. Most proposed reactors are aiming for lifespans of 40 - 50 years minimum (many current plants are sitting at around 40 years operation & still passing safety inspections).

So while they do use significant quantities of non-recyclable material they use it less often and probably manage to generate more power over their lifespan too - not zero impact (nothing is) but less impact per KW/h generated. It's also contained in a far smaller area, lessening the physical impacts on the surrounding area & wildlife.

Nuclear power is not without concerns, drawbacks and impacts that's for sure but they are at least comparable to quite possible lower than (I'm no expert, just an interested layman) the renewable alternatives.


Re: Deaths are not the only metric

Regardless of what's happened in the past (and the article does a very good job of explaining why that's not really relevant to this discussion), there is a choice to be made now.

We can invest in nuclear power alongside renewables and meet the goal of zero carbon emissions from power generation

Or we can decide not to.

If we decide not to, we either have to drastically reduce & rethink our use of power or we accept that we are not going to be able to reach the zero emission goal and we'll need to deal with the consequences.

If you have an alternative that doesn't involve drastic negative changes to our accepted way of life I'm all ears...


Re: Deaths are not the only metric

And how about you explain how we are supposed to keep the lights lit, industry operational, homes heated, electric cars charged and all the rest while not emitting carbon without it.

Wind is not reliable, has a huge physical impact, turbines with around a 20 yr lifespan (far less that a nuclear reactor) which aren't recyclable - it's not the answer.

Solar suffers the same reliability problem especially in the UK and has similar issues with physical impact. Making the panels relies on hard to obtain materials and energy-intensive processes - not the answer either.

Base load on the grid needs to be met by reliable sources - nuclear is ideal for this. Renewables are good for some things but there are needs they simply cannot meet.

If you truly want to get to zero-emission power generation it will need a combination of all of these technologies, as well as significant investment in energy storage to achieve it.

Major IT outage forces UK emergency call handlers to use 'pen and paper'


Fail to plan, Plan to fail...

We know systems will be compromised. It's fast becoming inevitable, no matter how comprehensive the security or how determined the management are when all is said and done defenders need to be lucky all the time, attackers only need to be lucky once. We need to start accepting that when we're designing, specifying and commissioning systems for critical infrastructure.

We need to look at enforcing diversity - multiple systems, multiple suppliers, multiple architectures, multiple access paths. Yes, that will introduce problems with interoperability but those shouldn't be insurmountable with careful design. We need to be mandating that level of diversity across the entire critical infrastructure - no one supplier, system or datacenter should be permitted to operate in more than a quarter of any given segment of critical infrastructure.

Monocultures are notoriously fragile (windows, I'm looking at you!) - they need to be avoided when dealing with life-critical services. Only by promoting diversity and actively discouraging monoculture can you both promote innovation and enhance the resilience of the system as a whole.

New Outlook feature: It freezes up when dealing with tables in emails


Plain Text

Round here, plain text has always ruled the roost. 72 characters wide, properly formatted signature separator, 4 lines of a signature at most and absolutely no damn HTML anywhere.

Yes, I'm old.

Ditching VMware over the Broadcom buy? Here are some of your options


Re: Grim on the low end.

Do have a look at Proxmox. It's robust, easy to manage from both GUI & CLI and frankly just seems to work. Converting the VM's can be a bit of a nuisance but it's a one shot deal - once it's done, it's done.

You get full clustering, failover, HA, backup and even distributed storage (if you want it, we didn't and went with a traditional SAN architecture). What's more, a single web-based GUI manages all the servers in the cluster. For us at least it's more than a match for Hyper-V and easily stands up against VMware.

And it doesn't cost a bloody fortune either!


I have to agree - we're in the process of shifting a whole pile of VM's from both Hyper-V & VMWare onto a common ProxMox platform. Relatively painless so far.

Micron aims 1.5TB microSD card at video surveillance market


Re: Oh great!

In quite a few scenarios recording at the edge (ie. in the camera) makes a lot of sense. Not only does it keep the network a lot quieter you'll often find the camera itself is far harder to reach/destroy than the NVR is.

Of course, the gold standard is on-camera, on-NVR and off-site but if you're that paranoid you already know that (and have the diamond mine to pay for it!)

Next six months could set a new pace for work-life balance


Explain to me, if anyone can, just how we can effectively operate a customer facing business on a 4 day week without increasing headcount (and thus costs) when our customers operate 5 days a week. I can't see how it's possible unless it becomes a legal mandate and everyone works 4 day weeks.

Then we get to the thorny issue of pay - 20 % less work without a drop in pay. How does that work? Do we carry out 20% less work but charge the clients 100% of their service contracts? I can't see that lasting long before the bean-counters start chipping away at it - "But you're only providing support for 4 days, why should we pay the same as we did for 5?". Before you know it, company income is down 20% while costs stay the same or rise....

It's a nice idea, in theory. In practice, it's simply not going to work which is almost certainly why none of those trials ever became permanent.

Elon Musk orders Tesla execs back to the office


Of timeclocks and toil...

Having come from an environment where the Timeclock was king and TOIL frowned upon I was determined not to allow such a culture to take root where I am now. I know first hand just how toxic a time-punching, clock-watching environment can be & just how creative people can get at undermining it!

We have never had a timeclock, no time recording and a flexible working policy that's focused on trusting the staff to get their work done, quickly and efficiently without us needing to breathe down their necks and monitor everything they do. It works. We have excellent staff retention, high staff satisfaction, good case-resolution rates and a very low customer complaint rate.

With that said, I do understand the desire to see staff back working from the office. It is considerably more difficult to manage remote teams as effectively. There are far fewer opportunities for ideas to form organically (at least two of our more successful products have originated from chance conversations in the office kitchen) and less chance for junior staff to learn by osmosis.Teams, zoom, conference calls; they all have their place but none can fully replace the office environment & direct personal interaction.

It's not impossible to do WFH long-term but it's a significant challenge, one we are still trying to find the best answer to. I doubt that answer will be a Musk-style edict to get back to the office full time but I can't see it being fully remote either - an acceptable middle ground will eventually emerge.

Zero-day vuln in Microsoft Office: 'Follina' will work even when macros are disabled


The 90's called...

They'd like their worm back please...

Son of Melissa??

Atlassian comes clean on what data-deleting script behind outage actually did


Sh*t Happens

No matter how many safeguards you build, checks you put in place or precautions you take the fuckup fairy will come calling sooner or later. The more systems you manage, the sooner she's likely to get to you - there is no escape.

That's why we have things like backup strategies and RTO's, so that when she does visit it's not a company-ending event. At least they've been honest about what happened and how long it's going to take to put it right. No marketing spin. No fluff. Just an honest 'we screwed up, sorry'. They should be commended for that at least.

Their lackluster RTO on the other hand isn't so easily forgiven....

'Bigger is better' is back for hardware – without any obvious benefits


Somewhere, if I dredge the depths of the storage boxes in my attic I still have a working Sinclair QL, complete with the original user manual. I suspect the microdrive cartridges are long dead though - they never were the most reliable of things.

I must dig it out sometimes and see if it'll still power on....

Millions of APC Smart-UPS devices vulnerable to TLStorm


Re: "the firmware updates are not cryptographically signed securely"

You are 100% correct. I've found myself having to explain networking basics (and I'm genuinely talking about basics here - networks, subnets, ports & protocols) to more developers than I can remember of late.

I wouldn't expect a dev to be able to design me a full blown enterprise network but surely it isn't too much to expect them to understand what a subnet is when they're writing network-aware code?

Do you know what TikTok is? Then you might make a good magistrate, says Ministry of Justice


Re: I looked into

Surely guidance is just that - guidance and as such the magistrate is free to ignore it if they see fit.

Thus those 'repeat customers' could well find themselves exhausting the magistrate's patience quite quickly and suffering the (probably well deserved) consequences..

I can almost hear them bleating about guidelines as they're led away to begin a year's stretch!

Zuckerberg wants to create a make-believe world in which you can hide from all the damage Facebook has done


Re: Oh for Gods sake

My money's on batshit.

(thumbs up for the Asimov reference)


If that's the future...

Count me out.

I'll happily retire to my little corner of darkest Scotland and live out the rest of my days far from the madding crowd of AR-Spectacled zombies thank you very much.

Reg scribe spends week being watched by government Bluetooth wristband, emerges to more surveillance


Re: Over-reaction

Most certainly not. Those who behave like that deserve the full weight of the law to descend on them from a very high height indeed.

Mobs, yobs and clowns do not make a totalitarian state. Pervasive, unending state surveillance and intrusion into people's private lives does.


Re: Over-reaction

I am not against controls and never said that. In fact, I clearly stated my support for controls while we were in the early stages of the pandemic and the disease was largely an unknown quantity.

What I am, very firmly, is against are ongoing controls that have little to no value against what is now clearly an endemic disease. What purpose does surveillance serve in the control of this illness? Given the vaccinated can still spread the disease, what is the value in vaccine certificates? Where is the proportionality test to justify these measures?

If Covid were significantly more lethal the picture could be very different. If we had no vaccine the picture could be very different. It's not, and we do.

The question remains unanswered - what is it about covid that means levels of state intrusion into our daily lives that were unthinkable 18 months ago are now suddenly not just acceptable but apparently desirable?


Re: Over-reaction

I don't, and I've 'peddled' no untruths at all. If you'd like to show me one I'll be happy to stand corrected.

I never claimed your life is worth less than anyone else's, nor did I ever say the vulnerable population 'cause me problems'. I'll thank you not to put words into my mouth.

To address the point I think you are trying to make, society cannot operate solely for your protection. It never has. People catch and die from all sorts of diseases all the time and nobody blinks. We don't shut down businesses, close borders, introduce surveillance or curtail liberties because of it. We manage the risk, individually and collectively.

Covid seems to have turned that on it's head, or more specifically the government response to it has. Over the last 18 months we've somehow gone from a nation who were solidly opposed to ID cards or pervasive surveillance to a nation who seem willing to welcome any level of state control and monitoring in the name of 'safety'.

I question why.

Why are we surrendering our freedoms for so little tangible benefit? Why are we tolerating this laser focus on Covid to the exclusion of all else? Why have we allowed ourselves to become so blinkered that we willingly ignore the collateral damage caused in the futile attempt to suppress Covid? Why do we now willingly provide far more information into our daily lives to the same governments we so rightly criticized for stealing far LESS information from us?

I DON'T value your life any less than anyone else's. I supported the initial measures to control the disease while we studied it. Those days are past. We have enough knowledge to effectively manage the risk without becoming a fear-riddled surveillance state.


Re: Over-reaction

Actually it's not. Factually there are many diseases roughly as transmissible as Covid and many which are significantly more transmissible.

Covid is not exceptional.

That's not conjecture, it's scientific fact. Please feel free to check for yourself, I provided one easy to use source already but there are many others.


Re: Over-reaction

I assume those handing out the downvotes are also the same people who had no objection to the conduct revealed by Snowdon, to facial recognition by police, to ID cards or to Apple's proposed scanning of the personal content on your phone.

There is no half way with totalitarianism. If we will not see the risks that are so clearly staring us in the face now then why were we all so outraged before?

Same authorities. Same problem. Different reaction.

Amazing what a few months of endless propaganda will do.


Re: We know who is most at risk from it

Agreed, vaccination is a key strategy and I'd encourage everyone eligible to get vaccinated.

It must however remain a personal decision and we must respect both people's choices and their right to choose.


Re: We know who is most at risk from it

As I'm sure you well know in every statistical group there are outliers. That's obviously little comfort to those impacted but it doesn't alter the fact that we do, statistically, know who is most at risk.

It also doesn't mean those not 'most at risk' are at no risk. The same holds true for almost any other illness you could mention.



The later stage response to the emergence of Covid has been a lesson in manipulation, coercion and power. Covid is NOT exceptionally deadly. Nor is it exceptionally contagious. It sits firmly in the bottom left quarter of the lethality vs contagiousness chart as seen here : https://www.informationisbeautiful.net/visualizations/the-microbescope-infectious-diseases-in-context/

It's new, and that (rightly) caused a rapid & extreme reaction. While so little was known about it extreme control measures (lockdowns, enforced quarantines, suspension of travel etc.) were justified. What is NOT justified is the continuation of such extreme measures for extended periods - well after we developed sufficient understanding of the disease to properly assess the risk it posed.

We now know how Covid spreads. We know who is most at risk from it. We know how to better treat those unfortunate enough to suffer serious illness as a result of it. We know how to limit it's impact through vaccination. We know that for the vast majority of healthy individuals it's an inconvenience. In short, we know it actually behaves very much like dozens of other diseases we've been living with an managing for hundreds if not thousands of years.

So, why are we suddenly so willing to throw away hard won liberties, to discard the limits that society rightly places on government power and hand over unprecedented insight and control of our daily lives to the same authorities that only a few short years ago we were condemning for invasion of privacy on a global scale?

We need to take off the blinkers and look long and hard at where we're going. I strongly suspect it's a place many of us do not want to visit.


Re: What do you want from your surveillance state?

How about not having a surveillance state in the first place?

Mankind has survived perfectly well for millennia in the face of far more dangerous pathogens than Covid and with far less options for vaccination or treatment.

UK VoIP telco receives 'colossal ransom demand', reveals REvil cybercrooks suspected of 'organised' DDoS attacks on UK VoIP companies


Re: Abandon Copper At Your Own Risk....

The likes of BT offering VoIP over it's own fiber is much less problematic (although not entirely without risk) as it controls the infrastructure end to end in that situation. It could, if it so desired, keep telephony traffic entirely separate from data, thus mitigating at least some of the risk of DDoS.

The chances of them actually doing that without a very firm regulatory imperative however are so small that I suspect I have a greater chance of riding in an electric flying car!


Re: This has been a known threat...

A 'private connection' (unless you're talking about a dedicated physical line, which for all but the very largest is utterly impractical) will not save you from a DDoS attack that swamps your provider's bandwidth.

Especially not when the underlying POTS network is gone and everything is IP based.

Much more work is needed before we start transitioning potentially life-critical systems such as telephony exclusively to the internet.

Microsoft sinks standalone Hyper-V Server, wants you using Azure Stack HCI for VM-wrangling


Re: Hyper-V role still available

Perhaps Veeam might like to look at expanding the number of hypervisors they support given the all-but-certain demise of Hyper-V in many SME's after this announcement.

'Hybrid cloud' AKA 'line Microsoft's pockets' is of no interest to the vast majority of our customers - those who want to go cloud have, those who don't/can't won't be railroaded into it.

If I'm going to have to pay for a host OS/Hypervisor than I may as well pay VMware & get a better product for my money. Free, there's KVM, XCP-NG & more (proxmox is interesting if sometimes a little flaky still).

BOFH: Despite the extremely hazardous staircase, our IT insurance agreement is at an all-time low. Can't think why


"they stopped bringing them in around the same time we started bringing hammers in"

I really MUST try that approach... theraputic & good for security!

Apple announces lossless HD audio at no extra cost, then Amazon Music does too. The ball is now in Spotify's court


Re: Yay!

Converting an analog medium into a digital representation and then back to analog again is, to put it mildly, less than ideal.

If you really want to hear it as it was live you need a good quality analog recording & quality playback equipment.

Digital is convenient but it's always going to be a compromise.

Train operator phlunks phishing test by teasing employees with non-existent COVID bonus


Re: If the email was from e.g., a gmail account...

Without the details of the message it's impossible to say whether it should or shouldn't have been spotted.

Doing (as I have to do on occasion) phishing training/testing for companies is a very fine line. You need to make the message as realistic as possible but not so realistic it genuinely cannot be spotted. You also need to consider the target audience - for example a message I'd send to a bunch of trained IT support people would likely only have one, hard to identify, indicator as I'd expect them to have a much higher level of awareness and skills than most. A message going out to a group of office workers/managers would have more and easier to identify indicators in it.

There is a lot of truth in the maxim 'Train hard, fight easy'


Re: Unfortunately,

Calling me up and then demanding I pass 'security' is one of my pet hates.

The lucky ones get away with a polite 'No, I don't give personal information to random callers'. The unlucky (or those who persevere) get the full on lecture/rant about just how stupid they are being.

I've even had one insist that because the caller ID was from their published number (they will remain nameless to spare the blushes of their IT/security team who I'm certain know better) that I had to give them my details. I don't think my offer to call them back from their own CLI was particularly well received....

OVH outlines three-point 'hyper resilience' plan after Strasbourg fire


Re: At last backups you will be able to use your backups

That really depends on what services you're using. We use bare metal servers. We install our own hypervisor on them and we back them up to our own, non-ovh facilities using normal backup tools.

We have contingency plans that allow us to restore to either AWS or Azure if necessary.

It's not really up to the provider of the DC to manage your backups. Sure it's nice if they will but there's no substitute for doing it yourself.


Re: communication


We are OVH clients, with a not insignificant number of servers hosted there. We had servers in SBG2. We also had no data loss and minimal downtime (what downtime we did have was largely my own fault).

OVH has the technology and network available to avoid building systems with a single point of failure. They have advanced networking capabilities if you want to use them (in our case, we've now added API access & scripts to our network monitoring to repoint IP's if a network goes dark).

The point is that the client has to use them. If they don't, they have only themselves to blame.

BC is more than just backup.

Openreach out and hike prices on legacy fixed-line products: Broadband plumber pulls trigger after Ofcom gives the nod


Re: We've switched to LTE

Ah yes, Edinburgh. The place where you might get ADSL, IF you're lucky. The problem there is the physical cabling - it's old & from what I've been told portions of it are actually made of aluminium and not copper which doesn't work with VDSL. Combine that with the almost herculean difficulty of replacing said physical plant in a city where you only need to look at traffic to cause a tailback and random 'conservation areas' making it difficult to install cabinets and you have the perfect storm.

Some areas are lucky and get Virgin Media service. Most don't Leased lines frequently carry stupidly high 'civil works' costs and thus are out reach for many.

Yes, I've fought that war.. many times. I feel your pain.

PS... Starlink isn't too badly priced although coverage is still patchy. 5G/LTE is workable but as you've found there are few options for routers (try Vodafone - the one they offer is probably the best of a bad bunch currently).

From Maidenhead to Morocco: In a change to the scheduled programming, we bring you The On Call of Dreams


Sun, Sea, Sand & Expense Account!

The scene, one dreary day in late march back toward the end of the 2000's. I was working as a lowly field tech for a now-defunct supplier of WiFi to some major hospitality businesses.The call, from a rather panicked account manager came at around 4 on a Friday - the local installation team had botched the job and the customer was threatening to cancel the contract, could I help?.

Calls like that weren't too unusual (the perils of outsourcing installation work to contractors are well known round here!) but the location of the problem certainly was - a top-end resort in Marbella, playground of those with far more money to burn than I'll ever see. Needless to say I didn't hesitate for longer than it took to make sure my passport was still valid and without further adoo a ticket was purchased for Monday. I was a little bit surprised to see it was an open return but put it down to the situation and thought no more of it.

Monday comes, things get stranger still. E-mail from the office with details of a hire car (we usually had to rely on buses or taxis!) that would be waiting at the airport. Business class seat on the flight too (apparently that really was all they could get!) - this was shaping up to be a good job! Got to the site to be met by the general manager who showed me the mess (and it was a mess!) left by the contractor - nothing installed, cables in a heap..a pigsty. He was naturally a bit p**ed off, especially since they were starting to get busy. After a quick inventory & walk around the complex (it's BIG.. lots of blocks) and I figured 3 - 4 days work would do it. Went off, found the GM & told him what I was proposing... which is when it got REALLY interesting.

Apparently, I could only get access to rooms when they were vacant - not just unoccupied, vacant. He & I looked over the booking calendar and worked out a plan... it'd take almost 3 weeks to finish the job. A hasty call to the account manager, who called his boss.... and the job was authorized. The GM was so pleased that it would get done before some VIP's were due he comp'd the entire account - accommodation, food, beer, the lot!

And that is how I ended up spending three weeks in a 3 bed suite of a 5-star Marbella resort, fully comp & working at most 2 hours day. I actually flew my girlfriend out for the second week...

Best call I've ever done!

Missile systems software dev leaker has sentence almost doubled after UK.gov says 4½ years was too soft

Big Brother

Re: Digital Era

You are assuming the motives of those issuing such warrants are benign. In many cases they are but when they are not the impact can be enormous. Encryption provides a way for ordinary people to take control of their information back from the grasping claw of the state - there is generally no need for the authorities to read every e-mail, record every website, monitor every phone call and read every document a person writes but todays technology permits just that, in real time, without the victim's knowledge and on a massive scale.

That is what is driving the uptake of encryption - state overreach and pervasive surveillance. If it wasn't for that there would be no driver to make encryption easier to use or to make it default for normal communications.

Proper encryption has been around since the days of PGP in the 90's but was always hard to use and so remained a niche product, then along came Snowdon and the dawning realization of just how much information was being hoovered up and suddenly there was a huge public interest in encryption technology and lots of effort went into making it easy to use, accessible & almost the default for everything.

That leads to the situation we now find ourselves in. Encryption is ubiquitous and very effective - by it's very nature you can't have encryption that only 'authorized' people can break - it's either secure or it's not, there is no middle ground in mathematics. So the authorities resort to laws compelling people to produce keys or face punishment - however distasteful the idea may be it's probably the best solution there is going to be and it carries one huge benefit over any technical solution - a properly drafted disclosure law (and I'm not saying RIPA is that!) will ensure that such requests get proper judicial and public scrutiny, hopefully limiting any attempts to use those provisions unfairly or without proper justification.

It's not ideal, but it's the best we've got.

OVH founder says UPS fixed up day before blaze is early suspect as source of data centre destruction


Re: Is there a lesson here about putting your eggs in one basket?

How badly impacted you were depended largely on what products from OVH you were using. Their cloud products did not seem to be badly impacted at all (at least not that we noticed) but they also provide a number of other services including hosted backup & the typical 'rent a server' product.

We don't use the backup services so I can't comment on those but we do make extensive use of their rental servers & associated networking. A number of our customers want the benefits of not having to deal with local architecture but also aren't ready to go full public cloud and OVH's rented server/network options are an ideal intermediate step for them but as with anything you have to design it right and use it properly to get the best out of it.

It is remarkably easy (and surprisingly inexpensive) to build a properly resilient architecture using OVH's services. By properly resilient I mean live replication across multiple DC's in multiple locations so you won't get hurt if even a whole DC goes south. Many probably don't bother, trusting that 'it'll never happen' - well, now they know better and that's not OVH's fault. Hard lesson, but the worthwhile ones usually are.

We did have some downtime - one of our client's netblocks was announced via SBG so despite them having a functional replica site they couldn't actually reach it. It took 24 hours to get that sorted out but a sizeable part of that was down to the fact I wasn't as aware as I should have been of all the capabilities on offer to us. After OVH support helpfully pointed me to their API & told me that we can use it to control where on the OVH network our blocks are announced in near real time the client was back online in less than 10 minutes.

We're now adding the OVH API to our config tools so we'll be ready if anything similar happens again....

In this business the learning never stops :)

Talk about a Blue Monday: OVH outlines recovery plan as French data centres smoulder

Thumb Up

Re: It's not the incident that's important

And credit where it's due, OVH got in touch just after 9 this morning and the issue was resolved by 10am.

Customer's happy, we're happy - can't really ask for more (and again, the level of flexibility in the OVH network is really quite surprising for the price point)


It's not the incident that's important

It's what you do afterwards that really counts. We're OVH customers, with servers in the destroyed SBG2 DC. We have redundancy (I've been playing this game far too long not to have!) but that depends on OVH's network actually passing packets correctly, which is isn't right now. I'm perfectly willing to give their support teams the benefit of the doubt for today but that runs out at 9.30am tomorrow because that's the time when I need to make the call on whether to initiate an expensive bare-metal restore to Azure.

If we DO need to do that it'll be entirely down to OVH's lack of support and not down to the fire. It'll also be the end of our relationship with OVH.

Accidents happen but falling over afterwards is avoidable....

OVH data centre destroyed by fire in Strasbourg – all services unavailable


Re: DR Plans

We have a (tested) 48-hr RTO for a full bare-metal restore from backup to an alternate provider (Azure in this particular case since these are all Windows VM's). Actually activating that process is expensive so the decision to do it will be made tomorrow depending on whether or not we can get any traction with OVH support.

To be honest, if we DO have to do it we're unlikely to return to OVH. Accidents happen, what's a lot more important to us is how the aftermath is handled (or not handled...)


Re: DR Plans

OVH has some very clever networking capabilities that in most situations make life much easier for us poor buggers who have to make it all work without upsetting the bean-counters too much.

Said capabilities make it easy(ish) to have a primary site in one DC and replica in a completely different country. You'd think that would be enough DR for just about any reasonable situation... except...

That same clever networking is the Achilles heel. Long and short of it, it's broken. That DR site is now sitting there, up and laughing at my futile attempts to get it to talk to the world.

OVH support, naturally enough, can't give any sort of clue when this is likely to be fixed. I can't blame them for that - I really wouldn't want to be working for OVH's network engineering team right now.

Next up, restore from backup - to another provider!

Lesson : there is NEVER enough DR.

Just when you thought it was safe to enjoy a beer: Beware the downloaded patch applied in haste


Re: Army Email time

I vividly remember spending hours clearing masses of 'snowman eating child' animations from a poor, struggling Exchange server one Christmas. Not long after disposing of 'I love You' either.

Shame I couldn't dispose of the users quite so easily!

Three rips up call centre outsourcing contract with Capita 2+ years early


Re: So a piss poor service

Exactly. Customer service being piss-poor was the reason I left three two years back. Their call center operation was utterly horrendous, one of the very worst I've ever encountered (and given I've spent more time yelling at BT than I care to remember that is saying something!).

It doesn't look like I'll be going back anytime soon...

Beware the fresh Windows XP install: Failure awaits you all with nasty, big, pointy teeth



Evil little buggers. Got hauled out to a warehouse one saturday because 'the wifi wasn't working right'. A quick check revealed that about 70% of the AP' (all mounted about 30ft up!) were offline. Switches & other ground based components checked & ruled out I commandeered a hydraulc platform and went in search of the fault..

There, up in the roof among the girders were the culprits - a family of grey squirrels who'd taken to gnawing on network cables.

I've no idea what it cost them to get the place recabled but I did suggest investing in a shotgun!