Manchester Arena attack: National Mutual Aid Telephony system failed Vodafone has come in for criticism for the "catastrophic failure" of its National Mutual Aid Telephony system (NMAT) in today's review of the deadly suicide bombing at the Manchester Arena last year. Twenty-two people were killed and over 100 injured in the attack at an Ariana Grande concert at the northwest England indoor …
the reason for subcontracting is that you lack the skills in house.
The problem with that, is that many time you need the skills to write te contract. So either it makes no sense because you have the skills, or the contract is probably wrong.
the next thing would be to outsource the contract writing (it is already done), and also to check the compliance to the program.
At that point, I guess you dont need employees...
the reason for subcontracting is that you lack the skills in house.
In which case you shouldn't effing well submit a tender. And the Contract Principal should ask bidders quite specifically "Do you currently have the necessary skills and other resources to fulfill this contract (if it is awarded to you) without recourse to subcontracting".
Outsourcing has been going on for so long now that this sort of blunder should simply not occur.
Ever.
Absolutely, 100% agree with this sentiment. Almost every single time that outsourcing is contemplated, be it government or private, the first question is "how cheap can you do it ?". In fact it rarely gets beyond that. If they ever do question the credentials it rarely figures in the final outcome.
It's almost always bean counters asking the questions rather than someone who actually knows what questions to ask and has a chance of understanding the answers. The requirements are invariably stated wrongly or are incomplete and almost the second the contract is signed the cost starts flowing back up the chain for "additional work not on the contract". You end up paying more than if you'd just gone to the experts in the first place for a worse service. Can you imagine getting a builder in under those rules ?
me: I need an extension built, how cheap can you do it:
builder: depends, how many floors ?
me: err, some floors.
builder: probably a few thousand pounds.
me: great when can you start. Oh by the way, you are a builder aren't you ?
builder: not as such but my mate once had a fence put up.
me: great, see you Monday.
builder: might be Wednesday my mate will need to put it out to tender.
Typical gap between what was promised by the A-team who bid for it and the F-team who were responsible for delivery (or in this case the third-party it was subcontracted to).
Outsourcing - for when you absolutely, positively can't bring yourself to give a shit about your service.
Falls on its arse?
Sounds more like a faceplant to me.
"Content Guru"?
You took a national emergency communications system and gave the responsibility for managing it to a company who clearly has no idea what they're doing? I'd love to see how they were chosen. Backhander to the Vodafone management is my guess.
OK firstly.
1. Review all Vodafone contracts, find out how many have been subcontracted without notification. Look to end all of those.
2. Find out who was in charge of the Vodafone contract here, sack them for a) not preventing sub contacting as a clause in the contract and b) not making anyone aware of it happening.
3. Tax Vodafone properly on profits made in the UK, seeing as they apparently are cost focused and were saving some pennies by sub contracting work.
4. Start the same investigation into G4S, ATOS etc.
Yes Minister...it's satire, not a "how to" manual.
Satire is humour applied to criticism of that being satirised. The foundations are reality, without a real subject to examine critically there can be no satire. YM was generally reckoned as being particularly well founded. A boxed set of YM and YPM together with a copy of "How to lie with statistics" should be the basis for any education in civics.
Speaking of which, have KFC managed to pull itself together again? Have read about that big hoohaa, most probably some beancounter trying to cut costs in order to get a bigger bonus.
And this is a prime example of why outsourcing is a big NO.
If properly managed, documented and handed over, then maybe. Not a yes, but a maybe, because you really don't know how that company will handle pressure when it arrives unexpectedly.
I'm not defending the guilty parties here, or saying that the situation is anything other than s**t, but...
The tone in the comments here is that outsourcing is always wrong. Often it's a means of the prime contractor doing something more cheaply but, for the sake of balance, I can think of times when outsourcing is legitimately better. In fact I was talking to a friend of mine just the other day about this.
If I'm something services from someone, I'd like to thing that I'm getting the best things possible in return for my money. If there is a specialist subcontractor who can do something better than the prime, then I'd like to see that aspect of the work subcontracted out.
Example - Cosworth make better engines than Ford, so that's why Ford subcontracted engine manufacture to them for some models of Sierra and Escort. (I fear I may be showing my age with this example)
Bottom line. Subcontracting is not inherently bad. Doing it just to cut corners and save money is. Doing it in order to get a better quality specialist product or service is acceptable.
(great name btw) Outsourcing is often used to allow a company to focus on its core business, what it does best. But it does rely on finding a trustworthy and capable partner, and puts an onus on the contracting company to know enough about contract management and also enough about what is being provided, technically, to ensure they are getting what's required. As with most things in life, going with the cheapest supplier is usually a mistake.
to allow a company to focus on its core business,
When companies say that they often seem to be breaking their routine process into a set of discrete activities then elect to label some as non-core, as far as I can see. It's less about buying in gear boxes and more about getting rid of the wages department. Which is fine if a payroll company can do the job cheaper, I guess. Until the staff don't get paid. Because cheaper isn't always better.
(Wife works for local authority who outsourced pay to our old Friend Crapita with the usual outcome).
@ Hans Neeson-Bumpsadese If there is a specialist subcontractor who can do something better than the prime, then I'd like to see that aspect of the work subcontracted out.
The above prompts the question "why didn't the subcontractor submit a bid to be the primary contractor then?"
While I don't actually disagree with you your example (of Ford v Cosworth engines) doesn't actually match the case you argued. Ford make engines, but contracted some specialist engine manufacturing to Cosworth. In that case Ford was the Principal, and Cosworth a Contractor. Ford was not a Contractor who subcontracted work to Cosworth.
The above prompts the question "why didn't the subcontractor submit a bid to be the primary contractor then?"
In all my experience of contracting and sub-contracting, it's a matter of scope. A subcontractor often has skills in one area, but not the whole piece. Example, prime contractor takes on responsibility for delivering a system/solution - they know a bit about security but are not specialists....so they subcontract the security element to a specialist company who live, eat, sleep & breathe security but who don't have wide enough scope to cover other aspects of the solution, or even have the scale to manage another bunch of subcontractors who do.
@Naamsgenoot [the other Hans]
If I'm something services from someone, I'd like to thing that I'm getting the best things possible in return for my money. If there is a specialist subcontractor who can do something better [..]
Outsourcing is never about better, it is always about cheaper.
If you think about it, assuming you are (or somebody in your company is) good at hiring good people, how can outsourcing possibly be both cheaper and better ? Ok, granted, something highly specialized and of which you have several highly specialized outsourcing companies with a proven track record, then, maybe ... but here, we have public emergency with 30 years experience ... very few private companies can come close to that ...
If you think about it, assuming you are (or somebody in your company is) good at hiring good people, how can outsourcing possibly be both cheaper and better ?
It depends. Unless you are very large then it is often very wasteful to do everything in-house - and given that I've had (cough years) of being part of that "provide the skills it's not worth the customer having in-house" I'm glad that this is the case :-)
So, for example, you are a small estate agent with (say) ten people spread across 3 sites. Do you :
a) employ someone with a remit to look after the IT (as part of their job) and give them the time to keep up to date with everything they need to know to fully support and keep up to date your systems.
b) outsource to someone like my previous employer who employs multiple people (so has different people who can be experts in different sub-fields) and who can spread the cost of keeping those people up to date across all their customers.
Same applies to payroll : Do you put the time and cost into keeping one person (and what if they are off sick ?) up to date with all the legislative changes, software updates, etc, etc - or do you outsource to a company that specialises in doing this and so can spread those costs across multiple clients, and has multiple people who can do the work and so remove the risk of having just one person able to do it.
How many times have you heard something along the lines of "only Fred knows how to do that" - and Fred is not available ? I know I've heard it many times in various forms. In one of my previous jobs, I was that "Fred" for a number of areas (in-house IT and some building services) and from a personal perspective it's not healthy as you can never really "switch off".
Even in big businesses where they do have the scale to justify employing teams, there's often a justification (other than just price) for outsourcing.
Contractor ( in this case the government) pays a company to provide a service. Said company then pays another company (some of the money) to do the work. Which raises all sorts of questions, including;
1.) Who was responsible for quality control?
2.) How was it specified?
3.) How was the contract priced if the subcontractor could ( in theory at least) do it cheaper - and cheaper to the extent that it could support not just one, but two sets of management and profit? and so
3a) Was there a fair competition to get the initial contract and
3b) How did the subcontractor get the job? On what basis?
Do they happen to do training during normal office hours, when the people supplying the service have enough staff available to support? If something kicks off after 5pm then they could be down to reduced/nightshift workforce who don't have the requisite knowledge, hence the resulting fiasco
Thursday 14/6/2016 - All day. "Exercise Triton II"
https://www.manchestereveningnews.co.uk/news/greater-manchester-news/emergency-training-exercise-manchester-police-11564284#ICID=FB-MEN-main
Tuesday 10/5/2016 - Evening - Suicide bomber planning trafford centre.
https://www.manchestereveningnews.co.uk/news/greater-manchester-news/trafford-centre-terrorism-training-video-11310160
I don't think the time matters because it's not like you can arrange the time prior to something happening for real. That's why I would ask why this wasn't tested then?
The do the training out of normal hours (don't want to scare the other shoppers when some person from the casualties union appears in full makeup http://www.casualtiesunion.org.uk/ )
you can bet your life though all parties know in advance and they have people on standby in case things go awry
Whilst they'd have trouble fighting their way out of a paper bag,
Vodafone’s CEO has offered formal and personal apologies to the those affected by the fault and assured the Home Office that migration to a new platform will provide the necessary fall-back.
looks like they are looking to solve the problem by persuading their customer to pay for an upgraded system.
Their customer would be best to also specify the configuration of the new system too.
I kid you not but Vodafone currently won't process our orders unless we tell them the config to apply to their routers, and even then most of the info fails to get into their commissioning engineers "MOP" which is written by VF's outsourced change writing team despite the detail being in the request spreadsheet we also populate for them!
tossers!!
So you go to commission a circuit and it doesn't work for some reason, it might be trivial, but you can't raise a fault for it as the SD claim its not live, yet they're billing for it. Your service manager is useless and states you need to pay for a change request to fix the fault on the circuit they should have delivered according to your specs. You get it sorted and then find the UC isn't working properly, you investigate and find out that they didn't apply the QoS you asked for and their reason is that you didn't put the config (yes actual ios commands required!!) on the order form and change request.
Learning from your previous mistakes you add all that stuff to your new orders, you then find the new circuits your cutting over still don't work. You eventually get the config of THEIR router, usually from the engineer from the night of the failed cutover, or reluctantly from the SD, find their mistakes, complete a new change spreadsheet with the corrections in it, deal with their outsourced config writing team who build the MOP's, eventually you get your change date and at the time of cutover discover the MOP the assigned engineer has doesn't include half the info from the change spreadsheet. You do get lucky from time to time and get a great engineer who realises the MOP is junk and cuts through the crap to get the change sorted, but sadly they are becoming few and far between.
There are many many stories of woe, i should probably send in for an On-Call or who-me.
infuriating is the most polite way of expressing this.
Silo's and VF don't mix.
Whilst they'd have trouble fighting their way out of a paper bag
That's my experience with them as well.
When I started at my last place, they had an internet connection from an outfit called YourComms, which IIRC had originally started out as Norweb Telecoms. YourComms was borged into Thus - with the loss off some knowledge as usually happens in these borgings - but still a decent outfit.
Then Thus got borged by Clueless and Witless and service went distinctly downhill. Finally C&W got borged by Vodamoan and it got worse again.
One day as we sat in our office, the internet went down - and after initial checks that it wasn't our end, I was lucky to get through quickly to their support desk and log a fault. 10 minutes later our connection came back on, and we found that a major site we managed services to was still down, and stayed down. By now their support system was in meltdown. The cause ?
Well this supposedly top tier communications company had a networking centre in London, and there had been a very simple problem - a single circuit breaker had tripped. Unfortunately, this supplied one of the power supplies for the kit, and the other one had failed. This highly professional tier one outfit had no monitoring that told them a power supply had failed, it had no monitoring that told them power had been lost to the other power supply, the diesel generator didn't fire up because there was still mains to the building - but no problem, the batteries still worked. Of course, with nothing telling them about these problems, and nothing telling them about the reducing battery voltage, they did nothing until the batteries gave out and the brown stuff properly hit the proverbial fan.
No problem, just flick the switch back on, and let the routers etc boot up again. Yup, that was what brought back our connection - but there's more. A 'kin big router didn't boot up, spares were brought in (and Cisco engineers), and guess what .... they didn't have a working backup of the config ! It took them THIRTEEN HOURS to get our other services back on - losing a complete working day for the cllients, some of whom have staff paid more per hour than I earn in a week.
To their credit they did provide a report of what had gone wrong - and promised to "update their monitoring". But really, needing such an event to find out you have suck basic monitoring in place - sadly this seems to be the level of collective professionalism at Vodamoan (hence the icon). Not having proper config backups - second vote for the icon. Apologies to those people "at the coalface" who have been good within the constraints imposed by the imbeciles further up.
We looked at getting a VoIP service that was underpinned by them. Their software was utter shite at the time, and given everyone's reluctance to accept that there was an issue with it I don't expect it'll have improved...
I really wouldn't have wanted them in charge of an emergency system.
"Vodafone’s CEO ... assured the Home Office that migration to a new platform will provide the necessary fall-back."
Why is it still Vodafone making this assurance instead of some other company?
"The report recommends that the Home Office ensures henceforth that it has guarantees that disaster recovery arrangements are put in place to avoid the failures that plagued the NMAT on the night of the attack."
I'd like to think that the HO realise that disaster recover arrangements aren't actually in place until they've been properly tested and passed. And that they only actually remain in place whilst they continue to pass regular tests. If those conditions aren't met they're not disaster recover arrangements, they're just words. Even more importantly in this situation the prime system is itself a disaster handling system and subject to the same requirements.
Reading through the report it appears that the system had been used for a previous incident in another force and not made ready for re-use.
Having read the report, or at least large sections of it, I wonder how the RUC managed 40 years or more ago when they may have had simultaneous incidents on the go. I suspect it wasn't a matter of trained Gold, Silver and Bronze commanders as everyone at all levels knowing what to do through rather too much experience.
Was this failure the cause or a significant contributing factor to the firemen arriving two hours late ?
DISCLAIMER: the following link will take you to a lefty site:
https://www.theguardian.com/uk-news/2018/mar/27/manchester-arena-bombing-fire-service-arrived-two-hours-late-says-report
PS: Honest question!
No, I can't see that the supply of an 0800 number for relatives to call to get information would have had any bearing on the response time of fire or ambulance. IIUC that was due to the police wanting to ensure that the area was safe before allowing emergency crew into potential danger, BICBW.
Pretty much every company outsources some part of its business. My company outsources our deliveries (would not be cost effective to maintain a Worldwide fleet of trucks and aeroplanes to deliver our products to our customers). We also outsource our injection moulding and PCB manufacture (volume not high enough to supply permanent work to an in-house facility). We also do not run our own telephone company or ISP, though we do run our own server farm.
ISTM that the first outsourcing that took place was when the government outsourced to Vodaphone rather than keeping it in-house(ish) by using BT.
And that's the point. The gap between buying in specialised services or components that you can't realistically provide yourself and buying everyday services in from a supplier who can somehow provide these things at less than it would cost you while paying the extra admin/management costs and making a profit on top. If they are that much cheaper then someone somewhere is cutting corners.
If they are that much cheaper then someone somewhere is cutting corners.
Not necessarily. If the bar to entry is high enough, and the task small enough then outsourcing makes great sense. As described above - worldwide delivery. Not feasible to be done by every company individually.
A contentious example around these parts - cloud computing. If I need a small server to do my own stuff, and it needs to be permanently available on a fast internet line, it's generally better / cheaper for me to grab a VPS from (say) Rackspace than to buy a 1U Proliant (with hot-swap RAID, redundant PSUs etc) and colo it in an ISP datacentre. And even moreso over getting a RIPE allocation, multiple lines into my premises with BGP coming out of my ears, generator out back, and a second server handy in case the first one falls over...
For a telco, this system should be easy to set up. It should be well within their normal excess capacity, so it makes sense for a telco to handle it on behalf of the government rather than the government spending my tax money trying to *be* a telco. Why a telco then subbed it, though, is beyond me.
"We have been running the National Mutual Aid Telephony service for the Home Office since 2009. It provides an 0800 number and call handling solution for police forces during a major incident, and has been successfully deployed on numerous occasions"
^^ completely pointless "look at the shiny thing over here, don't look at the car crash" distraction paragraph
"However, any failure is unacceptable"
^^ acceptance of the above.
This country is truly screwed isn't it. The older Reg readers here (including me) at least have had chance to build up assets, knowledge and experience. All the generations below us have done is build up their social media skills - and when the point comes to stop talking and start doing, most of them haven't a clue. Things will only get worse.
The real scoop which no news outlet has picked up on yet is how the HELL Vodafone contracted this service out to Content Guru.
I have worked in Telecommunications for 20+ years and Content Guru are a bit of a running joke within the industry. They are known for producing half baked software and then pushing it on to anyone who will buy it.
Their main product is for call centers!!! How did this happen!!
Its almost criminal if Content Guru provided this service to the police. They should be NOWHERE near something like this.
I think it was a well known mobile provider that got a contract for pagers for RNLI responders then discovered they had no actual cell coverage over much of the UK in the area's where the pagers were needed. Last century this was, and the reason the RNLI only stopped launching Maroons recently... BT had the contract for years, their pagers even worked when they got wet...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
