14-hour+ global blackout at Ingram Micro halts customer orders Widespread outages across Ingram Micro's websites and client service portals are being attributed to "technical difficulties." The outages at Ingram Micro, one of the world's biggest IT distributors, began at around 2000 UTC yesterday, according to Reg reader reports and social media. The distie turned over revenue of $12.28 …
"Is there a potential that they'll have access to client tenancies?”
There is always some potential. Some posters on the ReadIt thread claim that ‘no; as it’s on a separate system’. But, with the lack of any sort of communication from Ingram as to exactly what has happened, we have no way of telling. Just how separate is this ‘separate system’, is there really a ‘separate system’, could it have been compromised as well, has it coughed up credentials to the hypothetical attackers?
Just like nature, the internet abhors an (information) vacuum, and will fill it with speculation and worst-case scenario claims. Personally, I suspect that it is unlikely that any putative attackers can gain access to client tenancies, but until we know more and Ingram makes a proper statement, then it can’t be completely ruled out. Maybe if you are an MSP using Ingram to manage tenancies, then just keep a close eye on them?
Absolutely. They're trying to contain the spread of something...and I would go as far as to say they're probably using AD.
Fucking Active Directory man. It needs to be taken behind the fucking barn and shot at this point.
We're at the point now where I'd rather take the pain of managing decentralised logins over the risk of a malware outbreak. At the very least with a decentralised architecture, you know that it's much harder for malware to spread. Yes, users having to remember several different credentials is a pain in the ass, but that is what password managers are for.
There's no simple answer to this other than planning for the eventuality and being as diligent as possible. Decentralised has pros and cons. Unless automated for users they will not all take care so you swap blast radius for individual system security. I'm sure the answer that will be pushed by government after this has happened a lot will be biological and centralised id which is an even bigger problem but the answer government wants.
Man, I'd sooner have a smaller blast radius, knowing exactly where ground zero is than having no clue at all and having to deliver yet another company wide training session on why opening attachments from strangers is bad mmkay.
It's 2025...email needs to go...we should not be sending attachments to people anymore. It's dumb, high risk and will eventually be a problem for everyone.
Scrap. Fucking. Email.
We should stage a global IT walk out in protest of fucking email.
Apparently you aren't creatively thinking if that's the only explanation you can think of:
They've run out of money. The C-suite spent it all, are taking their golden parachutes and running for the hills. "Don't connect" in this scenario means "Don't bother trying, we're now DOA. Get your resumes ready."
Website says Cybersecurity Incident.
‘ Another individual claimed that staff at Ingram's service center in Bulgaria, which covers a large chunk of European sales, have all been sent home and told to keep their laptops disconnected amid a total internal system shutdown.’
Another cheapskate offshore-er… whose savings will be outweighed by the costs.
I'd just like to make it clear that even though IM made my account payment with order due to lack of usage (after about 30 years of doing business with them) I am NOT respobible for their online outage. I feel a bit sorry for them really. Tons of stuff on their webshop is never in stock and the lead times can be very long but they do try.
If you are shitting yourselves over "critical... etc" during a one day outage, your a bunch of fucking morons.
If they have expired / died well sucks for for you for a lack of planning.
If they are expiring / required in the next week, we'll it sucks for a lack of planning.
Or are these the same sort of Drama Queens that think the world is ending because they had a power cut at home for 15 minutes.
If they are expiring / required in the next week, we'll it sucks for a lack of planning.
No, some of the screamers may have done extensive planning, and are now screaming because their gambling plan went wrong.
Just-in-time delivery can easily transform into not-in-time delivery.
The screamers' "plan" may have been, "charge the customers for products up-front, don't order/pay for the product until the last possible moment, and use the customers' money in the mean time as a free loan, which we can milk for interest, or use temporarily to cover some bad checks we had written to keep our struggling business above water."
"Just-in-time delivery can easily transform into not-in-time delivery."
Something more people need to remember when picking the most profitable or lowest cost route. Let me make a prediction; sooner or later government will have a centralised id service. One day it will go wrong or get attacked and if we have been stupid enough we will not be able to get money which means anything including food. People will go crazy, loot and kill once they get hungry. The more sensible will revert to bartering. If you think this mad, last time they tried it the vision was for everything to use that id. When pointed out the amount of resilience (cost) required to stop the whole country being destroyed in one fell swoop they weren't prepared to pay. I.e., an absolute minimum of 4 datacentres, diverse systems, highly secure with military defences, probably resistant to bunker busting bombs or bombs in lorries etc. There were no hypersonics at the time. Then how does the country access; you'd need layered access and high security for some of those pops. Then there are always some special people whose accounts need to be well hidden even from government employees.
If they have expired / died well sucks for for you for a lack of planning.
If they are expiring / required in the next week, we'll it sucks for a lack of planning
Take it you've never worked for an organisation where the Finance bods sit on everything until the very last minute...
The techies can plan perfectly well, give management/procurement all the right paperwork, weeks ahead of time...and they'll still sit on it until 11.59 for "cashflow purposes", seemingly regardless of operational risk.
Sounds more like they have a good accountant. Taxes are paid on profits, not turnover, so companies try to make sure the money ends up were it needs to be (including various pockets) while appearing to make only a minimal profit, or even a loss which can then be used to justify tax rebates.
So here we are more than 24 hours later and still the website is ‘in maintenance’ and the entire company appears to be dead in the water.
Now this it either a self-inflicted cock-up, in which case they would seem to be incompetent and hence why would anyone deal with them; or; it’s a ransomware attack* and they just don't want to say
* It will be a ‘sophisticated attack’ something that absolutely nobody could possibly have prevented; it absolutely won’t be that some C-suite twat clicked on a link in an obvious phishing email, will it?
This is big isn't it. This and Connectwise being in another certificate mess is giving me lots of Reddit stuff to constantly refresh!
I've just had my Weekly Account Balance email from their im-cloud subsidiary, but of course can't log in to anything or make changes to Microsoft 365 licenses for customers whom I resell to via Ingram.
I am split something like 70/30 pax8/Ingram since about a year ago so could still add additional new subscriptions but this is still a problem.
Encrypting on a weekday night isn't the usual modus operandi is it but I guess they saw their chance. Oh wait, 4th July holiday... :facepalm: Christ.
I wonder if it will take Ingram as long to sort their stuff out as it did Marks & Spencer in the UK, who got hit at the start of a public holiday in April of this year? Two months of chaos, and an estimated £300m hit to profits, and a circa 20% loss in share price that's still not recovered.
4th July, also happens to fall on a Friday, so long weekend.*
If, if it a ransomware attack, (and too be fair this is all just speculation, but as the people on the ReadIt thread observe, the length of time and complete lack of any announcement from Ingram is screamingly suspicious), this will be exactly the best time to launch it; fewer staff (presumably) available, only real option is to shut everything down, wait until staff are back in and then start investigating what was hit and what the damage is.
The other possibility is that this a catastrophically failed upgrade (their XVantage platform, maybe), but why on earth would you schedule an upgrade just before a long weekend holiday? It doesn’t make sense!
But either way, there will be a significant knock-on effect if not sorted very quickly. As mentioned above in this thread, what do MSPs who have arranged M365 tenancies or Dropbox licences via Ingram do if they can’t manage them?
* Yes a long weekend in the US only, oddly enough here in the UK we don’t celebrate that date ;)
* Yes a long weekend in the US only, oddly enough here in the UK we don’t celebrate that date ;)
I wonder where we'd be if we hadn't lost that fight? A couple of decades ago I would've thought the world would be better if we had continued to rule. Now I'm not so sure the direction of travel in the UK looks bleakly dystopian.
at the end of the day, they might not be my favourite company either (nothing personal really, but that's the problem I suppose... nothing personal between us), but I sympathise with the sysadmin who might be feeling like they could have prevented this. I have been there and it is not a good place to be.
OTOH, if the story I heard about the IT team being soon-to-be off-shored is true, then I'd feel quite a bit different about the situation.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
