3CX thought supply chain attack was a false positive The CEO of VoIP software provider 3CX said his team tested its products in response to alerts of suspicious activity that was later found to be a supply chain attack, and assessed reports of issues with the software as a false positive. We noted earlier that 3CX confirmed its software had been tampered with a week after users …
PM: We were tipped off that our app is affected by a supply chain attack. We need to put the development team on looking into this.
PHB: We're fine, i checked our app on Virus Total - carry on with the scheduled development, the new colo(u)r scheme the marketing department asked for is more important.
We got an email about it on Saturday as a customer so they have sent something out finally. Their response has been pretty typical of how I'd expect them to act, their CEO is a liability and probably was good in the early days with a smaller product and smaller clients but he doesn't understand Enterprise applications and is often quite rude to his partners and suppliers when they point out issues. I hope this incident brings him down a peg or two!
That'll probably be because he has a proven track record of acting like a petulant child if someone dares to disagree with the sudden removal of a feature or make an even remotely negative comment about something publicly. Multiple cases can be found, where Nick Galea personally banned 3CX resellers/partners from their forums and then terminated their relationship:
https://www.reddit.com/r/3CX/comments/xev0u5/my_3cx_partnership_deleted_and_all_linked_clients/
https://www.reddit.com/r/3CX/comments/w7tyg7/anyone_else_ever_been_fired_by_3cx_as_a_customer/
https://www.reddit.com/r/3CX/comments/112gxj9/furstrated_partner_with_3cx_management_issues_is/
https://www.reddit.com/r/3CX/comments/126h7vy/3cx_ceo_tells_partner_to_fck_off/
You don't have to look hard to find multiple cases of this. Gives you some idea of the mentality/attitude of the CEO of a company turning over tens of millions of dollars.
That combined with them now deciding to start directly competing with their long-established partner network by selling direct to the public (Which they said they'd never do) and e-mailing their resellers' end-customers directly with information on these new 'direct with 3CX' services has pissed off swathes of their partners and understandably so.
Agreed, long term 3CX customer here with multiple installs globally. They may call themselves "Enterprise" ready, but they're not. Customer created workarounds and third party software make the platform usable. Changes improving the situation only come after long periods of customer / partner complaints. Adding the BLF fields to their web software has been requested for at least five years and only being done now because the spotlight is on them.
Their system does not work with Exchange DAG environment, you need a standalone email server. No multi-office conferencing, no multi-office queues, no central management of multiple installs. Reporting that cannot be amended and has not been touched for at least three years, if you have an account that uses the management interface, it cannot be seen by others. They forced the super-admin system account to be tied to a phone number so now it is accessible through external web interface where it used to be impossible to connect on that account unless you were internal.
They make changes to their system no one asked for and focus on BS like Facebook integration instead of basic logging of who did what and when. Now, they've amended their forums so you can only post if you have a registered 3CX install against that login. Sounds good, but I use a different email to stop technical forum posts going to the entire IT team (Do you really want posts criticizing 3CX going to the department VP?). If you run on-prem, you cannot get support through their forums. You have to pay $75 an issue. If you use custom firmware (just putting a company logo image on a handset counts as 'custom') you cannot get support even if you pay for it until you go to basic firmware.
They are ending free versions of their software (no more testing on my home lab unless I pay them $1,000 a year for the cheapest hosting package) and I suspect they will be trying to eliminate on-prem installs in the next 2-3 years to boost their income. Our customer queue stats reporting software and multiple workarounds will not work on their hosted platform and if they go that route they will lose a lot of customers.
3CX for a small business makes sense. If you have a dentist office or car shop it will work. For Enterprise, look elsewhere. We are.
Bluster aside, they were shipping a product to paying customers despite having no knowledge of, and no control over, what it was that they were shipping.
To me, as an otherwise disinterested outside observer, but one who cares about quality and what it really means, that's unforgivable.
It seems that when they were alerted to a problem they didn't even investigate their own build systems - they just did a quick check at VirusTotal. Presumably so had the malicious actors. Earlier.
I might use VirusTotal for a slightly less than typical spam email, but if someone suggested that one of *my* products had been compromised, I'd be locking downloads, looking at the toolchains, and checking SHA256 digests - through the night if necessary - for *everything*.
A truly epic fail. In my view this company does not deserve its customers.
Malware and the AV engines that VT is aggregating across are so 1990s. Attackers don't send you malware. They sent you _links_ to malware, or better yet they Macgyver it from bits you already have on disk using duct tape and zip-ties. In this case a fairly pedestrian dll abuse to download malware as part of an update.
As such, checking your binaries against VT isn't going to flag anything, and a goodly amount of the time neither is your AV scanner. This was a multi-stage attack - the malware part that VT would be able to flag is downloaded much later in the attack.
If you're using VT solely to check files against AV patterns, you're doing it wrong.
You can dive deep into those "_links_ to malware" you mention. VT Enterprise especially can be a very powerful tool, if someone has half a clue what they're doing.
Sandboxing, graphing, threat hunting...these aren't static checks.
"We could only realize the extent of the breach after Crowdstrike gave us full details and then we immediately responded to the best of our abilities which by no means was Olympic medal standard," added Galea, who conceded that responding to a supply chain attack is, well, rather hard."
Was not even school sport day medal standard
Of course the real failure here was on the part of the writers of the malware who, having infected 3CX software, failed to register their malware with Virus Total. If they had followed that procedure then 3CX could have easily confirmed that their software was indeed malware-infested and...
... oh wait, hang on....
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
