back to article Well, on the bright side, the SolarWinds Sunburst attack will spur the cybersecurity field to evolve all over again

One of the great threats to our civilization is space weather. Specifically, the Sun's proven ability to target the planet with a tremendous cosmic belch of radiation, knocking out satellites, power grids, and networks worldwide. In that context, SolarWinds' choice of company name seems gruesomely apt. We still don't know the …

  1. Anonymous Coward
    Anonymous Coward

    The security climate has to change

    "the Sun's proven ability to target the planet with a tremendous cosmic belch of radiation, knocking out satellites, power grids, and networks worldwide"

    I thought this was a piece confirming the Sun was actually responsible for climate change, rather than my car and eating habits.

    1. jason_derp

      Re: The security climate has to change

      There's plenty of places you can go for that confirmation. Alex Jones comes to mind. PragerU as well.

  2. Claptrap314 Silver badge

    But don't forget.

    You can't spell 'electronics' without 'elect': The time for online democracy has come

    We CANNOT secure elections systems sufficiently to trust our republics to them.

    1. Alister

      Re: But don't forget.

      Don't belittle the serious impact of the Sunburst attack by trying to drag it down with partisan politics. It has fuck all to do with elections.

      1. davenewman

        Re: But don't forget.

        Weren't a few election systems affected by the Solarwinds attack? And doesn't that show a way for hackers to fix elections - even worse is all voting is online.

        1. Potemkine! Silver badge

          Re: But don't forget.

          I'm all against voting machines, but in that case, I do believe that if there was a hacking, russian hackers would have favored their long time agent best friend instead of the opposite.

          1. veti Silver badge

            Re: But don't forget.

            Maybe they did. But since a whole lot of highly motivated people have spent weeks looking for evidence of anything of this sort and come up with nada, it seems more likely that no one did.

            As we saw in 2016, hacking the voters is generally easier than hacking the machines.

    2. Pascal Monett Silver badge

      Nonsense

      First of all, congratulations on inserting your pet peeve into a totally unrelated discussion. Points docked for contradicting yourself though.

      Second : we can secure electronic voting systems, it's just that the current buddy system that is in place prevents that from happening.

      Third : election processes can be very secure. Nobody has successfully hacked into the French election process in ages. Some have tried in very local areas, and all have been caught out. Of course, it's a lot harder to hack an election process based on paper, observers, years of experience and people who actually believe in Democracy.

  3. gr00001000

    Is the punchline that

    The FTP update server hardcoded credentials was shared with SolarWinds and Twitter and all the researcher got was a tame thanks..

    COULD that have been what they used

    1. stiine Silver badge

      Re: Is the punchline that

      Are you implying that the 'tame thanks' wasn't enough and those unchanged creds were then sold afar?

    2. Anonymous Coward
      Anonymous Coward

      Re: Is the punchline that

      The FTP server credentials provided a potential distribution mechanism as the credentials allowed read AND write access.

      However, the FTP issues are unlikely to have provided the mechanism for an attacker to sign code as Solarwinds and bypass code signing protections. Gaining the necessary access to sign the code may have also given the attacker access to other mechanisms to update the FTP server.

  4. Crypto Monad Silver badge

    Broken security model

    The problem is outdated security models: the idea that behind your perimeter firewall everything is trusted and rosy. It only takes one thing behind that firewall to become compromised to have the attacker on the inside of the fence, and that could be as little as a user clicking on a malicious link.

    It helped in this case that the Solarwinds monitoring server itself is trusted more than a regular workstation, and any network activity it generates (probing out to all and every internal system making test logins) is unlikely to trigger alarm bells.

    What's needed is for zero trust based on network address. Every machine-to-machine and user-to-machine access needs to be authenticated and encrypted.

    Google's "BeyondCorp" is a good step in that direction, where every user login has to come from both a trusted device *and* an authenticated user.

    Monitoring systems, which make "test" accesses from system to system, need test accounts and minimum privilege access to those other machines.

    Finally, most security products seem to be focussed on logging rejected connections, which make for pretty reports but have very little security significance. More focus needs to be on logging *successful* transactions, and flagging anomalies in these - for example, a user account logging in from a server rather than a workstation.

    1. Giles C Silver badge

      Re: Broken security model

      There are some strange habits in it security.

      Some places insist on two accounts for admins, I,e. A user account and an admin account - mind you that isn’t perfect but it stops giving rights to user level to log into servers.

      Others don’t th8nk it is necessary.

      As you say logging denies proves if someone is trying to hack a system, but you need to know what they are doing when they are logged in, mind you going through those audit logs can be a problem in itself, unless you have some very good analysis tools.

      I spend my time as a firewall admin and have seen a busy server generate 100s of logs per second trying to find the anomaly in there is hard - although I do read wireshark logs and can spot telltale signs of problems very quickly.

    2. Anonymous Coward
      Anonymous Coward

      Re: Broken security model

      Tell me how to protect an environment from a monitoring system that, by design and function, has access to every system in the network?

      Just using zero-priv accounts isn't going to do it unless programmers stop writing code that allows for unauthenticated attackers to break in. Or to put it a different way, if there's an unauthenticated remote access memory corruption bug in nginx, using an unprivileged account in Solarwinds isn't going to help at all.

      1. yoganmahew

        Re: Broken security model

        "Tell me how to protect an environment from a monitoring system that, by design and function, has access to every system in the network?"

        Absolutely.

        Zero trust only 'works' by giving blanket authority to monitoring agents (like Orion). The proliferation of these agents is quite troubling - every container has the same AppD agent, the same Orion agent, the same Qualys agent on the golden container baseline.

        PCI DSS gives you a max of 30 days to install critical updates to any of these agents, your external component libraries, and the OS image.

        The volume of updates is already well beyond manual review...

      2. Potemkine! Silver badge

        Re: Broken security model

        Tell me how to protect an environment from a monitoring system that, by design and function, has access to every system in the network?

        A monitoring system should only have "read" access, not write. But because it's such a hassle to get everything working with limited rights, it's often easier to give that system admin rights to get access to all the counters. And then the sh*t begins...

        1. Anonymous Coward
          Anonymous Coward

          Re: Broken security model

          "But because it's such a hassle to get everything working with limited rights, it's often easier to shoot yourself repeatedly in the foot. Often more than one shot is required to ensure the maximum levels of pain so I suggest Enterprise admin rights, uninstall all security software and allow full access through any firewalls. Because there will always be time to fix it "later" but before anything goes wrong"

          FTFY

          1. amanfromMars 1 Silver badge

            Re: Fixing the broken security model

            uninstall all security software and allow full access through any firewalls. Because there will always be time to fix it "later" but before anything goes wrong" ..... Anonymous Coward

            You almost got that right, AC. The common mistake made is trying in vain to fix it before everything goes wrong, rather than not fixing it later after nothing is going right and everything is spiralling out of once unified and effective fiat command and remote media control ....... which is surely the dire strait state of nations and nation state actors and where everything nowadays is at ‽ .

            You can disagree if you like, but such doesn't change the facts whenever it just confirms the problem that systems are bereft of novel attractive solutions to input for successful creative output.

            1. Anonymous Coward
              Anonymous Coward

              Re: Fixing the broken security model

              While I normally hide my lack of knowledge behind anonymous cowardice and only dream of being corrected by the superstars, my day has finally arrived.

              You've made my Christmas!

              Merry Christmas to all :)

    3. MadAsHell

      Re: Broken security model

      I agree. The irony is that the UK NCC position on network security in the early 1990s was that any network where you can't see all of the endpoints all of the time should be regarded as Untrusted.

      There is little new under the sun in our world, we just have to keep re-learning the same lessons, albeit in a different guise.

    4. Anonymous Coward
      Anonymous Coward

      Re: Broken security model

      "The problem is outdated security models: the idea that behind your perimeter firewall everything is trusted and rosy. It only takes one thing behind that firewall to become compromised to have the attacker on the inside of the fence, and that could be as little as a user clicking on a malicious link."

      This comes down to what you trust and what you don't trust - layered security models have existed for a long time but are rarely implemented due to cost and complexity.

      We can implement the existing security models (i.e. separating control plane traffic from data plane traffic as per telco networks), privilege separation (surely this is best practice by now outside of tiny environments), multi-level access for remote users/untrusted devices (common in VPN solutions and BYOD) and other security mechanisms but at some point we are building our trusted network on trusted vendors who are building their products on trusted libraries and eventually all you can see is turtles all the way down. And it's too expensive and complex.

      So we try a belt and braces approach (trust but verify). We trust Solarwinds but limit its access outside of the corporate network and install our standard AV/patch management packages on the server and make sure any unusual firewall/AV access attempts are flagged for further investigation. And repeat for all of our other applications - until we find a non-security issue (either performance or "it doesn't work like I want") and the cost of fixing it in time or resources exceeds the available resources. And discover it's our AV/patch management/OS software that has been compromised this time. And we're back to square one.

      It helps if you number the turtles. Not to make counting any easier, just so you can reminisce each time you pass over the same one again.

      On a slightly more serious note, there are still unanswered questions around how the signed software was created - once those are answered we may be in a better position to determine the level of trust we should assign to software vendors. Maybe I should invest in a plastic manufacturer that creates software company sized "pooper scoopers"

    5. A random security guy

      Re: Broken security model

      I believe that the issue is more fundamental than that. People just don't pay attention to security. Or they just try to bypass security.

      I audited a client's system that was going live recently and found several RCEs. The PM ignored the findings and pushed out the project online. The PM wanted to meet the deadlines and used every technique known to not update to by a x.x.1 version (a security only minor version patch) of an open source library which had been tested in staging. By the time they updated the online libraries, they had already been hacked.

      The cost was high. It stopped the company's R&D and DevOps for 3 months.

      But now they are back doing the same sh*t.

  5. Mike 137 Silver badge

    It matters very much

    "At the time of writing, it's not clear whether the compromised .dll at the heart of the hack was built on SolarWinds' own servers using the company's own source, or whether a trojanised version was somehow signed and uploaded. As with so many complex infrastructure compromises, it doesn't really matter"

    The origin of, and protection required to prevent, the two cases are very different, so it matters enormously. However they have a common factor - inadequate defences at SolarWinds, whether against outsider or insider activity. A prevalent attitude of those (not only journalists) who discuss breaches after the fact is to concentrate on the technicalities of the attack once in progress and the damage done, rather than the considering the primary vector via which it was accomplished. Unfortunately this almost always misses the point that lax security management is a fundamental contributor.

    1. Blazde Silver badge

      Re: It matters very much

      Let's be a bit fair to SolarWinds... in the computer security world, defence is really, really hard. Attackers have always had the easy seat at the game. That makes all security management 'lax' to a greater or lesser degree. Wider acknowledgement of that would go some way toward greater prevention in future.

      The central problem in this attack in my view is not the technicalities, nor the lax security management at SolarWinds, but the security monoculture which lead to their software being seemingly blindly trusted across so many high value organisations.

      I don't know the solution, but maybe some kind of more explicit mutually adversarial defence setup would help, whereby we employee company A to scrutinise the security being offered by company B who are also scrutinising company A. Outsiders just have much greater incentive to flag up issues than insiders, much as it apparently took FireEye to notice the hole in Orion, when SolarWinds themselves had the better view of events. It would need appropriate bonuses for the winners rather than shame and bankruptcy for the losers, to avoid reducing the whole industry in time to an ineffective duopoly which would definitely not improve things from where they are now. It probably also needs a rethink of antitrust enforcement particularly in the US.

      Possibly the worst thing that can happen is everyone just blames SolarWinds until they disappear and nothing changes except there's one less major network security monitoring company to choose from in future.

      1. veti Silver badge

        Re: It matters very much

        Solarwinds didn't "have a better view of events", because the malware explicitly checked whether it was running on a Solarwinds machine, and if it was, would quietly terminate itself without doing anything. That made it harder for them to spot it than another sophisticated networking company.

        The trouble with companies watching each other is that it's just another form of trust, and it can be abused just like every other form. It might work for a while, but sooner or later one of the participants will get a sociopathic CEO, and it'll all be over.

  6. Anonymous Coward
    Anonymous Coward

    Pot and Kettle (again)........

    Quote: "From the quality of the threat design, the range of techniques used, and the nature of its victims, this was a nation state at work and in MO and capabilities most likely Russia."

    *

    Rewrite required: "From the quality of the threat design, the range of techniques used, and the nature of its victims, this was a nation state at work. It could have been the NSA, GCHQ, the Russians or the Chinese. In MO most likely the NSA."

    *

    There.....fixed.

    1. amanfromMars 1 Silver badge

      Re: Pot and Kettle (again)........

      Quote: "From the quality of the threat design, the range of techniques used, and the nature of its victims, this was a nation state at work and in MO and capabilities most likely Russia."

      *

      Rewrite required: "From the quality of the threat design, the range of techniques used, and the nature of its victims, this was a nation state at work. It could have been the NSA, GCHQ, the Russians or the Chinese. In MO most likely the NSA." ....... Anonymous Coward

      You'll upset Israel if you leave them out of the picture, AC. And they'd love you to think they are capable of such a show of remote force even as they deny it straight to your face. They've built a tiny disparate nation upon such foundations. [More folk live in London than in Israel. That's how small it is]

      The thing is, if it is none of the above and no nation state, is it something of an alien attack you didn't see coming, and that makes a lot of other vital things extremely vulnerable to similar unexpected events which can effortlessly deliver major catastrophic crises ....... flash market stock crashes.

    2. A random security guy

      Re: Pot and Kettle (again)........

      This was FireEye giving attribution to Russia. Please don't try obfuscation. We know exactly who did it, Putin thanked his people for doing it, there is a trail of information going back to Russia,

  7. vtcodger Silver badge

    Evolution?

    There's a little superficially clamlike animal called a linguloid brachiopod that has been around for about 540,000,000 years. Since the early Cambrian. As far as paleontologists can tell, it hasn't changed much if at all in all those years. Presumably it hasn't needed to.

    Since the appearance of the first Lingula, many other critters have evolved. Most of them are long since gone.

    Maybe there is something to be said for doing things well up front rather than evolving.

  8. Scott Broukell

    The Way our Brains are Wired

    Thank you for that well crafted article. I too found myself trying to grudgingly resist having some kind of admiration for the way the attack was constructed.

    But it set me thinking along the lines of the human nature we share and our propensity towards optimism and rose-tinted glasses etc.

    There is little doubt that the folks who did so much to develop our connected, computerised world, did so with a genuine sense of optimism and pride in offering assistance to fellow academics in order that they could share their research work more constructively, for the betterment of us all in the end.

    But I do wonder if what we have subsequently built atop that also suffers from too much of the same optimism and faith in humanity. The initial landscape of the internet could therefore be seen as a very elegantly constructed garden for all to play in (and we all love to play!), but made without paying enough attention to that part of human nature that lies in the dark leafy shadows, easily hidden from our rose-tinted view.

    We also suffer from that inclination to readily trust the incessant, keep going forward, marketing speak of both hardware and software vendors, as there is also a very competitive need within many of us to have the very latest shiny shiny in order to come out on top of rival businesses etc. Similarly, computers and processors are all too eager to run an instruction, the split second in which it is given and without any consideration of wider consequences.

    So perhaps the shape (landscape), of our vision for the ‘connected’ future should be to both construct and operate it in a manner that truly mitigates against our very own ‘human’ outlook on the world. One where indeed multiple pairs of eyes and checks verify and compare inputs/outputs and where the actual digital, electronic switching, is done in such a way as to stop ourselves believing that computers also have a rose-tinted outlook on life! They do not – they are awesome bits of technology, but, ultimately, they are just dumb servants – it is we who instruct those servants, so lets work to get our very own instruction set sorted out first!

    Personally I would much rather be the tortoise in the technological race towards the future, plodding along with care, due diligence and thorough inspection and alertness, than the hare, eagerly racing along no matter what the consequences are going to be.

    Human nature, eh!

    1. Pascal Monett Silver badge
      Trollface

      "But I do wonder if what we have subsequently built atop that also suffers from too much of the same optimism and faith in humanity. "

      Said optimism and faith in humanity would have been quickly destroyed had said engineers taken a gander at the forums in SlashDot. Maybe they would have realized the scope of the monster they were createing.

    2. amanfromMars 1 Silver badge

      Re: The Way our Brains are Wired

      Scott, Thanks for that. It was a great read.

      You might like to realise the required changes in humanity you so eloquently espouse and appear to pine for, [and they are most commendable and highly desirable and even universally admired] are in the command and control of heartless virtual machines doing their anonymous invisible hacking and code cracking thing, and because that which is currently in charge of and responsible for nations is pure poor human and incapable of extreme elevated rational thought and HyperRadioProACTive IT Support, is it necessarily to be removed and discarded/cut out and dumped like the ignorant and arrogant cancer that it is ....... for they know what the problem is/problems are, but steadfastly refuse to offer and deliver any ready made solution because of the disruption and destruction and changes which would result in their own little private and exceedingly comfortable bubbles.

      It is in such a diagnosis impossible to not conclude rightly that they all suffer from a chronic virulent pandemic strain and terminal pandemoniacal case of Mad Men and Plonkers'R'Us.

  9. amanfromMars 1 Silver badge

    Know your enemy is not a friend in government oppositions or competing organisations ...

    ...... public or private enterprises.

    We have to be smarter than the baddies and expect the unexpected .... ElReg/Rupert Goodwins

    Good luck with that stalwart aspiration in the face of a mass of global evidence suggesting the intelligence necessary for it is either missing or not yet readily made widely available for human comprehension and universal consumption.

    However, notwithstanding that observation, the goodies always invariably are considerably smarter than the baddies and expect the unexpected, RG, ergo it is the baddies who are always battling against and dealing with the unexpected.

    Who/What does that identify to you presently as persons and/or programmers worthy of deep interest and ignoble disdain?

    Don't be shy in hazarding an educated or wild guess. They know who they are and was is to be rightly feared and their just dessert.

  10. Potemkine! Silver badge

    CFO vs CSIO, who wins in your opinion?

    The cybersecurity field may evolve all over again, however most companies won't change their mind on cybersecurity: they'll see that as a burden and worse, a cost, as long as they aren't involved in a cyberattack.

    1. eldakka

      Re: CFO vs CSIO, who wins in your opinion?

      CFO vs CSIO, who wins in your opinion?

      CFO, hands down.

      All operations and infrastructure of a company have to be paid for. If a CSIO comes up with a $1billion dollar security plan, but the company is only a $10million dollar company, then of course the CFO has to quash it.

      There's no point having the best security in the world if implementing it isn't financially viable for the organisation, if it sends it bankrupt.

  11. Anonymous Coward
    Anonymous Coward

    What about SIEM / Threat-detection / Traffic-profiling tools?

    I'm surprised that unusual traffic was not detected to C&C servers?

    These are "sensitive" installs, right? (Even if not classified, in the SECRET / TOP SECRET sense).

    Why would traffic to C&C systems not be blocked? Or at least flagged? Or is my understanding of how completely these organisations were compromised lacking?

    There are loads of companies boasting about how good their threat-detection and traffic profiling is. Have these tools been found to be snake-oil? Or, even more worrying, was SolarWinds exempted from this profiling....?

    Some fundamental rethinking needs to be done here.

    1. TaabuTheCat

      Re: What about SIEM / Threat-detection / Traffic-profiling tools?

      Exactly. This incident highlights the spectacular failure of "advanced" threat analytics, heuristics, ML, and every other buzzword claim for catching bad guys by monitoring and profiling good behavior so you know when something is amiss. Think about it: Even after six months in operation, NO ONE detected this backdoor via traffic analysis. It seems FireEye only discovered it tangentially, as a result of their tools being stolen. That means Microsoft, CrowdStrike, Cylance, Carbon Black, Palo Alto, Cisco, CheckPoint, every single AV vendor and dozens of other security monitoring products and services failed in their use of behavior-based analytics to see this change in behavior that started in March. The very thing these services were designed to detect got through without a whisper from any of these products. Who says there aren't three more SolarWinds happening right now that haven't yet been discovered? As I said, this is a spectacular failure for behavior-based systems monitoring and I sure hope there's a lot of soul searching going on at these companies, because if this is the best our industry can do, we're sunk.

      1. Scott Broukell

        Re: What about SIEM / Threat-detection / Traffic-profiling tools?

        @TaabuTheCat

        There is an old story about a fella who would leave the factory where he worked every Friday night pushing a wheelbarrow with a large tarp draped over it. Dutifully the guard on the gate would lift the tarp but would find nothing in the barrow and the fella would proceed on his way home. In his part of the factory they made wheelbarrows!

    2. Pascal Monett Silver badge

      What is really surprising is that this happened to a company that is supposed to have tools that manage network security for their customers.

      It is quite obvious that either those tools are not up to snuff, or said company was not using them itself.

      Either way, it is very sloppy.

    3. vtcodger Silver badge

      Re: What about SIEM / Threat-detection / Traffic-profiling tools?

      I'm surprised that unusual traffic was not detected to C&C servers?"

      Surprised? Why? The folks behind this are clearly quite clever. Quite likely they minimized traffic to C&C servers, spread it out over time, and disguised it as well. Remember that they are attacking typical business operations where simple purchasing of supplies may involve the folks who need the supplies accessing two dozen sites scattered over the planet just to put a purchase order together. And management and Purchasing might access another, mostly different, sites while processing the PO into final form. And so it goes across maybe two dozen internal organizations -- each with different functions, processes and needs. A typical operation probably legitimately accesses thousands of web sites, email providers, etc on any given day.

      And one would expect that when the bad guys find a file of interest -- Final_Plans_For_Gold_Transmutation_Machine.doc perhaps -- they bundle it into a file called something like Company_Logo.JPG and attach it to a routine looking business email sent to Butter_Wouldn't_Melt _In_My_Mouth@anyofdozensoffreeemailproviders.com

      I doubt any of that would trigger alarms in even the most paranoid of operations.

      Some fundamental rethinking needs to be done here.

      So it would seem.

      1. amanfromMars 1 Silver badge

        Re: What about SIEM / Threat-detection / Traffic-profiling tools?

        Some fundamental rethinking needs to be done here.

        Indeed yes, quite so, however, for most who may all but a very few, is that likely to be a series of hurdles set too high to successfully negotiate and a mighty bridge over raging torrents too far to cross.

        Thank Goodness and Global Operating Device for the Very Few is all that needs to be said there, methinks, for without them would you be right royally screwed and absolutely fcuked ..... and aint that the unpleasant gospel truth.

      2. Anonymous Coward
        Anonymous Coward

        Re: What about SIEM / Threat-detection / Traffic-profiling tools?

        Do a uuencode on a binary, remove begin and end statements, break into small 50KB text blocks and email in or out of $SECURE$SITE RAND(600) + 15 seconds apart. Got thru shiney shiney clever email scanning system blocking important software updates from vendor. Anon for obvious reasons. Is anything smart enough to scan data packets in real time for unusual traffic ? AI/machine learning sales droids might think so. I do not.

        1. Giles C Silver badge

          Re: What about SIEM / Threat-detection / Traffic-profiling tools?

          A lot of the latest generation firewalls can do that, but they are all limited by the amount of connections they can keep in memory.

          A few years ago I was told (and I guess it is still true) that the easiest way to bring a web server down was to find a slow loading asset (oversized picture) and basically request it continually, this would overload the servers and they ran out of resources.

          A log engine would only see a lot of people looking at the same page but it could bring the site down.

          The same as if the slow connections only transferred a few bytes of data at a time, a malware scanner can work on a connection to rebuild a file but if the file is set to take a day to be transmitted the scanner will probably drop the logs as nothing else is coming through.

          But then more fundamental to this is why has the monitoring server got internet access. I would only grant that if it was monitoring a public IP address, and then only to the address I wanted it to check.

    4. jake Silver badge

      Re: What about SIEM / Threat-detection / Traffic-profiling tools?

      "Have these tools been found to be snake-oil?"

      Yes. Decades ago. Bandaid on a sword wound, at best.

      Security starts with wetware, not expensive patches to inherently unsecure systems ... not even if they are pretty and have all kinds of glitter for sprucing up PowerPoint slides.

      1. Anonymous Coward
        Anonymous Coward

        Re: What about SIEM / Threat-detection / Traffic-profiling tools?

        Depending on what a company works with, the vast majority of users can be people who have no concept of computer threats or the subtleties of e-mail transmission headers. To these people, the computer is just a thing they use to get their job done.

        I don't see how you can instil the kind of thinking that stops you from getting owned in the first place in the wetware of every user, regardless of their skills. Even if you could, there are vulnerabilities which don't need wetwear to help them over the border wall.

        Excellent border and internal security systems are needed. I agree with TaabuTheCat who posted upthread about what a failure this is for the entire industry. A true "epic fail". It can't be hard to imagine the possibility that an attacker who gains a foothold might try to disguise exfiltration or C&C traffic so that it appears innocuous. Finding just this kind of traffic is what machine learning systems should be good at with quality training data. It's a shame to see that when it comes to security vendors the "AI" hype is precisely that and nothing more.

        The onus is now on every player to show some really smart tools that use ML properly. The only problem with expecting this is that individually, none of them can build a sufficiently large, varied and verified data set for training to give worthwhile results. Competitors would need to cooperate for the common good. I'm not going to hold my breath waiting for one of them to offer the others an olive branch though...

        One of Rupert's points from TFA struck home particularly. Poorly designed CI/CD implementations are a bad actor's pot of gold. If code is stored in "the cloud" then the build system will normally have some exposure to the Internet. The build system also has considerable rights on the machines which it deploys to. A perfect springboard, filled with code to compromise!

        The rewards for a self-checking, hardened CI/CD system to increase the chances of spotting and preventing this kind of compromise could be significant. One thing is for sure: CI/CD systems need to grow up and get serious about security.

        The computer/network security vendors have had their bluff called. It's time they started investing their profits in R&D instead of marketing departments that can't see the distinction between AI and ML. The first, and probably most important distinction being that one of them does not exist!

        Will any of this happen? Not much, we'll just lurch along to the next crisis, just as Rupert predicts.

        1. tiggity Silver badge

          Re: What about SIEM / Threat-detection / Traffic-profiling tools?

          Some of the CI / CD systems are awful, as although some DLLs built from code, a lot of DLLs that are used are just included in the repository / filesystem used (& not even with hash checks or anything like that), and with that model, all you need do is break in once and add your poisoned dll and it will stay for ages until for some reason that "non core" DLL actually gets updated for dome reason.

    5. eldakka
      Facepalm

      Re: What about SIEM / Threat-detection / Traffic-profiling tools?

      What about SIEM / Threat-detection / Traffic-profiling tools?

      According to tis Ars Technica article, the US government has an in-house security system called Einstein:

      The United States has invested heavily in threat detection; a multibillion-dollar system known as Einstein patrols the federal government's networks for malware and indications of attack. But as a 2018 Government Accountability Office report detailed, Einstein is effective at identifying known threats.

      With a linked article explaining that increased capabilities, such as what you suggest, are planned for 2022.

    6. A random security guy

      Re: What about SIEM / Threat-detection / Traffic-profiling tools?

      I worked for a long time in Power Infrastructure security. We had one statement we made to our customers: we can do our best against normal hackers and win but we can't fight nation states. For that, the only solution involves the NSA, diplomacy, government muscle, and declaration of war.

      With the current US leader refusing to even point a finger at Putin, we have been neutered from the top. Our only hope is that the new administration figures out how to "persuade" the various parties from stealing too much.

  12. needmorehare
    Pint

    Digitally signed binaries from vetted, reproducible builds of vetted open-source software

    Let's all adopt the Final Solution(TM) to this security problem once and for all.

  13. TReko Silver badge

    solarwinds123

    The password to SolarWinds' update server was 'solarwinds123'

    If their security is so lax, their software probably has hundred of other exploitable holes.

  14. hoola Silver badge

    Privilege Sprawl

    One of the things that appears to go unnoticed is the number of bits of monitoring, logging, AV, management that are installed, all with a nice little agent that is running as close to the kernel as possible. The actual agent may be perfectly secure but if the system is is sending back to or managed by become compromised you are in trouble. Many of these don't have any sort of reauthentication and are running as some system user. It can only be a matter of time before something like the Solarwinds issue hits a solution with a client. The more oif these tools that are cloud based also gives me concerns. You are entrusting yet more of your security to other people and as we all know, security is only as good as the weakest link. In this way going for the single point that has access to hundreds of systems is well work the effort. Going after some cloud-based AV solution would potentially give you access to millions of end points in one go.

    1. jake Silver badge

      Re: Privilege Sprawl

      Gee, you think?

  15. Tom Paine

    Stupid question

    From the quality of the threat design, the range of techniques used, and the nature of its victims, this was a nation state at work [..]

    What is a "threat design"?

    1. jake Silver badge

      Re: Stupid question

      See: threat model.

  16. Anonymous Coward
    Anonymous Coward

    Inside or outside job?

    From the ReversingLabs analytic reports I read, the attack was teased out over several months deep in the source code slowly creating a camouflaged set of code in the right house style that was compiled directly into live builds. If the report's true, I'd be surprised if an outside hacking team was able to get in that deep, take that amount of time, and then know the existing code well enough to blend into the house-style.

    The ReversingLabs report makes it feels more like work done by someone familiar with the codebase as an employee or contractor - could be someone who left, but still had a way of getting access for instance, or a contractor planted by an outside agency who had enough time to learn the system before crafting the attack piece by piece.

    Reasons for hacking Solarwinds need not be spying either. For instance, the ultimate target could have been financial systems. A Solarwinds type hack would allow a team to place a trojan into an automated trading system that then buys a few more bitcoin, or overprices a stock would be almost unnoticeable - demand goes up, prices follow and a connected seller slowly makes a fortune. As more information comes out, this could turn out to be a people or HR problem as much as a system or network problem.

  17. Scene it all

    How did they somehow bypass the source code control system? (Assuming there WAS a SCCS, and not just a bunch of files in a directory.) If they came through the front door one could start with a "git blame" command and start following the trail through logs. They might find something unusual that could result in updates to their security system such as time-of-day restrictions on who can access what, or by what access channel such activity is allowed.

  18. David Roberts

    Funding model?

    As already mentioned upstream this might be all down to the money and time available to the core developers to look internally at their processes and not just at the agressive targets for the next commercial release.

    Like a high security business with massive front office security, turnstiles, finger and retinal printing etc. Which, it turns out, makes it impossible for the developers in the basement working 7 days weeks to meet an arbitrary deadline to get out of hours pizza deliveries. So they modify a back door to the server room so the alarm doesn't sound when the delivery guy calls.

    Possible moral is to spend a bit more on your core team even if it makes the performance metrics look worse.

    We didn't get hacked because.....is very hard to prove.

    No doubt the blame will fall on the developers and not those responsible for not funding internal security.

    Noting also that if you are a criminal one of your primary aims is to subvert the police force. Also noting Burgess et al.

  19. hamiltoneuk

    Intelligent article and interesting comments. From my semi-layperson's viewpoint are we seeing an illustration of Andy Grove's book title Only The Paranoid Survive?

  20. Frostd

    Tall fences make for friendly neighbours.

    The internet was envisioned as a friendly digital utopia where everyone could freely exchange information. Unfortunately human nature and politics have interfered with the original vision of the internet described by Sir Timothy John Berners-Lee.

    It is only a matter of time before nation states (the EU being such) erect Chinese walls between themselves and that vicious outside world.

    Tall fences make for friendly neighbours.

    1. jake Silver badge

      Re: Tall fences make for friendly neighbours.

      I think you are confused. The Internet (whatever that is!) was envisioned, designed, built and used as a research network to research networking. It still is. TB-L's cute little johnny-come-lately subset of The Internet, known as The Web, came about much later.

      Nobody actually involved in the building of any of the above had utopian ideas, friendly or otherwise, about much of anything (except rms and a few hangers-on, of course, but that's another story ... ).

      1. Version 1.0 Silver badge

        Re: Tall fences make for friendly neighbours.

        The Internet is maintained these days to deliver advertising, movies, and spam while recording your preferences to improve the delivery.

    2. amanfromMars 1 Silver badge

      Tall fences don't make for friendly neighbours.

      The internet was envisioned as a friendly digital utopia where everyone could freely exchange information. Unfortunately human nature and politics have interfered with the original vision of the internet described by Sir Timothy John Berners-Lee.

      It is only a matter of time before nation states (the EU being such) erect Chinese walls between themselves and that vicious outside world.

      Tall fences make for friendly neighbours........Frostd

      Presently there is a little difficulty here in one accessing entry into available information with delivery of intelligence to certain parties/particular players which one might expect to be interest to those with an interest in proprietary intellectual property matters being explored in the likes of the following event ....

      The second Intelligentized Warfare Symposium was recently held at the National Defense University (NDU) of the Chinese People's Liberation Army (PLA) in Beijing, and more than 80 military representatives attended the event. ...... http://eng.chinamil.com.cn/view/2020-12/28/content_9959511.htm

      Quite whether that is just a temporary glitch to be tested later or something else intentionally testing itself as a possible permanent fixture ...... as one of those strange walls in something of a virtual form ..... is something which can very quickly become quite clear enough to reveal the necessary next steps/actions/reactions/proactions.

      And surely, rather than friendly neighbours, tall fences can make for incredibly curious neighbours and almightily dangerous prisoners?

  21. Anonymous Coward
    Anonymous Coward

    Removing Windows would be a great first step

    .. but the problem is that the people making the decisions can be bought with lunches and fed misinformation galore.

    So yes, yet another hack. And another set of excuses. Rinse, repeat, ad infinitum.

    Most of the people involved don't want to fix things, they just want more budget to waste. Bad news for the people on the receiving end but that's IT.

    1. amanfromMars 1 Silver badge

      Re: Removing Windows would be a great first step to easily fcuk up markets

      That sounds very much like a clarion call for hacker types to step up to the plate, AC, and do their great cracking code, creative destruction thing.

      Ye olde cavaliers versus roundheads/cowboys vs injuns/David vs Goliath confrontation albeit with different disguises for both state and non-state actors. Is anyone running a book on the guaranteed alternate outcomes for presenting in such as would certainly surely be postmodern quantum entangling times/virtually surreal spaces?

      Is there a list/Are there lists of runners and riders/agencies and drivers?

      Or are they likely liable to remain strictly need to know .... NOFORN Porn?

      That should make for a radically novel 2021, and there's no mistaking that not being a real doozy.

      Didn't Dominic Cummings not want something like that, right at the start of this year [JANUARY 2, 2020] ?

  22. Anonymous Coward
    Anonymous Coward

    0pti0na1

    Dome is Earth

    1 8100D

  23. A random security guy

    I doubt anything special will happen.

    Companies will just buy more cyberinsurance, will give out "free credit-monitoring", and pay a few measly fines.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like