back to article Danger zone! Brit research supercomputer ARCHER's login nodes exploited in cyber-attack, admins reset passwords and SSH keys

One of Britain's most powerful academic supercomputers has fallen victim to a "security exploitation" of its login nodes, forcing the rewriting of all user passwords and SSH keys. The intrusion, which is understood to be under investigation by GCHQ offshoot the National Cyber Security Centre (NCSC), rendered the ARCHER high- …

  1. Chris G Silver badge

    I can imagine there may be more than just hostile states who would be interested in either the disruption or the contents of Archer and other supercomputers doing research into the pandemic.

    Considering everyone is talking about world population vaccination and the search for a practical test that is also economical, there are millions if not billions of dollars to be made, whoever gets the patents out first will be s happy bunny.

    1. Yet Another Anonymous coward Silver badge

      Yes that's how we know it's a state sponsored attack.

      If N Korea or Iran get the data and make a vaccine first, the USA will have to pay them for the patent rights.

      ps. I also have a bridge for sale

    2. macjules Silver badge
      Facepalm

      Naughty GSK and their outsourced hacking research teams ...

    3. John Brown (no body) Silver badge

      "whoever gets the patents out first will be s happy bunny."

      You'd think the situation might result in whoever finds a useful vaccine might gift it to the world, but I suspect it will be "protected" in some way that someone will get very rich from it.

    4. osakajin Bronze badge

      World population vaccination.

      That is scary given the timescales for normal vaccine production and testing. Do you want something thats been rushed out?

      What about if you cant function in society if you cant prove vaccination? Why is the WHO funded by vaccine corps?

      Fishy?

  2. John H Woods Silver badge

    if you can get into that ...

    ... you can probably mine some cryptocurrency!

  3. Alan Birtles

    "change passwords and SSH keys on any other systems which you share your ARCHER credentials with"

    Umm, isn't the point of SSH keys that the server only has your public key so a compromise of your public key on one server wouldn't need to change your private key?

    1. Korev Silver badge

      A lot of HPC places need to you have both keys so you can SSH between systems in the centre

      1. s2bu

        Dear god no, that's not how it's supposed to work AT ALL.

        1. Korev Silver badge

          With a shared file system how else would you recommend to do it?

          1. Peter Gathercole Silver badge

            @Korev

            How about agent forwarding?

            1. brotherelf

              Re: @Korev

              This – though, since both the GoDaddy hack and this one have rumors about infected sshd binaries, I'm not entirely convinced it's not the forwarding aspect of the protocol that is part of the issue. (I.e., if you connect into an infected host, it uses forwarding to log "you" into other nodes. That still doesn't explain the privilege escalation to modify sshd in the first place, though.)

          2. stiine Silver badge

            how i'd do it

            separate keys.

            1. HPCJohn

              Re: how i'd do it

              Quite often when you first log into an HPC system a script is triggered which generates a passwordless key pair, for use within that system. So yes, you have separate keys.

          3. HPCJohn

            I would rather glibly say munge https://github.com/dun/munge

            I guess though if an attacker gets root then all bets are off.

            1. Alexandre Strube

              The attack was used with normal user keys, where the attacker ran an exploit and got root.

              This has happened on Cray Linux and on CentOS and RedHat and SLE systems throughout Europe.

              Plus, things like parallel debuggers need at least a private/public key pair IN the system to be able to ssh from the login nodes into the compute nodes, even though the batch stuff is managed with munge.

          4. This post has been deleted by its author

    2. Anonymous Coward
      Anonymous Coward

      But, it seems in this case it is users private local keys that have been compromised. Many people use the same private key for multiple systems.

    3. Jamie Jones Silver badge

      One ting not mentioned is that the serverkey ("authroised hosts") would now be untrusted, and should be removed from users authorised_hosts file

  4. Scott Broukell
    Coat

    Hmmm . . . was this the result of a Cray Phishing attack!

    <gets chest waders and coat>

    1. Korev Silver badge
      Pint

      Well done, that was Cray-zy good -->

    2. This post has been deleted by its author

    3. Anonymous Coward
      Anonymous Coward

      These are Cray-zy, Cray-zy, Cray-zy, Cray-zy nights!

  5. HildyJ Silver badge
    Angel

    I blame

    Dr. Krieger.

    But I don't know if you Brits get the show.

    1. Korev Silver badge

      Re: I blame

      Phrasing!

    2. SanctifiedByDynamite

      Re: I blame

      Lana, be careful! Jesus, the helium!

      1. J.G.Harston Silver badge

        Re: I blame

        Ah, yes, Fisherman's Wife and Fisherman's Wife II - The Retentacling.

      2. The Oncoming Scorn Silver badge
        Angel

        Re: I blame

        LANA!!!!!!

  6. amanfromMars 1 Silver badge

    "It must be" doesn't cut it when imagined a very unusual attack therefore Crays are being probed ‽

    Professor Alan Woodward of the University of Surrey told The Register: "To see a Cray being attacked is very unusual so I imagine it must be the computing infrastructure around it that has been attacked.

    Not necessarily so, Professor, whenever core processes/source algorithms are effectively suspected attacked, cracked and realised stealthily pwnd/copied/mirrored/exposed/exported by others, and for others too, practically unknown and virtually invisible.

    Quite whether that sort of AI Attack and Virtual Assault is Almightily Invincible is in Proofs Submitted something worth surely more than just a fleeting ponder and mindless wonder.

  7. YetAnotherJoeBlow

    Vaccine

    The most important process in China at the moment is to be first with the vacine at all costs. Failing this, I would not want to be anyone connected with that effort. Their families, their homes, their livlihood, and their liberty all depend on being first.

    1. teknopaul Silver badge

      Re: Vaccine

      Citation Needed.

      Its the US that needs a vaccine, because peeps there see a mask as an infringement on their freedom.

      Countries with piss poor covid responses need a vaccine.

      China, and various other countries, are doing quite well with masks and travel restictions.

  8. Aidan Whitehall 1

    Due to be retired soon?

    It'll likely end up in some academic's study, like the boxes from our workplace when they're slung out.

    1. Anonymous Coward
      Anonymous Coward

      Re: Due to be retired soon?

      That's a VERY big study

  9. Alexandre Strube

    Not only ARCHER, not only Cray Linux

    Most European supercomputers are offline now because of this attack. All big German ones and the ones involved in PRACE are down.

  10. amanfromMars 1 Silver badge

    For when you decide to realise the Future is Never the Same as the Past and Always a Beta AI Present

    Is there a healthy appetite and voracious market for non state actors leading acting state heads along perilous politically poisonous paths for/to catastrophic dead ends?

    Such is surely Murderously Suicidal Mediation to Present and Realise with Main Stream Media Machinery ....... but is does appear to be currently popularly rife most everywhere struggling to rule and reign over/mentor and monitor masses/populations without Commanding Control Information Relaying Advanced Intelligence Sources through and for Absolutely Remarkable Virtual Machine Enrichment Reprocessing.

    Why would they consciously do that, ..... and there's no escape in thinking to mention and question whether such may be an unconscious reaction rather than not inconsequential deliberate misaction ........ a Major Fault verging on a Fatal Error? Is Madness in such a crazy case the reason lauded in explanation of the most unpleasant of circumstances being avoided.

  11. Anonymous Coward
    Anonymous Coward

    SSH keys & the coming apocalypse

    [Anonymous because I can hear the black helicopters hovering overhead.]

    I used to work somewhere with a machine closely related to ARCHER: I think I may have had an account on ARCHER but I'm not sure. One of the reasons I eventually left was the repeated amusing cycle of:

    ac: 'you have a security problem around x: here are some approaches you could use to fix this, I'm very willing to help you do that because I am quite good at thinking about this stuff'. them: [very many polite words explaining why they couldn't be bothered fixing it as it would inconvenience the science people and would also just generally mean they would have to think about security and who cares about that]. ac: 'you didn't read my suggestion: if you did what I suggest the science people would not even notice. Also you are doing stuff here which is critical to the security of the country, you really do want to think about security'. them: [repeat previous statement with more and different words]. ac: [gets depressed, gives up, leaves organisation, rants on The Register, dies in ditch].

    So, the basic deal with SSH keys here is this: you need to be able to run jobs on the HPC. Those jobs involve lots of stages: extract code for model from repo, do configurationy stuff, compile code for machine, do lots more configurationy stuff, run model for n cycles, postprocess, archive output, run model for another n cycles, postprocess, archive iterate that a few hundred times, clean up, shut down. During that time things will die: the model will hit some bad thing and will fall over, so you'll need to be able to go in, perturb things a bit from the previous cycle and restart the cycle that died and so on. All of the substantive steps run by submitting jobs into whatever Cray's batch scheduler is, which I forget, but it's some open source thing, so there's also a lot of stuff like 'wait for job to find enough nodes to run on'.

    Many of these jobs run for months of wallclock time: possibly years in some cases: I ran jobs which ran for hundreds of days.

    So all of this is run by, essentially, some agent on your behalf and you talk to the agent by ssh from your login machine, and it then talks to other bits of the system, also by ssh. The agent itself will often die during this the course of a job, so it carefully keeps all its state in files. In general you really want to be able to recover things when things die, so everything dumps state in files every fairly short while. The whole system needs to be able to recover from a complete cold start of everything: you really do not want to lose half-a-year's run because someone did the equivalent of tripping over the power lead. There's a good chance the OS will be upgraded on some of all of the system during a run.

    So things like SSH agent forwarding are not even close to being solutions for this, because no single part of the system, other than the filesystem, will survive for the duraton of a job.

    So there are some possible solutions to this, which would at least include manufacturing per-job pairs of SSH keys which never got reused. But that is, of course, not what they do. What they do is say that your SSH keys need not to have passphrases, so the system can get access to your private key and things can work. Because that will be fine.

    Well, of course, it's not fine. Will they learn from this? No, of course they won't. Are there other security problems, both logical and physical: hell yes there are. Will something really bad happen in due course as a result: yes, of course it will.

    I have a paper I wrote on all this before leaving which I'm tempted to make public but no doubt I would be in violation of everything if I did.

    1. HPCJohn

      Re: SSH keys & the coming apocalypse

      Oh DO come on... You want security?

      PBS used to use rhosts trusts in the old days.

    2. Anonymous Coward
      Anonymous Coward

      Re: SSH keys & the coming apocalypse

      >> What they do is say that your SSH keys need not to have passphrases, so the system can get access to your private key and things can work.

      This categorically not true. I've used ARCHER, my SSH keys have passwords, they don't need access to the private key at all, it never left my personal system.

      1. Anonymous Coward
        Anonymous Coward

        Re: Dead on arrival

        She's talking about the system-generated-at-account-creation passphrase-less keys that live in your account on the system and allow the queue daemons to communicate on your behalf between the nodes that make up the architecture.

        I would guess.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020