back to article Sysadmin infected bank with 'alien virus' that sucked CPUs dry

Welcome once again to Who Me?, The Register's weekly reader-contributed column in which IT pros share stories of times their work spun off into eccentric orbits and they (mostly) brought them back for soft landings. This week, meet a reader we'll Regomize as "Alf" who once worked on a team that tended Windows NT on thousands …

  1. chivo243 Silver badge
    Thumb Up

    NIce one!

    Alf looking for aliens! If this hadn't been set in a bank, and a previous colleague was still alive, this could have been a network I worked on... Whaddya mean there was work done over the weekend?

    1. Joe W Silver badge
      Pirate

      Re: NIce one!

      Ha! I know of a friend who did something like that in our university's computer lab. Learned quite a bit about obfuscating processes, ressource management, etc. We also learned not to mess with the sysadmin.

      1. Anonymous Coward
        Anonymous Coward

        Re: NIce one!

        We also learned not to mess with the sysadmin

        .. proven by the fact you're still around to talk about it. Never mess with a sysadmin.

        :)

        1. David Robinson 1

          Re: NIce one!

          I remember reading about the exploits of a sysadmin/operator back in the late 90s. He was working at a university at the time. I wonder what ever became of him?

          1. Scott 26
            Boffin

            Re: NIce one!

            Wouldn't be the University of Waikato in NZ, perhaps?

      2. Anonymous Coward
        Anonymous Coward

        Re: NIce one!

        This continued to produce blocks well after the people involved had left the university, essentially until all of the machines were rebuilt. You can even tell when a whole computer lab was rebuilt/upgraded.

        https://stats.distributed.net/participant/psummary.php?project_id=205&id=281848

    2. logicalextreme

      Re: NIce one!

      Perhaps not as noble but I do remember the odd shout from the CTO along the lines of "guys, please limit your torrents for the next few hours, we've got a client in and we need them to see the release go through" in the early days of my career when office bandwidth tended to outstrip home bandwidth.

    3. Korev Silver badge
      Coat

      Re: NIce one!

      > Alf looking for aliens!

      Did he think about checking behind the SETI?

  2. Pascal Monett Silver badge

    "perhaps best not to use a program called SETI@Home somewhere other than, you know, home"

    I disagree, somewhat.

    The mistake was in implementing the plan on his own. What he should have done would be to present the idea to his manager and get approval.

    Of course, that meant his idea could also have been shot down, but hey, them's the breaks.

    Disclaimer : I was a long-time contributor to SETI@Home myself, and ran it on every computer I had at home, plus my work laptop if I could.

    1. lglethal Silver badge
      Stop

      Re: "perhaps best not to use a program called SETI@Home somewhere other than, you know, home"

      I've always been of the belief that anyone who follows the creed "It's better to ask for forgiveness than to ask for permission", is not someone I want to be working with.

      Something that might seem like an easy, no harm install, could easily cause massive problems that the installer knows nothing about - from a security point of view, from compliance, from legal or regulatory points of view. How often have we all had meetings where someone says, "Well what if we just install this to solve the problem..." followed by someone else shouting "Noooo...." and then explaining how that would cause much MUCH bigger problems.

      Asking permission, might see your idea shot down, but asking for forgiveness, might see your career shot down...

      1. Joe W Silver badge

        Re: "perhaps best not to use a program called SETI@Home somewhere other than, you know, home"

        Ha, I almost got sniped there.

        Yes, all those points. I sometimes toy with the "better ask for forgiveness" idea, but I know where (and to whom) I can do this.

    2. Anonymous Coward
      Anonymous Coward

      Re: "perhaps best not to use a program called SETI@Home somewhere other than, you know, home"

      it was a bank, they would have charged SETI@Home with very huge fees...

    3. Sandtitz Silver badge
      Meh

      Re: "perhaps best not to use a program called SETI@Home somewhere other than, you know, home"

      "Disclaimer : I was a long-time contributor to SETI@Home myself, and ran it on every computer I had at home, plus my work laptop if I could."

      I also ran it with my puny AMD K6-2 back then - without the graphical screensaver because the graphics slowed down computation measurably.

      Anyway, for some time my job consisted of putting together PC's and I used my SETI@Home account for some CPU stress testing back in the day.

      The it all went BOINC and I lost all my interest.

  3. Mast1
    Unhappy

    SETI go home

    In a former job, one of my colleagues wanted to play that game of getting on the leaderboard so set up SETI@home on all of the office computers.

    I left a hardware simulator running overnight. It was only supposed to take about 5 hours, so I could pick up the results and start work first thing the next day.

    It eventually took about 17 hours, and so messed up a day's work plan. Not such an invisible cost.

    I requested they be removed from every machine except at most, his.

    Obviously this was in the days of a more liberal security & management policy.

    See title.

    1. hittitezombie

      Re: SETI go home

      Your config must have been weird, Seti@Home only kicked in when the CPUs were idle, so if your simulator was running then the Seti would have been throttled down to zero.

      1. An_Old_Dog Silver badge

        "Idle" CPUs

        Rather than "detecting" when the CPU is idle, perhaps SETI, BOINC, et. al., should have themselves installed as the OS's standard idle task. Let the OS decide when the CPU is idle. Some OSes have an official idle task, but I don't know if MS NT, etc. do.

        1. Ken Hagan Gold badge

          Re: "Idle" CPUs

          I think the whole concept of an idle task was swept away when processors introduced frequency stepping and low-power states.

      2. Mast1

        Re: SETI go home

        Hazy recollections are that it was supposed to be the screen saver. Presumably, it was activating on lack of keystrokes/mouse interaction, which was the state when running a simulator (Win95/Win2k era).

        My main point was that feeding someone's "home office" vanity was costing me and my firm time and money. (and creating extra work/distraction to get it removed).

    2. Anonymous Coward
      Anonymous Coward

      Re: SETI go home

      They even had a version that ran on AIX.

      No internet? no problem, you could set your machine to 'serve' work out and collect the results, then when you got home upload/download from SETI

      1. AustinTX

        Re: SETI go home

        I used PHP Seti@Home Monitor, which worked like a man-in-the-middle to download batches of work packets and serve them to your cluster of local machines. It let you manage packets before they were worked on, I think it even let you chuck packets that would take longer than others to process, so you could get just that little bit ahead of competitors. Also, you could display a star chart that showed the direction each packet was recorded from.

        1. AustinTX

          Re: SETI go home

          Er, I meant SetiQueue!

  4. Red Sceptic

    No good deed goes unpunished.

    1. doublelayer Silver badge

      The people doing the punishing weren't aware of any good deed, and there wasn't any benefit to them or to the business. They had every reason to question whether their production systems were going to be hit by something. I'll also note that they didn't punish anybody. No good deed and no punishment makes your statement a bit difficult to understand.

  5. thosrtanner

    I suspect the regomiser had a hand in picking ALF as a name. Somehow.

    1. Ordinary Donkey

      Funny, I always suspected the regomiser already had hands.

    2. Pirate Dave Silver badge

      The question is - how does he like his cats?

      1. Trollslayer
  6. Anonymous Coward
    Anonymous Coward

    We've all done that......

    ... if we are a certain age.

    I did exactly the same, we had a ton of rack servers that outside of biz hours did bugger all, but otherwise were SQL clusters, or webservers behind F5

    These were obviously all set up and worked swimmingly until the clocks changed, and something went t*tsup and comms went down

    Cue immediate need to investigate the issue.

    Now I had a gut feeling what it was, it was, and removed trace of it.

    Excuse given?

    There was a known issue between Broadcom network adapters and HP Procurve switches when auto-negotiate was relied on, where comms would fail and infra refused to set the TX speed manually despite this.

    I recommended they be set properly and the issue went away... Infra successfully blamed, my bullet dodged.

    Still awaiting comms from E.T.

  7. Wanting more

    sitting idle means they are using less power

    At least modern PCs / Servers when the CPU is idle they are using less power. Give them a heavy load then the power consumption goes up. So it's not actually "free" to give them something like seti@home to do. Also generates more heat therefore the cooling system has to work harder also drawing more power.

    So in effect it's stealing to use the "idle" computers in this way.

    We've had incidents where I work of people mining bitcoins using the organisation's electricity.

    1. Killfalcon Silver badge

      Re: sitting idle means they are using less power

      The trick is to do it in winter.

      Since heating electrically is 100% efficient (or near enough, I guess some might get wasted as noise or blinkenlight), it would be 'free'- in the sense that you spend as much electricity using a 1000W server rack as a heater as you would from a 1000W heater, and get the same thermal output.

      I've heard of offices, back in the glory days of on-prem servers, heating swimming pools with 'waste' heat from the CPUs. It's only a waste if you don't find a use for it.

      1. Anonymous Coward Silver badge
        Facepalm

        Re: sitting idle means they are using less power

        But by that logic the heat exchangers used in HVAC systems are ~250% efficient. Therefore using computing to generate heat is less efficient and still costs the company money (albeit less than in summer).

        1. Killfalcon Silver badge

          Re: sitting idle means they are using less power

          Good point, I'd forgotten about heatpumps somehow! They are indeed even more 'efficient' than using heating elements - able to move more heat for a given wattage than you could produce. That does rather put a dent in the scheme.

          (for anyone worried about breaking the laws of thermodynamics - the trick is that you have to consider the whole system. Ultimately, the heat you get out of HVAC is produced by the sun outside your building, and then you move it indoors by trickery, rather than trying to make it indoors directly by heating a wire)

      2. KittenHuffer Silver badge
        Linux

        Re: sitting idle means they are using less power

        I agree with finding a use for waste heat that you're going to generate anyway (swimming pool, etc.) but the problem with saying that during Winter a 1000W server rack costs the same as a 1000W heater is when you wouldn't use a 1000W electrical heater but the equivalent output from a gas central heating system.

        But these days it would probably be cheaper to buy nice warm jumpers and not turn the heating up.

        ----------> Starting to miss the cold more and more!

        EDIT: Beaten to it by the original and one and only AC!!!

        So when the AC above goes AC do we get a recursion error?

        1. Killfalcon Silver badge

          Re: sitting idle means they are using less power

          I think right now a 1000W electric heater would be cheaper than the equivalent gas burner! Something about there being a war on.

          Normally gas heating is indeed cheaper than electric, and as as AC pointed out, heat-pump based stuff like HVAC is cheaper than either.

          1. Elongated Muskrat Silver badge

            Re: sitting idle means they are using less power

            You might think that, but for political reasons*, the unit price of electricity is tied to the most expensive generating source, so when the price of natural gas sky-rockets, so does the price of electricity. I believe the price per KWh for 'leccy is still higher than gas (apparently currently 34p vs 10.3p), although the gas price might be higher than what the 'leccy used to be before extreme capitalism took hold.

            *As far as I can tell, those political reasons appear to involve brown envelopes and record profits for producers.

            1. TheMeerkat

              Re: sitting idle means they are using less power

              Most of electricity is produced mostly by burning gas.

              1. Anonymous South African Coward Bronze badge
                Thumb Up

                Re: sitting idle means they are using less power

                Most of electricity is produced mostly by burning gas.

                Save gas, fart in a jar.

        2. phuzz Silver badge
          Flame

          Re: sitting idle means they are using less power

          The downside to a central heating system is that you're generally heating a whole house. Sometimes (eg. if you're working from home) it's more efficient to just heat one room.

          This winter I've found a better option than a fan heater, an electric foot warmer. It draws 25W max, but keeps me feeling nice and warm, which would take a lot more gas/leccy than if I was trying to heat the whole room.

          1. Chris Evans

            Re: sitting idle means they are using less power

            Yes heat the person not the room is by far the most efficient. The problem is my fingers get cold easily, fingerless gloves help a bit.

            1. J.G.Harston Silver badge
              Coat

              Re: sitting idle means they are using less power

              All these people who've followed the TV make-over fad for removing all their internal walls are going to be cursing when they realise they have to heat the entire house in order to heat any of the house. When working from home I ensconce myself in my bedroom office with curtain over the door, and after the morning heating turns off the waste heat from the computer keeps things ticking over.

              Coat, as that's what I put on to venture to the kitchen for the kettle. :)

        3. Stuart Dole
          Stop

          Re: sitting idle means they are using less power

          For a while I loved SETI and ran it on my (then powerful) home PC. All the time. But then I noticed that the fans kept running. Hmmm. I put a watt meter on the PC, and sure enough, it used a lot more power running SETI in the background. Looking at my power bill - this is at home, not work - I figured it was about $200 a year.

          So I was donating $200 a year out of my pocket to SETI. So I killed it. Quiet fans...

      3. david_hay

        Re: sitting idle means they are using less power

        I’m sure I saw a story here, or somewhere else, within the last few days about someone selling servers that sit on ( perhaps under ? ) water tanks to heat the water with their “waste” heat - perhaps crypto mining isn’t dead after all :-)

        1. Killfalcon Silver badge

          Re: sitting idle means they are using less power

          I saw the same thing - I think they tested with Helium miners, but the plan for roll-out is distributed compute, basically renting folding/SETI@home type stuff to companies/orgs that need a lot of processing done.

      4. doublelayer Silver badge

        Re: sitting idle means they are using less power

        Doing it in winter is only beneficial if all of the following is true:

        1. The place where the computer is generating heat would be heated to that level anyway (possibly not at night).

        2. The electricity powering the computer is cheaper or the same cost as the way the room would be heated.

        3. The same is true for any other systems involved, for example the networking systems that would also heat up if the task involves a lot of data transfer.

        4. There are no other limited resources that are consumed (for instance a data cap on the network).

        In most cases, cryptocurrency mining for heat production is not the most efficient method to do it unless the crypto is worth a lot to you.

    2. Stuart Castle Silver badge

      Re: sitting idle means they are using less power

      Where I worked, I had a Seti@home client set up for out of hours work. I got rid of it because our IT department noticed, and banned it, stating that if we were going to support any projects like that, the company would do so officially, and set aside equipment for the task. They were mainly concerned about us running unauthorised code, and also they tended to run maintenance tasks overnight, so wanted a light load on the network, so as not to slow them down.

    3. Mark 85

      Re: sitting idle means they are using less power

      At least modern PCs / Servers when the CPU is idle they are using less power. Give them a heavy load then the power consumption goes up. So it's not actually "free" to give them something like seti@home to do. Also generates more heat therefore the cooling system has to work harder also drawing more power.

      I'm totally surprised that the bean counters didn't notice the electric bills suddenly got higher and start an inquiry. Those folks seem to or should notice any increases in costs of just about everthing.

      1. david 12 Silver badge

        Re: sitting idle means they are using less power

        I'm totally surprised that the bean counters didn't notice the electric bills

        Those old PC's didn't use more than a lightbulb when running. More at startup, or with heavy disk use. And the monitor used quite a bit.

      2. John Brown (no body) Silver badge

        Re: sitting idle means they are using less power

        Sounds like it was a while ago, when power usage was less related to CPU usage and an "idle" CPU still drew more or less the same amount as one under full load.

  8. F. Frederick Skitty Silver badge

    We had some idiot go round installing SETI@Home on every Windows machine they could get access too. Turned out to be a contractor who used the simple expedient of asking a user what their password was, then logging in once they left for the day. My manager was an ex-army officer, and the bollocking he gave the contractor before dismissing him could be heard from the other side of the building.

  9. Anonymous Coward
    Anonymous Coward

    I admit to SETI, and when that finished I would run other BOINC stuff, but working at a hospital set it for running tasks such as finding cures for cancer, TB and COVID.

    I do have scruples - drew the line at bitcoin!

    1. TimMaher Silver badge
      Boffin

      World Community Grid

      They are doing cancer, Covid and other stuff.

      You could also do climateprediction.net for global warming.

    2. Anonymous Coward
      Anonymous Coward

      PS - Only did ran it on my PC, no others. The PC I had was an old HP Workstation with TWO CPU's (not cores, two phyiscal chips), so that beast was just twiddling it's thumbs)

      Now you mentioned the World Community Grid, does sound familiar, think I did that instead of BOINC. Whatever it was, suddenly stopped working, stongly suspected that was due to new security measures, so took the hint and removed the client and never spoke of it again!

  10. Kevin Johnston

    Missing snippet?

    Yet somehow El Reg managed to not mention that they themselves had teams in all these Grid Computing schemes... I was in the one working on various medical projects and as a team we were doing quite well

    1. Korev Silver badge
      Thumb Up

      Re: Missing snippet?

      I was thinking the same.

      El Reg of old is not the same as what we have today though :(

  11. Pirate Dave Silver badge
    Pirate

    Yeah

    I did similar at the university where I used to work, although I used the screensaver client, not the always-running client, so it generally didn't cause any problems like ALF ran into. I put it in the Ghost image that we used on the Dell GX1's in our computer labs. It took those 233 MHz PII's a few hours to grind through each packet (or whatever it was called), but they were far faster than the 486's and Pentiums I was running it on at home, and there were 50 of them. I did manage to crack the Top 500 for EDU (maybe even the Top 100) before all was said and done.

    AFAIK, we never did find E.T, though.

    1. Anonymous Coward
      Anonymous Coward

      Re: Yeah

      I would occasionally run it on a PC / laptop, just overnight and a few units, just for soak testing etc....

      Used to use an old fractal generating application for the same reason - cool to watch, but where it would max old older CPU's, modern ones not so much

      1. Alistair
        Windows

        Re: Yeah

        Hey!

        That was *my* line!!!!

        (still have some awesome screen bg's from the fractal program in my catalog)

    2. Marty McFly Silver badge
      Pint

      Re: Yeah

      Ha! I also had racks of out-of-warranty Dell 233Mhz desktops running SETI in my test lab. Dumb switches, Win2k or Win98. Hand-crimped cabling. 10/100 NICs. Good times!

      1. Pirate Dave Silver badge
        Pirate

        Re: Yeah

        Yeah. I didn't get in trouble at work, but the wife was PISSED one month when the power bill came in and was $75 more than usual. I had gotten up to about 20 or 30 various 486/pentium motherboards and power supplies strung up in the crawl space under the kitchen, all dangling from wires tied to nails and screws I'd put in the joists. Pfft, we don't need no fancy computer cases to look for aliens. All running Linux and the SETI client 24/7. She was not amused in the slightest and I spent a good week in the doghouse for that stunt. So I cut down to my 5 fastest mobos and took the rest to work and found a spot to run most of them there.

        Things were simpler then. Good times indeed.

  12. Anonymous Coward
    Anonymous Coward

    The other developer at the startup I working at did just this, then set the instances to run 24/7 before going on holiday for a couple of weeks grinding the network to a halt. That and the bit torrent server he had set up

  13. Anonymous Coward
    Anonymous Coward

    If he'd mined bitcoin instead of SETI..

    .. he'd be a multi-millionaire by now

  14. This post has been deleted by its author

  15. Nightkiller

    The only thing that saved this guy's ass was that no one in the IT Department knew how to use Task Manager or the lot of them colluded to conceal the operation in solidarity with SETI@Home.

    1. Robert Carnegie Silver badge

      In settings such as "The Invaders", those people probably are aliens themselves. A lot of us are, you know... v'ir fnvq gbb zhpu. :-)

  16. hittitezombie

    Sometimes the boss also cooperates

    Early 2000s I was working on a satellite office of a corporation. We had plenty of lab servers and PCs only used approx 10 mins a day each, each having a particular version of our software being tested.

    So, Me and my boss ran Seti@Home on all servers and workstations overnight. No one cared about heating/electricity costs. We ranked up rapidly in the UK lists at least, definitely into the first 100, but we didn't expend it to the whole company... I even had an SSI cluster of Seti@Home at some point, the PCs would reboot to the CD overnight and at 8AM they would reboot back to WinNT / Win2k / Linux.

  17. TFL

    It was to save the hardware, honest!

    A gaggle of us at one workplace got into the whole SETI thing for a bit,easy enough since we had full access on our workstations.

    There was a Windows admin guy who started to get persnickety about it, not there was a policy against it. He was well-meaning enough, but not too bright.

    One of our more creative engineer-types came up with the idea that running the SETI client was actually beneficial for the hardware longevity, because it reduced the physical expansion and contraction in the CPU due to thermal cycling. Brilliant!

  18. Anonymous Coward
    Anonymous Coward

    that could have been me....

    I'm not ALF but regularly had dozens of Sun servers in my DC totaling a couple hundred cores running SETI over the weekend.

  19. druck Silver badge

    Before SETI was RC5

    Before SETI there was the RC5 cracking challenge by distributed.net which I had been taking part in for a while, when I decided like Alf, to boost my rankings I'd co-op a dozen of the Pentium 90 and 133 industrial PC systems which were sitting idle at the back of the lab. There was no direct internet access on the network in those days, with a single PC being available for anyone which needed to access the internet to transfer files from, so I knocked up a script to copy results and new block files between the shared machine and the RC5 crackers. It worked so well I forgot all about it and a year later left the company, it wasn't until 6 months after I had gone someone must have noticed them and switched it all off.

  20. miguel4k

    So i wasn't the only one :-)

    Working for a large HW vendor, and had a lab with all the brand new servers we showed the customers.

    Among them an Itanium powered quad cpu server. In total I had approx 56 CPU's on Seti at all times, was in top-25 at some point.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like