How to keep your enterprise up to date by deploying the very latest malware Start your week with a trip back to the early days of the World Wide Web with a tale of imaging peril and malware malarky from the files of Who, Me? Today's story comes from a reader whom we shall call "George", for that is not his name, and an experience at what he delicately called "a very large retail store" during the …
No, we don't know why some IT departments insist on crippling computers during working hours either
Maybe because some users keep shutting down their computer when they end their day, whatever the numerous times you said to them not to do it to enable the scans during the nights?
At the end, either you disturb a lot of users or you have computers never scanned blinking in red on your monitoring app. Some choose the former, some the latter.
Wake on lawn is great for this sort of work, provided they are using desktop machines as said below laptops are a bit of a problem.
However it can have its drawbacks….
A few years ago I had to go to a remote site to upgrade the network switch firmware. Because it was a 24x7 operation I needed to be on site to make sure the users where aware.
My change is scheduled for 11pm, about 10:20 all the agents stop taking calls. I thought this was strange and it turned out the automatic updates policy (which did a WOL call, applied the updates and then shut the machines down at the end) was applied to these machines.
So every night at 10:30 the machines would look for updates and it if didn’t find any shut the computer down….
I told the staff to raise it as a call with the service desk as they would get the policy changed. I left the company a month later and as far as I knew they hadn’t made the call yet - wonder why?
WOL was an easy option years ago when there was just one setting in the BIOS to enable WOL. I gave up on it when there were multiple settings that had to be made in the BIOS to enable WOL and the correct sleep states to allow WOL to function, then setting the network adapter in Windows to enable WOL, just to discover the WOL still wouldn't turn-on a computer. Several years later, HP and other PC manufacturers discovered a bug in their BIOS's that required a update so that WOL would work. I haven't even bothered trying WOL after the BIOS updates, I just have users leave their computers on. If I want particular computers to be available for remote maintenance in the evenings, I will just enable the automatic turn-on at a specific time every day.
I used to keep my work laptop in the bedroom in a backpack. It would regularly wake up in the small hours, sometimes fan would run hard, laptop got very hot and occasionally it'd flatten the battery. On investigating the event viewer I discovered it was coming out of sleep mode to do updates. The BIOS seemed to be able to wake the machine on a schedule. It took some digging to disable this feature.
If I'd shut down instead of using sleep, the overnight updates would've been deferred until switch on (HDD in those days, 5min+ to boot up). I was baffled by the maintenance policy.
About 15 years ago I was working for a business where each PC was powered via an adapter from which the monitor and any dedicated peripherals were also powered. When the PC was shut down, the rest automatically went off, too. No monitors left on (or monitors turned off and the PC left on). WOL took care of updates, as described elsewhere (but successfully).
I have a couple of such power blocks here, at home. If the PC power drops below a couple of Watts, it powers everything else down.
I made the mistake of plugging my laptop into it... When the battery is full, the charger pretty much shuts down for a few minutes (less than 1W draw), until the battery has lost 1 - 2%, then it starts up again... Very annoying when the external monitor keeps turning off mid sentence!
This post has been deleted by its author
We've been instructed to shut down desktop PCs at the end of the day. We've also been instructed to leave them on. So I shut down by rebooting, and leave. (That's before COVID-19 and us all working from home.)
While as far as I can see, software updates requiring reboot were and are issued at 11 a.m., although the reboot delay is set to 2 hours.
You do once. And only once in a career.
Leaving a freshly minted machine all alone without proper access controls where any Tom, Dick and Harry (it's never the women I note...) can mess with it.
In my work place an unlocked machine results in SFW leather fetishism image searches involving a serious gray beard and outdoor settings.
There is not enough mind bleach...
Anon because I'm supposed to work for a respectable company...
"Proper access controls" is such a situation mean, if I'm responsible for setting up a machine to be cloned, no one, absolutely no one except me will have physical or logical access to this machine before the job's done. I've learned this both the hard and dumb way. Hard: someone else messing up the machine and dumb: me forgetting what it was over lunch break (or night) and messing it up all by my own.
I've solve the 'dumb' issue with the 'do it three times' process. If i'm installing a new application on a new system, I document all of the steps necessary and I start counting at the first successfull complete installation. I then follow my instructions twice to make sure that I can, in fact, complete two successful installations. If I can, I know that I now have a procedure that my compatriots (or me...) can use to reinstall it without assistance.
A friend who works at a computer store, was brought a machine to clean (as in remove all traces of a web search) as his wife had seen him searching for 'local escorts'. (Definitely NSFW)
So he cleaned it and the customer was happy.
Friend's partner reckoned that searching for 'local escorts' would turn up lots of old Ford cars of that type, so opened up his mobile phone and searched for 'local escorts'. Hastily shot down web browser, and was revolted by what was displayed that we were not even allowed to view it.
It seems that searching for 'local escorts'* is definitely NOT SAFE FOR WORK OR HOME.
*Only try this if you have backed up your computer, disconnected from the backup and are prepared to do a complete rebuild after a full disk low level format. You have been warned.
(I wonder how many Register readers will try this despite the warning. Maybe go to your local computer store and try it there, if you must.)
*Only try this if you have backed up your computer, disconnected from the backup and are prepared to do a complete rebuild after a full disk low level format. You have been warned.
(I wonder how many Register readers will try this despite the warning. Maybe go to your local computer store and try it there, if you must.)
Using a private window in a Tor browser should suffice (and no, I am not going to try it, no need).
When I worked in product development I was contacted by some one (in the same company) who wanted to ship our product as part of theirs. I was asked could I get them an early driver. I said go through the official channels, but we can provide some alpha level code so you can develop your install procedures. We fixed the code so it had a drop dead date of 2 months away. "No problems mate". He got the install working, so was happy.
Two months later my senior manager got a call from someone angrily complaining that our product was rubbish - it kept starting and shutting down with no error messages. They were going to ship their product next week, so this was a critical problem which we had to fix. The demo to their executives was a disaster ... etc
Instead of getting the golden master of our product, they had cloned the software from my friends machine, without his knowledge and were going to ship it. I believe there was a "senior manager to senior manager" discussion about the incident, and I believe the person who caused all the problems was asked to "review his options" and moved to pastures new.
I learned that if you are going to give someone a b*llocking - be very sure of your facts, and keep it low key.
I work in Luxembourg.
Check it out on the world map. It's a very small country. You can cross it in an hour on the highway.
It's even smaller in the workplace. I learned that very early on in my career.
If you want to chew someone out, you had better be damn sure you're right, because your reputation can be trash quicker than you can blink.
It's a very small country.
You're being generous there. In matter of fact, Luxembourg's front drive opens onto Belgium and the gate at the bottom of the garden leads to Germany. If you want to visit France you have to jump over the neighbour's fence.
I was once in the receiving end of a phone call with an extremely irate ranter who went on for a good five minutes without letting me get a word in edgeways, so eventually I hung up on him mid rant.
He called back, got my manager, who was very apologetic about the hang up and the 'issues' being reported. Manager called me in, asked me how on earth I could treat one of our clients like that. I finally got to point out that the idiot wasn't a client, or even a prospective client. Turned out to be someone from a major multinational with its own IT department who'd googled 'IT support', rung the first number they found, and started ranting.
A friend of mine was demonstrating the marvels of ghost for rapid deployment of multiple workstations when he had a Bill Gates moment and the PC he was cloning to dind a classic NT blue screen. We all laughed a lot, not least since I had just made the jump to supporting UNIX by then (Solaris and DEC Alpha for the nostalgic).
similar story. We had built the first image for a power control plant that was to hook up and manage a number of nuclear power stations. Thanks to the local telecoms muppets we were way behind schedule and hadn't started our clone of the standard image on to the workstations. Until one evening, the system came online, and we started doing a long suite of tests with the power station. And left for the night.
When I got in next morning all hell had broken lose. People were going mental and the site managing director wanted to rip me a new one. It seems that we had sent a "SCRAM" command to the reactor. For those not knowing, this shuts down the reactor hard and fast, and can take months to recover from. I asked for an hour to find out what was going on.
With the hour up, we went back in to the meeting for round two of lets beat up the IT techies. Where I asked if "Jerry" was joining us (not his name). Jerry was there. He was from the company building the reactor control equipment that we connected to. His work was behind schedule, caused by our work being behind schedule. So I asked him how come all the PC's in the control office were all up and running. He then somewhat sheepishly admitted that he had, over night, cloned our working machine that was doing tests and installed the image over all the new machines that we had not yet configured. He then started doing some commands to see if the reactor was behaving.
The problem was that he had not changed the unique addresses of the workstations. This is pore internet, so they were sort of like IP addresses but exactly the same. The protocol allowed the boxes to come up. So Jerry booted all 8 boxes. And sent a command to the reactor.
Now the protocol was: Send a command. Reactor sends the command back for confirmation. The PC then replies, yes, that's what I want, and the reactor says OK, here is the result. Trouble was, one PC said yes, that's fine but 7 said - no, we didn't ask for that. So the reactor does another round of checks. And agsin 1 says yes and 7 say no. The reactor then assumes it's under attack and shuts itself down. Hence the scam command.
Fortunately we didn't cause too much damage as we were only talking to the control test rig, not the actual reactor. But it did get me the missing lock on our computer room and Jerry a flight home. Oh, and we got our names on rather splendid mural at the entrance hall for having saved the say!
The explosion icon, well, its obvious.
...and THIS is why I'm not comfortable with nuclear power (well, this and what to do with the waste, and what happens to the waste while we're trying to figure out what to do with it).
It only takes one "Jerry" to really mess things up. Sometimes, it's not even Jerry, it could be running out of fuel for the emergency cooling pumps because the fuel tanker can't get through after the tsunami.
When there's NO room for error, eventually, there will be an error.
Not only did this not cause damage, but it is designed to prevent damage. Had that been the actual reactor, the company would have been very angry. Because the reactor melted down? No, because it failed safe and the company would lose money. That feature is designed specifically so a "Jerry" or someone doing the same deliberately can not cause a safety failure.
Well that's the problem with CURRENT nuclear power. There are more modern designs that are truly fail "safe" and wouldn't have had a problem if installed in Three Mile Island, Chernobyl, or Fukushima.
Though connecting a PC to anything that can send commands to a reactor, even if it is air gapped, is beyond stupid. It is a lot harder, but you can get past that "defense" (see SolarWinds for one example of how) Those commands should have to be input directly from a control panel wired with something simple like serial cables to the reactor. Then you can concentrate on physical defense and not worry about cyberattacks.
TMI did fail safe. It was a financial disaster, but there was no human or environmental cost. It was just very expensive. It didn't even affect ongoing operation of the other reactors at that site.
Fukushima is obviously much worse, but I would argue that Fukushima is an example of how good Nuclear safety is. Considering how many things went wrong and the circumstances they were under, the harm was remarkably light. If the comparison is to fossil fuels...deepwater horizon is comparable IMO.
Chernobyl: Soviet Russia. Nuff said.
> but I would argue that Fukushima is an example of how good Nuclear safety is
Hardly. Fukushima's original design called for emergency generators in the basement of a building next to an ocean, which was approved; a mistake which was pointed out several times over the course of its operating life, including by TEPCO's own in-house engineers.
Of course, TEPCO ignored the reports and did nothing about it, every time, with entirely predictable results: a flooded generator room and subsequent inability to source emergency power for the reactor cooling system.
You trusted PCs to run the reactor? Holy crap! I interviewed at Rolls Royce Associates once, they do reactor controls for nuclear subs. They told me in the 1980s obsess over analyzing every component mode failure for discrete and small scale integrated parts, then throw their hands up and say it's too complex for the large scale ICs. I did some not entirely unrelated work for UK MoD. We didn't trust any commercial operating system for our safety critical work. PCs later killed US Ticonderoga class guided missile cruisers. Over the years the USN has talked down the severity, but I took some classes with one of the engineers who was on the ship at the time and he said they were dead in the water and needed a tow. To this day I would not use a PC to run a critical system.
"You trusted PCs to run the reactor? Holy crap!"
+1 on this. I used to do some support work on misc hardware for a couple of nuke plants. They were *extremely* conservative with changes, to say the least. I can't imagine they'd let a PC talk to the reactor.
I also did some work for an aircraft manufacturer. I questioned why some functions were implemented in complex looking hardware versus just using a small microcontroller like a PIC. Even for non flight-critical parts, it was too difficult to prove the software was always going to work. There'd be zero chance of getting a Windows PC approved.
"This is pore internet"?
I assume you meant pre internet? If I connect 2 machines with the same ip address to a DHCP router it'll refuse to connect them or maybe connect the first. The response from wrong machine thing I have seen, but usually when there's a 3rd party system running with 2 computers claiming the same application node ID.
> I assume you meant pre internet? If I connect 2 machines with the same ip address to a DHCP router it'll refuse to.....
I assume you missed the fun times.
Around 1991, my boss's boss called me in to look at a copy of NetWare Lite (it wasn't even Personal yet) and a box of used telephone cords.
FYI, the PCs in those pre-286 days were barely capable of being a DCHP server, and maybe DCHP services existed somewhere in the world, but for sure not where those useless telephone cords reached. IP in general was mature in well-wired campuses, but not out on the fringe. NetWare had well-developed protocols before/alongside internet protocol.
NWL was peer-peer, needed no central server. It used the netcards' MAC number (in ROM) for uniqueness. With proper netcards all worked well. BUT in those days the best-price netcards had their ROM images all cloned the same. Apparently this even worked in some network schemes. But I remember preparing custom "spoof" override files for each machine so they would be unique.
Much later we got into IP addressing (probably on a '386 NT server). There's a story here, and it was at work, but the tale is not safe for work(R).
At Xerox, with ALTO computers, the machine 8-bit address was wire wrapped in. This was OK until someone shipped a machine from one net to another. We were running byte-stream-protocol, leaf-sequin and a bunch of other pre-TCP/IP protocols. Very hard to find the interfering machine.
I was working for a large electrical manufacturer, and was sent to install some contactors in the control room at AWRE Aldermaston. The security guard who was sent to "accompany" us told us that in no circumstances were we to interrupt the 415V supply, we would have to make our connections onto live busbars. Not something I relished, but needs must. I asked why the supply was so critical, and he said that there was a reactor on the other end of that outgoing cable. To me, as an electrical engineer, a reactor is simply a coil of wire, sometimes with an iron core, that introduced a lagging factor into the current waveform. No, in this case, it was a nuclear reactor, and if the control power was lost, it could be "Goodbye Basingstoke".
Reminds me of an organisation that deployed brand new laptops for their developers. The machines were so locked down, they couldn't install almost any software required to do their job. For a couple of weeks those who had a personal laptop, had to work on that and those with desktops worked from home...
One guy offered to install everyone a fresh Ubuntu so they could get on with it, but management said what if those machines get hacked later on and we will be on the hook for allowing that? Yet, they had no problem with people working on their own computers for a time being...
We just sequestered first articles in a locked, limited access room. No one who was not part of the development team (5 - 15 people) was allowed access to this item until the pre-release HW/FQT and SW/FQT.
Then the HW configuration management folks were allowed supervised access to the system. They generated the final HW P/N and BOM.
Then the SW FQT was similarly run with a resulting SW P/N and BOM.
These days, a third group of Cyber Security folks also run audits with a CS P/N and BOM.
This system works well to this day. So far, given the systems sensitive use, we've never had an issue in 30+ years.
Expensive? Not really. The development budget may be $100,000 or so higher than a process that doesn't do this. The savings, imagine this story played out 1,000 times over a world-wide distribution... $100,000 is cheep insurance.
Working for an anonymous 'city' client in the early part of the Noughties, we discovered that our Sun Workstation profiles were copied from the system administrator's own profile. Including Internet favourite links with lots of pink images, and nothing to with finance.
I was totally unaware as I had not used the Internet from that machine (no need to as I had my company's laptop) until their head of IT security asked to see my web browsing history. Fortunately he accepted my story that 1) I had not accessed the Internet form that machine and 2) being gay there was no way I would ever had accessed those websites, let alone installed them in my 'favourites'.
Oh happy days. ;o)
At one time I worked for a large Fortune 500 corporation. Fantastic company, fantastic benenfits and perks.
Internet access at that time was restricted to users who could demonstrate a business case for access and included all of IT.
Every week during our group's weekly meeting, someone else would be missing, and we would be informed yet again that internet traffic was monitored and users visiting porn sites would be let go. Every week. A job with pay and benefits far above the norm lost over that.
Discovering that one of our IT staff had been hosting porn sites on the company servers for a couple of years provided a bit of... irony?
Ah, reminds me of being back in the day in a London borough, building custom 'gold master' Ghost images for various departments (Births, Deaths and Marriages still used Lotus Notes so custom image for that, etc), and remembering to make sure the UUIDs were set to be changed on first boot, etc.
I do recall that any 'master' machine I used was kept locked in a cupboard that only I had the key to, in a secure/maglocked workshop. Was the easiest way to keep unwanted hands away. Something I image a few people telling stories here learned as they went along - I was lucky enough that it was policy that no-one touches the master machines without written authorisation/project involvement, upon pain of actual, real disciplinary action.
Mind you, I built most images from scratch for each image every time anyway as with Win XP and downloadable service packs it was feasible to do that fairly quickly - I haven't worked on images since Windows 7 (and the wonders of Server 2k8 and WDS - pushing out 30 machines in 30 minutes from bare metal to working desktop, correct AD ownership, Office installed etc was doable) and I do linux these days so I'd assume it's a lot easier now, what with VDI being a thing.
Bloody glad we don't have to deal with Novel Netware and it's application deployment framework. Just take the bits that the application needs to run, and deploy it to the workstation.
Now, what does Docker do again....?
Truly, nothing is new.
Steven R
(just a nostalgia trip really)
So I worked at one certain 3 letter named Pharma in the early noughties, we had 3 standard tightly controlled images depending on job role/function & a menu system to install additional software on.
One of my colleagues got tasked with putting a fresh image on a suite of machines in a training room with suitable customations, this was commenced by a boot disc with a quick Q&A for things like department & machine name. This took him overall the best part of 4 days (Between other duties).
Within 3 days we get a ticket asking for help installing software by a seniorish manager, he's tried to install on half a dozen machines in that suite with it crapping out when it ask's for admin rights, so he tries another machine each time.
My colleague was most annoyed at the ticket, we flagged it to our boss who was even more annoyed & to her boss who also was most annoyed. Said manager was forced to pay the bill for the time spent re-imaging all the machines a second time out of his budget as he couldn't identify which he had attempted his installs on (Under one of the generic accounts).
My colleague (Who by now was not a happy bunny) duely set about the re-imaging task (Though I think our boss took pity on him & asked any of us who were free to assist).
I remember when we first got an internet gateway, online site checker. The board members were often up to no good and they were immediately all put into an exempt group that did not get their activities checked or recorded. This was done with the full knowledge that they were the worst bunch for just doing anything they bloody well liked!! The rest of us, especially IT were coraled into very restrictive checking groups where we actually bought a new ADSL line just so we could get security patches downloaded onto a PC, write to CDR, check for nasties and then put into the network!
This lasted about 2 months until the nasties found their way in and it took the CEO and the CTO days of begging to be allowed to put the board members into the checking groups and then it was only if the history was never recorded and was wiped every 24 hours! It took another 3 months before two board members were asked to leave when it was found one had been looking for call girls and one had actually downloaded some CP. Police involved and all that jazz, IT people called to report. Once that all happened there were no excuses, no exemptions under any grounds. It took some CP to appear on a corp network in order to actually do the right thing.
Back in the early '90's, I started work at a new company while it was literally in the midst of a move to new, larger facility. There were only two IT guys, and they were hugely overwhelmed, so setting up my workstation was one more thing to deal with. So someone got the idea to just clone the workstation of one of my co-workers who had all of the programs and utilities I would be using.
Except it turns out that, having had a private office, he had a "special" screensaver, unbeknownst to the IT guys. They set me up, turned on my workstation, and up came the cycling photos of the screensaver, which apparently was named "Beach Bums." A bit of panic ensued, and then they tried to disable the screensaver, which was having none of it. Finally, a manager came in, laughed, turned my monitor to the wall, and told me to take the rest of the day off while they "worked out the kinks." And apparently told said co-worker to be careful of what he loaded on his computer.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
