Seething CEO shoulder surfed techie after mistaken takedown of production server

Hold your nose, gentle reader, as we dive headlong into the bucket of ice water that is the start of the working week. But fear not, for The Reg is here to warm your innards with a dose of Who, Me? – our weekly tale of technical shenanigans gone wrong. This week our hero is a reader we'll Regomize as "Emily" who worked at a …

[Michael H.F. Wilkinson]

Calling all servers "server"

is a bit like the situation on Sqornshellous Zeta, where all mattresses are called "Zem". Makes it impossible for them to find out which one has been captured and killed.

I'd better be going. Mine's the one with the HHGTTG radioplay cassette tapes in the pocket

[Korev]

Re: Calling all servers "server"

You could also give each server the IP address 42.42.42.42

[Unoriginal Handle]

Re: Calling all servers "server"

The answer is South Korea Telecom.

Mine's the one with the complete Internet DNS registry printed out and bound, in the pocket ...

[stiine]

Re: Calling all servers "server"

Mine are all 69.69.69.69

[Paul Crawford]

Re: Calling all servers "server"

Mine is 68.68.68.68 and it seems I do the work and everyone owes me one...

[cosmodrome]

Re: Calling all servers "server"

what the hex... :)

[KittenHuffer]

Re: Calling all servers "server"

That sucks!

At least you can get a good lick in!

And everyone is happy!

Did you have to take an oral test to get the job?

[Anonymous Coward]

Re: Calling all servers "server"

Bit of a mouthful?

Better make sure you're not biting off more than you can chew. Actually, biting in general may be frowned upon.

:)

[Paul Crawford]

Re: Calling all servers "server"

Actually, biting in general may be frowned upon.

Ah, so you have seen Stag Night of the Dead ?

[Korev]

I'm pleased they didn't appear to punish Emily to server right...

[The commentard formerly known as Mister_C]

The punnish meant that she remained in servitude.

[Korev]

Hopefully the server didn't have to wait too long

Wanders off to wash mouth out with soap after making a pun in American "English"...

[Dinanziame]

"They also serve who only stand and wait"

[Red Sceptic]

Angry upvote

[Anonymous Coward]

They did punish Emily. However, Emily had created so many additional Emily's, even Emily is unsure which Emily actually got fired...

[ColinPa]

Labelling production

A colleague of mine went to a customer post mortem as to why they lost their primary server during production. It turned out that they did not identify which system/screen was production and which which was test, and someone had shutdown production by accident.

The top recommendation was to colour code the production systems so this could not happen again.

He went back 6 months later for another reason, and spotted they had not enabled the colour coding of production.

He pointed this out (forcefully!) and the customer quietly ignored it (this was Asia).

Two weeks later he was on a plane again to do another post-morten on why they had lost heir primary server during production - again!

This meeting was pretty short. There was one chart (so he said) saying "do the recommendations from 6 months ago"

I heard heads rolled - and he was not invited back again (which pleased him)

[Killfalcon]

Re: Labelling production

It's such a silly seeming thing ("make production red"), but colour coding works astonishingly well. Even if you don't consciously notice it, opening the wrong environment _feels_ off.

Might be one of the best time-returns there is in terms of preventing downtime. Simple to implement, simple to explain, basically no cost at all.

[Tim99]

Re: Labelling production

^This^. Many years ago I saw that a colleague had set a Windows admin account screen background to a bright red. I thanked them, stole the idea, and have used it on everything "admin" ever since.

[ComputerSays_noAbsolutelyNo]

Re: Labelling production

Some Linux distro, I don't remember which one, had a red background with a big cartoon-ish bomb with a lit fuze for its root account.

Subtlety is nice and all,

but there are situations,

when you want everything but subtlety.

[Anonymous Coward]

Re: Labelling production

I have iTerm 2 on my Mac set to produce a red background shading if it picks up that I've switched to root level privileges on any of my servers.

Simple but effective, but I do miss it when I use other systems.

[cosmodrome]

Re: Labelling production

IIRC it was an ancient release of KDE, independent from distro.

[tiggity]

Re: Labelling production

Might need more than just a red colour background, e.g. also text or an image to identify the type of server - as some people struggle with seeing some colours - if you must just do a colour, need to ensure your production and non prod backgrounds are identifiably different to people with the various variants of colour blindness.

[An_Old_Dog]

Re: Labelling production

BGINFO !!

And, carefully-selected hostnames go a long way in avoiding this sort of mistake.

[A.P. Veening]

Re: Labelling production

While your remark about colors is valid (deserving an upvote), you are overlooking that it is an individual account setting (deserving a downvote).

[Korev]

Re: Labelling production

It's such a silly seeming thing ("make production red"), but colour coding works astonishingly well. Even if you don't consciously notice it, opening the wrong environment _feels_ off.

Just make sure that the colours you choose are colourblind safe

[Paul Crawford]

Re: Labelling production

Or for a desktop that the background image is something that really reminds you to be careful. Like a bomb...

[Anonymous Coward]

Re: Labelling production

Make it something so offensive that people will want that off their screen as soon as possible - they won't stay at root level then.

Some people can't handle subtlety :).

[Anonymous Coward]

Re: Labelling production

There is *nothing* that's offensive enough for everybody, unfortunately.

[J.G.Harston]

Re: Labelling production

My Admin background is loads and loads of Windows flags. ;)

[Anonymous Custard]

Re: Labelling production

There is *nothing* that's offensive enough for everybody, unfortunately.

There seems to be a worrying trend of certain "celebrities" these days who make their livings trying to prove that to be wrong...

[Someone Else]

Re: Labelling production

There is *nothing* that's offensive enough for everybody, unfortunately.

That is true, but tRump orange should go a long way toward meeting that criterion...

[A.P. Veening]

Re: Labelling production

As the colours depend upon the individual account settings (and account sharing is a BAD idea), making sure the colours are colourblind safe isn't a valid concern.

[blackcat]

Re: Labelling production

Long ago I worked for a very small company and we had a windows server and a couple of linux servers in a room on a KVM. The person who looked after the windows server had a bad habit of giving the keyboard the three fingered salute to wake the server from screen powersave. Until one day the KVM had been left on one of the linux servers, which promptly rebooted itself.

[Alan Brown]

Re: Labelling production

"Until one day the KVM had been left on one of the linux servers, which promptly rebooted itself."

This can be changed or disabled. After the 3rd time a windows "admin" did that to a Linux box I provided, it was altered to respond with "Don't do that"

[Eclectic Man]

Re: Labelling production

Arthur Dent: I wonder what happens if I press this button

Ford Prefect: Don't!

Arthur Dent: Oh!

Ford Prefect: What happened?

Arthur Dent: A message appeared saying "Please do not press this button again".

(Probably mis) -remembered from the Hitch-Hiker's Guide to the Galaxy, series 1, fit the second

[Anne Hunny Mouse]

Re: Labelling production

This happened to us on a former gig with a Sun Boxes..

[ComputerSays_noAbsolutelyNo]

Re: Labelling production

Power move: "do the recommendations from 6 months ago"

[Pascal Monett]

And if you don't, don't come crying when it all falls down . . .

[JamesMcP]

Re: Labelling production

Good advice but some tools tend to "forget" those settings. I.e. Microsoft Sql Server Management Studio can have customization settings for each specific server BUT those go away during most software updates. Which happens a lot more than you'd think.

The work-around, such as it was, was to use the "default" color for prod and use special colors for dev/uat. That way if someone connected to dev and got the "prod" colors, they would know to go re-apply the config change.

[cosmodrome]

Re: Labelling production

Yes, what if not a database server woud be expected to drop information when updated?

[DS999]

Re: Labelling production

I was doing Unix/storage consulting at a place that was implementing SAP. They had just gone live, and were still busily adding functionality to production while they brought different business units into the fold. So the test environment was very active, as one might expect.

The way they had laid it out was that one server in the test environment was the configuration master, and a previous sysadmin had set it up so that it was trusted by everything else in the environment for passwordless root login (leave aside the security concerns, this was 25 years ago when that sort of thing was normal) The configuration master was very active, since any changes to the environment had to be made on the CM server first, copied to a test instance for testing, then to QA, and finally when properly blessed to production.

Luckily not everything had the same name, so the server's hostname showed up in the prompt (which makes me wonder if every "server" in Emily's story had the same root prompt) You'd think that would be enough of a hint, but I guess not as not once but twice one of the client's sysadmins meant to reboot another system but rebooted the CM server because it was in every login chain.

I made a simple alias in the root shell startup across the whole environment, so you had to type the hostname as part of the command to reboot it. So if you were on server foo and issued the reboot command it would say "use reboot-foo to reboot this server" and if you typed reboot-foo it would execute the reboot command.

I don't think I'd ever seen anyone color code environments back then, but since their problem wasn't rebooting production systems they'd have needed to color code a server in the test environment. Which was at that time almost as important as production!

[TSM]

Re: Labelling production

A teammate and I independently selected the same two contrasting colour schemes to differentiate between production and test terminal sessions.

However, the colour scheme he uses for test is the one I use for prod, and vice versa.

Always makes it fun when we have to look at things on each other's computers :)

[that one in the corner]

Avon calling

The only way to tame a server-LAN.

[Chloe Cresswell]

Re: Avon calling

Upvote, standard by 2

[UCAP]

Re: Avon calling

Vila might be useful to help unlock the meaning of that pun.

[Paul Crawford]

Re: Avon calling

Ask Orac, cheaper than Oracle

[Chloe Cresswell]

Re: Avon calling

And be patronised for the use of his intellect that is so far below the level he thinks at? No thanks, I'll just remove the activator.

[Paul Crawford]

Re: Avon calling

I'd rather be patronised by Orac than liberated of so much money.

[Eclectic Man]

Re: Avon calling

That's "Madam President" to you!

[Bebu]

License tied to host name?

I thought the classic flex lmgrd licenses tied to the hostid (normally one of the host's ethernet interface's MAC address) was pretty feeble (and easily subverted.)

I assume Emily's software used gethostname(3)/uname(3) etc to retrieve the host name rather than retrieving the host's IP address(es) and doing a reverse (IN PTR) lookup(s) for the (FQ) host name. Either way fairly easy to subvert just for the applications benefit. :)

DEC's lmf and hardware dongles were the only two that I declare no contest.

Always had (more than) enough licenses but trying to move/modify the licenses, typically when a motherboard or ethernet interface has been replaced, makes having teeth pulled seem pleasurable.

The main problem is that application developers know SFA™ about the licensing code which was normally purchased (licensed) as a secret sauce library that was added to the application code without further thought or understanding.

The license file generation application from the secret sauce package is given to support with even less understanding so that new licenses or renewals are mostly fine but anything else gets duck shoved back on to the developers who likely being a new crop have even less idea again.

The application's vendor usually being on another continent (arguably another planet in many instances) it is rather difficult to turf them out the window so the thwarted BOFH is compelled to exercise his or her powers of deviousness to keep critical applications running.

(Having an intimate knowledge of the toolchain, truss/strace and adb are very devious enabling. :)

[Anonymous Coward]

Re: License tied to host name?

Is that like virtualising a server and finding out that the application running on it had a hardware key plugged into the parallel port...and the new vm server doesn't have a parallel port...

[DougMac]

Re: License tied to host name?

There exists software hardware virtual port redirectors over the network for situations like that.

Provides a virtual serial or parallel port on a VM to talk to the hardware box that has the hardware dongle somewhere else on the network.

You did have to do weird things in the past, like figure out how to load the USB stack by hand, because the VM didn't trigger putting on in, but your software needs it. Gotten smoother now.

But once it was going, it worked a treat.

[Tabor]

Re: vm dongles

I feel your pain. Had it 2 months ago, but luckily it was a usb port. I don’t like usb passthrough (mostly because it’s an embuggerance if you move vm’s between hosts) so bought something from SEH, a network device that supports all kinds of dongles. utn server something. Works really well, no idea if they selll the same stuff with parallel ports. Or if it would work with usb to parallel adapters.

[TheWeetabix]

Re: vm dongles

Embuggerance. Love it.

[Anonymous Coward]

Re: License tied to host name?

Licence daemon crapware is invariably so infuriating to try to get working as to make even scrupulously honest businesses give at least passing thought as to whether it might not just be easier to pay for the licences but install pirate versions of the actual application instead… (only half-joking)

(It's also certainly another point in favour of choosing open source applications (+ suitable donations) rather than user-hostile commercial apps, if you can!)

[Oh Matron!]

Managing apple....

I work in MDM..... And one of the biggest issues I face with customers is that apple, when bring a device under management, call the device, well, you've guessed it, iPad or iPhone

Why they can't use some non persistent random string, I don't know. I do get customers calling to ask if a device can be brought back under management because the wrong "iPhone" was wiped...

Bootnote: Yes, I know, devices can be renamed, but, if you've every worked with managing devices in education, you'd be fully aware of the toomfoolery that the yoof of today get up to.

[Pascal Monett]

if you're working in education, you should not use a vendor who doesn't give a fuck about how you work.

[Korev]

Don't like your educational software vendor, then rm them....

[parrot]

rm -rf (not big not clever)

Many years ago on 1 April, whilst working in a school, I logged a support ticket with RM saying their command had broken my Linux server. It was a stupid obvious joke and wasn’t really very funny, but I must have worded it more convincingly than I realised because I got a call from an account manager who sounded very worried.

Somehow the words “April Fool” did little to diffuse the situation. My boss seemed unconcerned and just sat opposite laughing.

[Not Yb]

OK, that joke is much better than it looks at first glance.

[Anonymous Coward]

you should not use a vendor who doesn't give a fuck about how you work

Where I work, there's just been an update on what devices are allowed where on site. I've been told (third hand so could be spherical(ish) body parts) by a a colleague that it went along the lines of :

Apple added some "AI foo" in the last iPhone update. Business asked Apple for details of what it did, how, etc. - all the questions that a business involved in sensitive government work would reasonably ask and expect answers to. Apple didn't answer. iPhones now banned on security grounds - they were previously allowed into certain areas and for certain types of work.

[jmch]

too many servers

https://www.best-poems.net/poem/too-many-daves-by-dr.-seuss.html

[The commentard formerly known as Mister_C]

Re: too many servers

Dave? Dave's not here

https://www.youtube.com/watch?v=ACfNSqsi_mE

[J. Cook]

Re: too many servers

How about Bob, then? Is Bob there?

[J.G.Harston]

Re: too many servers

Hello Dave.

https://www.youtube.com/watch?v=lnymNwClMWc

[chivo243]

Who Is, Who Has?

wouldn't all servers named 'server' create an awful lot of chatter?

[Dizzy Dwarf]

All servers should be called ...

... gandalf, elrond, frodo, strider, sauron, etc ... everyone knows this.

[jake]

Re: All servers should be called ...

Naming systems after Tolkien characters officially became old after I ran across the fifth server named "Bilbo" in a single day (two at Berkeley, one each at Stanford, San Jose State and Mission College). That was in roughly 1980.

[A.P. Veening]

Re: All servers should be called ...

The current version of that standard uses characters from Game of Thrones.

[Not Yb]

Re: All servers should be called ...

Hodor would be the license server?

[Anonymous Coward]

Re: All servers should be called ...

Does it show my age that I once had servers named Crichton, Zhaan, Sun, Rygel, ... ?

These days my home servers have names like Saffy, Patsy, Serge, Bubbles ...

Yes I have a beard, and yes it's grey

[Niek Jongerius]

"the server she had just restarted was not in the test environment"

Been there, done that. Trying to restart the SCO Unix box I was working on, and after the "reboot" command was greeted with "connection closed by foreign host".

[Boris the Cockroach]

It was

build the machine network time at my place of attendence...

Machine #1 duely checked for its IP and its 234.123.123.56, ok stick that on the router

Machine #2 checked and its 234.123.123.56 too

Hearts sank

Machine #3 yupp the same...

Machine #4 always hope... nope

Machine #5 thats brand new... bugger

Machine #6 bloody germans !

etec etc etc

So its dig out the manuals .. and the controls are slightly different due to their age..... so those efficient germans changed the way you set the IP on nearly every one...

Wheres the 'banging head against a wall" icon ?

[Emir Al Weeq]

Marklar

Here on Marklar, all servers are called Marklar.

[Anonymous Coward]

Let me point out that FlexLM, and many other License Managers, determine whether or not the server they're running on is the "official" one by checking the MAC address and/or other serial numbers of the computer.

Back before the "TPM" existed, serial numbers were frequently stored in flash memory, and were fairly easy to change on certain platforms. The only time I've seen this used, was when another company, during a somewhat hostile spin-off, took all of Company 2's servers off the network that had Company 1's desktop clients on. No more licenses for anything that Company 2 "owned", and getting the licenses transferred would take about 2 weeks and probably lots of legal wrangling. In order that Company 2 didn't completely waste several weeks on employees waiting for licenses, someone else at the company setup a machine to have an identical MAC address and serial number to the official license manager.

Shortly after all of this, the 3 companies involved went bankrupt, and the CEO of one died in the middle of a board meeting. Terrible business all around.