The self-disconnecting switch: Ghost in the machine or just a desire to save some cash?

The weekend is a day away, but before you swan off, please join us for another episode of ticketing system terror with The Register's regular On Call. "Forgive the beancounters, for they know not what they do" might chime with today's contributor, Reg reader "Arnold", who found himself (and his customers) on the sharp, pointy …

[defiler]

I had all kinds of hacks in a previous job to get good VMware functionality for minimal cost.

Amongst my favourites were my multi-threaded VM export which was used as part of the daily backup, and using OCFS2 for the VM storage so that I could move VMs between hosts quickly. Of course this was with GSX. When ESXi became free you could use shared VMFS volumes.

[Evil Auditor]

And this is one of the reasons why no one should have access to your stuff without, at the very least, you being notified beforehand. (If your housing provider goes bust, you might be out of luck though.)

[TonyJ]

"...And this is one of the reasons why no one should have access to your stuff without, at the very least, you being notified beforehand. (If your housing provider goes bust, you might be out of luck though.)..."

Indeed. 13 or so years ago, I was leading a server room migration. The company had moved out of an expensive rented office, with a rented space in the server room to a edge-of-city, less expensive premises and rented a cage in a nearby hosted data centre.

We'd basically done the physical move and we were getting on with the second part - a systems upgrade and refresh.

I was sat in a status meeting with their programme manager when someone popped their head round the door to say "all the IT is down."

A quick check and something was indeed wrong at the server end.

So off I toddled to the DC to investigate. I walked into the cage to find the DC contactractors installed a cable tray above the customers racks. Thoughtfully, to stop dust and debris, they'd covered the racks in blankets from top to bottom...yup...no airflow. Overheat and shutdown.

Yep there'd been alerts, but alas the persons that would see them, were in the status meeting sans mobiles as the programme manager didn't like folks having them in his meetings (that rule changed pretty quickly as you can imagine).

I was less than impressed with the DC hosting staff - no warning, no request for permission to enter the cage, nothing. Not the electricans' faults - they were handed a job card and were following it.

[Anonymous Coward]

Electricians

You don't happen to remember who the electricians were do you? If they really put down dustcovers over kit rather than the industry-standard method of coating everything in half an inch of concrete dust and copper strands from the cut-up cables then we might have some work for them.

Overtemps can be recovered from more easily than a fire caused by a screw dropped into a PSU.

[steviebuk]

Re: Electricians

I know a company we use that know to cover the racks but depends where you're based.

[JimboSmith]

Almost the opposite of trying to save money happened at one biz I worked for. The company ran a radio network that was broadcast via satellite. Subscriber radio stations would sell advertising themselves and then played it out locally. It was triggered by either a data stream or DTMF tones in a jingle depending on which equipment the subscriber had bought. The data stream boxes were quite a bit more expensive than the DTMF ones. Therefore those who had bought the more data stream boxes were annoyed they could hear (part of) the DTMF tones.

The bloke who looked after relations with the stations was tired of the feedback. So unbeknownst to us he contacted a firm who said they could build a piece of kit to fix this. They came in having analysed the issue and said their vapourware box would solve this. It would add half a second delay (don't think this was variable) to the audio only. Therefore the data stream would trigger those boxes first and they wouldn't hear the DTMF tones anymore. They wanted some money upfront to produce a prototype which was interesting as well. Myself and my boss excused ourselves for a minute and after a brief chat decided this was a waste of cash. We came back and said we could just add half a second of silence to the start of the adbreak jingles. No complicated box to go wrong or even be built for that matter. Relations bloke was stunned we'd solved the problem so easily. The company left with their tail between their legs. We told the relations bloke to consult us first before going off contacting firms for technological solutions.

[TRT]

Cardax locks should be on the front and back doors of ALL co-loc cabinets, along with rack top web-cams facing front and rear. If you can't lock the room, you lock the cabinet.

Physical access control should be part of the lease agreement.

[TonyJ]

"...Physical access control should be part of the lease agreement...."

Yuh, it was. The DC owners chose to ignore it.

CCTV coverage was ubiquitous, too. Also ignored.

To be fair, it was tier-3 at best and you get what you pay for.

[John Brown (no body)]

"Cardax locks should be on the front and back doors of ALL co-loc cabinets, along with rack top web-cams facing front and rear. If you can't lock the room, you lock the cabinet."

Except, of course, when you only rent half a rack, as per the situation in the article. :-)

[Alan Brown]

"Except, of course, when you only rent half a rack, as per the situation in the article."

Rent half a rack, rack should come with doors to match and a solid internal divider

[shedied]

How to tell which half is yours

Ah yes, the internal divider. One must have it installed correctly: the other side (other company's side) can be left unpainted, but covered with the filthiest porn available. Nobody asks questions, but you will know for certain which half belongs to your firm.

[Anonymous Coward]

> And this is one of the reasons why no one should have access to your stuff without, at the very least, you being notified beforehand.

Back when I used to be in field services, I once had to do an install at a colocation data center with a large number of civilian government customers. There was a great deal of security to pass through before reaching the raised floor. Once there, however, it was just one giant open room.

One of the site staff was giving me a tour and at one point I remarked to him what a bad idea it was not to have individual cages for each customer, which was the norm for the industry. I was actually shocked at the lack of physical separation.

Just my luck, a senior tech for one of those government customers was within earshot of my conversation. He didn't like what I was saying at all and voiced his displeasure to the facility's management. Shortly after, security approached me, escorted me out of the building, and informed me that I was banned for life.

About a week later, I was informed that the government customer I was there to do work for (a different one) had just installed a cage around their servers, the first for any customer in the facility. I got a laugh out of that.

[jmch]

How much?

This is completely not my area of expertise, but I'm sure someone out there knows...

Exactly how much cost savings could you achieve by renting half a rack instead of a whole rack from a 3rd-party datacentre?

[David Robinson 1]

Re: How much?

Oh, you sweet summer child. You have no idea how the mind of a beancounter works.

[Anonymous Coward]

Re: How much?

We decided to stop kitting small hardware and just put it in big bins on the shop floor and have the supplier manage the stock directly. The chief accountant vetoed this because the supplier might bill us for 1000 M3 screws but only supply 999 and thereby get rich on our lack of controls and leave us with unbalanced books. Cost of M3 screw? much less then 1p; cost of ordering, receiving, checking, storing, kitting M3 screws - many, many times more than 1p per screw. He was eventually put in his place by the MD - probably at the Christmas party, held in London and they all stayed up there for the night - paid for by the company.

[trevorde]

Re: How much?

Worked in engineering, many years ago, and was in the weekly engineering meeting (circa 1995). There were 10 or so of us, so we were burning $1000 AUD per hour. The manager and some other people spent 10 min debating whether M8 or M10 bolts would be sufficient for some job; whether or not they needed to be stainless steel; and the grade of stainless steel. I was thinking: "There must be at least 10K bolts involved here; probably more like 100K the way they're arguing." Turned out it was a mere 10 bolts! They had just burnt $150 AUD on something that was worth $10 AUD. That is when I learned about a 'bike shed' (https://en.wikipedia.org/wiki/Law_of_triviality).

[lglethal]

Re: How much?

Not knowing your situation, but I've been in similar meetings, and they can be super important. If the risk is your bolts failing and your part falling off an aircraft, then $1000 to get 10 engineers in the room and decide on the best screws to be used is definitely worth the cost. Same goes in the Space Biz - You screw up there and you cant exactly fix it at a later date.

Oh and 10 bolts might sound like nothing, but if your series production is rated in the millions, a few cents saved per screw can add up pretty quickly.

So such meetings can definitely have a use. They can also be a complete waste of time, but i rarely find thats the default setting. Unless upper management is involved of course, then they're guaranteed to be a waste of time... ;)

[Anonymous Coward]

Re: How much?

Many years ago manglement decided that the storemen needed to keep a log of their work so that 'costings' could be made (in an attempt to slim the headcount) and a time of 2mins was allocated to book each piece of kit in... so that's 2mins to book in a high-end data centre router filled with lots of cards, lots of serial numbers and extras... and 100mins to book in a box of 50 microfilters

For some reason it didn't last long.

[ICPurvis47]

Re: How much?

When I was a Development Engineer in a large electrical manufacturing company, we were developing a line of heavy current switches (25kA @6V) for use in chemical plant. I specified that all the hardware (ie, nuts and bolts) were to be of A4 Stainless Steel. The bean counters objected and said we had to use standard High Tensile Steel bolts, chemically stripped, replated and passivated in our own plating works, because they were so much cheaper. I looked into the cost of this plating and passivating, and found that it actually cost us more to do this post treatment than buying stainless would, but as it was performed in-house, it didn't appear on the bean counters' spreadsheets, and so wasn't factored in. It was a long battle, but I had my way in the end, and the switchgear lasted for at least five years in a highly corrosive atmosphere.

[Evil Auditor]

Re: How much?

... how the mind of a beancounter works.

Or financial auditors, for that matter. This is like a beancounter on steroids. After the audit: "your petty cash is off by £2.40." (And that was the only finding he had. I don't remember though if it was 2.40 or 1.40)

[Aladdin Sane]

Re: I don't remember though if it was 2.40 or 1.40

But I do remember that it had a heart of chrome, and a voice like a horny angel.

[IHateWearingATie]

Re: I don't remember though if it was 2.40 or 1.40

"But I do remember that it had a heart of chrome, and a voice like a horny angel."

Have an upvote for maybe the first instance of a Meatloaf quote sneaked in on the Reg forums?

[Prst. V.Jeltz]

Re: I don't remember though if it was 2.40 or 1.40

You took the words right out of my mouth!

[The Indomitable Gall]

I would do anything for an upvote...

...but I won't do that.

[RizKat]

Re: I don't remember though if it was 2.40 or 1.40

That's a Bad Attitude...

[j.bourne]

Re: I don't remember though if it was 2.40 or 1.40

Voted down for a quip too far.

[Prst. V.Jeltz]

Re: I don't remember though if it was 2.40 or 1.40

Two out of Three aint bad ....

[BillG]

Re: I don't remember though if it was 2.40 or 1.40

You've got a hell of a lot to learn about Rock n' Roll.

[TRT]

Re: How much?

Hm... we had to deal with a Mr Jones, who was Regional Head of Security. He was an unholy trinity of financial auditor, Stasi officer and red team.

If the cash receipts in the store were down £10 at close-of-business on Monday, and up £10 at close-of-business Tuesday, then he'd be waiting at the shop door at 8am on Wednesday wanting an explanation for the £20 discrepancy. You can bet he'd have actually arrived at the store at 7am, had commanded the mall security (successfully) to let him into the back-of-house areas, and he would have spent that hour trying to pick the locks of and jemmy open the back door, jemmy the stock room windows, remove the extractor fan covers... he was the kind of a person who carried a large brief case around with him, said to contain a 6-cell MagLite, step-ladder, crowbar, screwdriver, lock picking set, smoke bombs, and a calculator with a roll printer. The last implement being the most feared of them all.

[Antron Argaiv]

Re: How much?

We engineers (and BOFHs) love the Mr. Jones type.

A file cabinet left unlocked? With a spring-loaded tray of confetti in the middle drawer?

...or worse. But let's start off with a gentle warning first. We know what you're up to, Mr. Jones, and it needs to stop. Otherwise, Steps May Be Taken. Steps, which we hasten to add, could lead to an unpleasant day for you. But we wouldn't know any thing about that, would we?

[Doctor Syntax]

Re: How much?

"Steps, which we hasten to add, could lead to an unpleasant day for you."

Steps down which you might fall. Onto the roll of carpet.

[TRT]

Re: How much?

Bastard Store Manager From Hell...

Our store manager, Nick, was ex- Household Cavalry. A good 7 footer, hands the size of ham hocks, could probably have carried his horse round an assault course. Lovely bloke, but yeah, the back stairs up to the office and stock room WERE very narrow, not well lit, very steep and covered in loose concrete dust from where Nick kept hitting his head on the overhang, gradually chipping a head-shaped hole into the bottom of the beam. A carelessly unboxed R/C 4x4 monster truck left upstairs...

Ours wasn't a carpet shop, unfortunately; that was next door. We were unbelievably overpriced consumer electronics, but there was a parapet outside between the first and ground floors that covered most of the width of the mall's walkways in an attempt to provide shelter from the near perpetual rain of that city. It didn't work - the wind through the artificial canyons just blew the the rain sideways and carried it a good 20 yards into the interior. One side of our store wasn't overlooked by the tower blocks, so you could have hidden a body up there for years, it was so covered in discarded sand bags and assorted building and shopping centre detritus.

[Anonymous Custard]

Re: How much?

"Steps, which we hasten to add, could lead to an unpleasant day for you."

Steps down which you might fall. Onto the roll of carpet.

All this talk of steps is giving me flashbacks to last week's installment...

[TwistedPsycho]

Re: How much?

.... and then you wait for him to over steps the mark, in 5, 6, 7, 8...

[Dyspeptic Curmudgeon]

Re: How much?

"With a spring-loaded tray of confetti in the middle drawer?"

Not confetti. If you can find a source, the chads from Hollerith punch cards work MUCH better.

My brother found punch card chads in a suit pocket 10 years after his wedding....

[Daytona955]

Re: How much?

Don't mess about with paper products not normally found in an office environment. The proper substance to use in these situations is toner...

[disgruntled yank]

Re: How much?

At one of my first jobs, the accountant told me that one can be off by thousands (45 years of inflation ago, so no doubt millions now) in other accounts, but somebody will notice immediately if petty cash isn't perfectly balanced.

[Anonymous Coward]

Re: How much?

That is correct. Was hired by a large US firm for an emergency team to clear out a few imbalances because the auditors would not sign off on the books.

The imbalances totaled over 1 Billion $US. That is not a typo. The auditors were willing to sign off if we were able to get the errors under the level of materiality. Which they defined as 1 Million $US.

Two years later was working for a company that was spending $1,000 per month on monitoring the petty cash which on average spent $500 per month. My suggestion to just give them $750 a month and don't monitor it anymore, which will cut total expenses in half, was not well received. I then seriously suggested to just give them a credit card with a $500 limit. This would cut out the bulk of the monitoring costs which were required because cash was present. This was also rejected because "somebody could misuse the credit card." Yes, but now you have an audit trail of the expenses and even if somebody does abuse it to the full limit once a month you know who did it and you would STILL be saving money.

Nope.

[Elfoad Regfoad]

Re: How much?

In 1986, someone investigating an unpaid $0.75 USD computer bill in California found it was caused by a German hacker, who worked for the KGB, was using their system to break into military computers.

https://en.wikipedia.org/wiki/The_Cuckoo%27s_Egg

(If you aren't familiar with the story, the book and the NOVA TV episode are both interesting)

[Prst. V.Jeltz]

Re: How much?

That wiki page seems to have left out the hacker's LSD habit , and eventual suicide by fire.

[T. F. M. Reader]

Re: How much?

@Evil Auditor: "And that was the only finding he had."

You underestimate the deviousness of beancounters. Any beancounter worth his salt will take $2.40 from petty cash before an audit on purpose. It is very important to let the evil auditor find something trivial and obvious so that he leaves with a feeling of a job well done.

[chivo243]

Re: How much?

OH, yes I do, How much? Too much!

[Anonymous IV]

Re: How much?

> Exactly how much cost savings could you achieve by renting half a rack instead of a whole rack from a 3rd-party datacentre?

Enough to satisfy a bean-counter, I would imagine.

[Ordinary Donkey]

Re: How much?

So a bean and a half, a bean and a half, half a bean and a bean and a half?

[Anonymous Coward]

Re: How much?

Ahhh.....so the answer is "some beans"

[Will Godfrey]

Re: How much?

Or even "has-beans"

[David 132]

Re: How much?

Baldrick, the ape-creatures of the Indus have mastered this... now, try again...

[SImon Hobson]

Re: How much?

Or the start of a very small casserole

[David 132]

Re: How much?

No, we've moved on from advanced mathematics, now we're doing elementary dress-making. What do you think? Thinking is so important.

[defiler]

Re: How much?

Not much. Not much.

A half rack is maybe 25-30% cheaper than a rack. But if you start actually trying to fill it with equipment it's amazing how fast it goes!

For a start, it's not an *actual* half-rack of 21U - it's more like 18U.

Add a couple of PDUs - 16U.

Add a couple of switches - 14U

Add a failover router/firewall - 12U

Lob in a SAN with some decent storage - 8U

Chuck in some compute nodes - 4U

None of these are large items, and you now have enough to run a moderate (tightly-packed) network, but you don't have any space to grow.

If you're just chucking in some routers and switches I can see that a half rack would be plenty. But by the same token when was anything ever specced up that didn't grow and grow and grow?

[Anonymous Coward]

Well, two switches maybe would have still filled half a rack?

I know core switches are a bit larger than other ones, but the lesson here is maybe that core infrastructure should not be installed in a shared facility with pitiful access control? Who let the technician remove another customer's device? The fact they discounted the full rack price later shows they understood they made a big mistake.

[Prst. V.Jeltz]

Re: Well, two switches maybe would have still filled half a rack?

two words: Academic Institution

[TRT]

Re: Well, two switches maybe would have still filled half a rack?

The asset tags alone should have clued the bailiff in. What do you mean "what asset tags"?

[Doctor Syntax]

Re: Well, two switches maybe would have still filled half a rack?

As a precaution against bailiffs - add asset tags showing it belongs to someone else.

[Robert Carnegie]

Re: Well, two switches maybe would have still filled half a rack?

As a precaution of bailiffs - probably are prepared for that one and don't take any notice of asset tags?

[Anne Hunny Mouse]

Re: Well, two switches maybe would have still filled half a rack?

We used to use the metal backed asset labels which were hard to remove and left a mark if someone removed one.

However, these were deemed too expensive and replaced by standard plastic labels which can be very easily removed, leaving no trace on the equipment.

[wyatt]

Slightly different to saving cash but I've failed to secure equipment only to have it fall off its mount. We had an enclosure in the back of the vehicle containing a Nortel ATM passport, switches and various other bits of expensive kit. I'd not secured it correctly having been working inside it and drove off, only to hear a loud 'thud' as it moved as I went round a corner.

Fortunately all the armoured fibres that were connected stopped it from falling too far and I managed to put it back with only a small dent in the side. It still worked..

[Anonymous Coward]

Armoured cables no problem mate

I worked with some engineers who looked after the EPOS systems on cross channel ferries. If the EPOS system was down there was the potential for untrustworthy staff members to sell their own products from the Calais supermarkets pocket the cash and then reduce the sales figures. The company knew it was happening as sales would be 20-30 % lower if the EPOS systems were 'out of order' than if they were working. The nefarious staff members used the excuse that if there was a heavy sea the equipment moved and stretched the cables, that wasn't their fault. heavier cables were provided which were screwed into connectors but these still 'snapped' you could literally have towed a car with these. In the end the tills printers etc were bolted to the counter, somehow the cables still snapped, armoured cables were then installed and the kit was finally 'fit for purpose'. I have no idea what we billed for all this non-maintenance work but it must have been worth it as the Ferry company kept paying.

[Headley_Grange]

Re: Armoured cables no problem mate

When surveillance cameras were first introduced to supermarkets in the 80s they were placed over the tills to watch the staff, not the customers.

[Doctor Syntax]

Re: Armoured cables no problem mate

I remember a case from about then. Not what you'd call a supermarket but a privately owned shop big enough to employ several staff. The owner couldn't have been keeping close enough watch on his accounts as it was either his accountant of bank who warned him. Surveillance cameras went in and worked out that several staff were working together. The cashier would wipe her hand along her thigh which looked innocent enough until you realised she'd palmed a note and the action rolled it up. The note was then deftly passed to another member of staff who just happened to walk by.

[keith_w]

Re: Armoured cables no problem mate

Subway sandwich shops still do that.

[Anonymous Coward]

Single point of failure

Talking about single point of failure, I worked at Canal+, a French cable network. In 2016 I had the opportunity to visit their broadcasting center; well, I don't know the exact term, but it was the building from which left all their content, sent to the French TV network, to their satellite broadcasts and to internet. Apparently it was the only such site, they had no redundancy. So if it ever went down, Canal+ wouldn't be broadcasting anything and couldn't do any failover.

[Aristotles slow and dimwitted horse]

Re: Single point of failure

DR and that sort of thing is not really relevant to broadcasters because everything is shown an hour later on their +1 channels anyway.

[Yet Another Anonymous coward]

Re: Single point of failure

The beancounters suggested that they record the - 1 channel and play it back an hour later = a big saving in production ciata

[Anonymous Coward]

Re: Single point of failure

Their +1 channels were also broadcasted from the same building. Everything they produced was sent out from a single building.

[Yet Another Anonymous coward]

Re: Single point of failure

That must have been exhausting for the actors to have to redo each scene an hour later

[Anonymous Coward]

Re: Single point of failure

It's even worse for the Cartoon Network. Those poor animators...

[David 132]

Re: Single point of failure

Wasn't it Futurama that once had a titles caption: "Hand-drawn in front of a live studio audience!"

...or am I misremembering?

[TSM]

Re: Single point of failure

"Painstakingly drawn before a live audience". As seen at #14 on

https://www.buzzfeed.com/andyneuenschwander/25-funniest-futurama-title-jokes-ranked

[Anonymous Coward]

Re: Single point of failure

There was also a Simpsons episode where Homer was doing cartoon voiceover work. He asked if the episode was airing live. His co-star deadpanned "no Homer, cartoons are rarely done live anymore, it puts a terrible strain on the animators wrists"

[Andy Miller]

Bailiffs

In a previous century the company I was working for wanted to port their software from HP Rocky Mountain Basic workstations to these new PC things. We hired a sub-contractor to assist, which involved loaning them one of the large and expensive optic fibre test machine that we made.

At some point I got a whiff that the subby might be having cash-flow problems, so I suggested to our TD that it might be worth us moving the kit back to our site. The subby could carry on working on it, but in our office. So the TD and I get into the biggest car we could borrow, and set off for the subby.

Shortly after, we are sitting in the subby's MDs office, and the MD was denying that there was a problem, when the secretary pops her head round the door to say that there are a couple of bailiffs in reception. The MD kept the bailiffs talking whilst me, my TD and the subbies software engineer passed our large optical instrument and the subbies development machine out the window, into the car and away....What fun !

[Anonymous Coward]

Re: Bailiffs

Reminds me of the time that we had a round of layoffs (in a scene straight out of Office Space) and I packed my stuff in cardboard boxes, on top of a rolling office chair. The chairs were no Herman Miller

[John Sturdy]

The article title made me think of the Useless Machines, which switch themselves off, like this one: https://www.youtube.com/watch?v=Nqk_nWAjBus

Surely a good addition to any data centre.

[Anonymous Coward]

Not just the beancounters

Sometimes it's PMs and architects.

At an old job, after many a TITSUP TPTB finally signed off on new datacentres. Costed to run in 2 datacentres each with cluster hardware split across physically-separated racks. Almost too good to be true you cry!

Staff arrive on site to setup the first cluster and it's all in a half-rack shared with another tenant. "It doesn't need it's own rack" PM insists.

No antivirus is bought. "You don't need AV" PM insists. When it was explained that AV was a certification requirement PMs toys come out of the pram and he stops attending meetings.

We ask how many Windows VL licences the single product key covers so we don't go over. "As many as you want, they're virtual machines" architect assure us.

After pointing out this isn't true for VMs on multiple physical hosts not running Windows more toys part ways with pram and we're labelled "difficult".

Never saw anything vaguely official from MS about the single licence key so I still suspect their entire "World-Class" business is running on a contractor's MDSN Windows licence.

Anon to protect the innocent still there.

[Will Godfrey]

Re: Not just the beancounters

I would have thought the innocent would have made a pretty quick exit ->

[Jou (Mxyzptlk)]

Re: Not just the beancounters

> We ask how many Windows VL licences the single product key covers so we don't go over. "As many as you want, they're virtual machines" architect assure us.

Missing a details there. MS-Server Datacenter covers as many as you want as virtual machine, as long as it is on the same machine. Or one 2-node cluster, if you have two Datacenter licenses.

But I suspect your PM ran one MS-Server Standard License key on 5+ hosts with 50+ virtual machines?

[Anonymous Coward]

Re: Not just the beancounters

This was dozens of physical servers running a Linux-based OS with many more Windows VMs running on top. There were no bare-metal Windows installs.

The closest the PM and architect could get was the hypervisor console, the environment would hold the public's data so an audit trail was needed. The architect was bypassing firewalls and proxies to download software to air-gapped VMs before that was restricted.

Actually we wanted to run on Hyper-V but we had to buy an expensive product from a company someone's son worked at. In Sales.