Money for nothing, it's the best
--> sipping a cool one at the shores of the rivers of cash flow
Welcome once again, dear reader, to Who, Me? – The Reg's weekly attempt to soften your re-entry into the harsh reality of the working week with tales of workplace mirth. This week, we once again meet "Bernard", who in times past found himself working for "a rural local authority in Middle England" – which sounds like he was …
That actually happened to me, I've mentioned this before so won't repeat here:
Tried to get in to everything at my current place, asking what was in the server rack, did we have a map, what was each bit's function but told to not focus on that as it's not a priority, here's other stuff to do, then a hard drive failed, data was lost, suddenly became a priority...if only someone had brought up "hey, what's the backup plan" and been told to not worry about it...mind you, least they didn't make me redundant, still busy with "other things" though to be allowed to get my teeth in to infrastructure that is now very much covered, honest...
Odd fun fact. While this was some years ago, I remember reading how the majority of AOL's income comes from people who forgot they never canceled the service and just keep paying the credit card bill every month.
I wish I had that kind of money to burn.
When it's under $10 or so, a lot of people don't notice for a few months because they're too busy dealing with everything else. Especially if the description is vague.
After a few months, they notice and then the cancellation games begin. Some of these places won't accept a cancellation without the account number, which of course you don't have.
It can be extremely difficult to remove a recurring payment authority.
Oh, that's easy to handle, and I think this should be done anyway....
Change all your credit card numbers annually. Just "Lose" your old card and tell the CC company the same. They'll go through the protocol of changing your number.
The services you actually want to keep and use, you'll notice when they start complaining about not getting their money - update those.
The services that you don't use anymore might complain, but you probably won't notice, and if you do, won't care.
Has the side benefit of rendering any old leaked CC data incapable of impacting you.
"so basically get told your new number automagically so the company can continue to bill you"
In that case you close an account completely and open a new one. If your credit is good, it's not a big problem. You can even change banks if you need to go that far. They're all about the same level of horrible on average.
privacy dot com
On-demand credit card numbers. On first use they tie to the vendor and cannot be used elsewhere - so go ahead and read off that credit card number to a stranger over the phone. Set & define limits on cards to prevent unwanted price increases. Single use cards which close shortly after use.
Free tier & a paid tier of service. Have fun.
I've got one card number that got buggered. Once a month they try again to see if the card has been re-opened. I get the denied notification and laugh.
I use prepaid cards for this.
Forces me to stay aware of where my money is going, because if I don't take the time to push money to that card, it Auto invalidates and so does the service.
If they do find a way to make the card go under, credit card company has to eat it.
I've never been comfortable with allowing billers to pull money.
When you're making a deal with the devil (commerce profiteers), it's best not to give them the benefit of the doubt.
They're there for one reason and one reason only. To do whatever they can to extract the most money from your account they can find a way to get away with.
The services they offer is merely one means to that end.
"I've never been comfortable with allowing billers to pull money."
It's very dangerous if you do. You may get billed a fortune for something like electricity if a neighbors service gets cross connected to yours or somebody taps in illegally. You would still have to contest the billing and go through the whole process of figuring out what happened, but if you pay manually, that money is still in your hands and not theirs.
With a push to delete gas and have all appliances in homes electric, there will be the need to do lots of service upgrades and re-wiring. If you live in a multi-tenant block of flats where the wiring my have been done rather creatively, the sparkies working on the upgrades might not get it right on the first go. It would suck to see your whole paycheck get siphoned out of your account one day and having it take several months to get sorted. That would mean that anything else that's auto-billed could be returned as well. If you are renting, you can likely make arrangements with the manager since they would have been the ones to order the work, but if you have a mortgage, the lender isn't going to care why you can't make a payment or two and will stack fees and penalties.
Well, at least in the US, about $50 to hit them up in small claims court if you catch it before it exceeds $500.
If the company fails to send a qualified representative you win by default.
If the amount they are inappropriately billing you exceeds 1000 it might be worth it to pay a grand or two to a lawyer to show up to court and make this go away.
If you're losing money anyway by doing nothing and continuing to pay the bill, you might as well hold on to that money until the amount becomes large enough to pay for a lawyer for an hour in court to present what you've already generated to the judge.
It's important to take things to court.
Laws have no relevance until redeemed in the courtroom.
If Nobody brings it to court the law is useless to protect.
Business executives and their lawyers know this and take advantage by it.
"Well, at least in the US, about $50 to hit them up in small claims court if you catch it before it exceeds $500."
The maximum is much higher in most places, but you have to make sure you have the sort of case the court will hear. Also keep in mind that while you will be representing yourself, the credit card company will be represented by an attorney. If you have to bring the suit near their headquarters, it's not a problem for some junior person in the legal department to appear. If you can get the case heard in a court near you, there's a chance they will no show up if the cost to have somebody show up is more than the contested amount.
Once you've won, you still have to get paid. Most small claims court don't award expenses to the prevailing party so you are out whatever it cost you to file, appear and collect. This is why I'm not likely to take somebody to court that doesn't pay me or refuses to make good on a bad check. I just won't work for them ever again. I'd lose more money by taking the day off to sit around in the court house waiting for my case to be called than I'd likely collect from the deadbeat.
I had a friend who had a problem with an intransigent ISP which kept charging despite his requests to cancel the service & stop it. After a few attempts, going to the bank and showing them a bill and a letter (snail mail) cancelling the service then posting it got the bank to stop the bills.
"Impressive he could find a branch, let alone one that was open! 10 till 3, 4 days a week... if at all!"
I think it's more impressive that a local branch would be able to help. They are just human powered ATM's these days and can often only go as far as providing you with the correct forms and helping you to make sure you've filled them out correctly, which still won't happen and 30 days later they might be returned to you with a request for more information, a police report you can't get and a notarized this and that you also can't get.
I once received invoices - in separate envelopes, on the same day - for a couple of years subscription to an ISP which I had left some time previously. Followed a few days later by reminders (one for each monthly invoice, again in separate envelopes), followed not long after by the same from a firm of debt collectors.
A phone call to the debt collectors elicited the response "we do as we're told, but we don't think they know what they're doing".
They only stopped after I provided them with the date the telephne line was ceased (which took some effort to extract from BT). Part of me wishes I'd just left it and let them take me to court, since I could prove I no longer lived there.
(Said ISP, after years of good service, changed hands several times and has since disappeared completely).
There was a vintage BOFH story sort of like that. The company with the support contract first set it up so that the phone menu system is virtually impossible to navigate, with options like 12, then if you get past that they ask for some account or support contract number which is not clearly labeled on piece of paper, and I think there was some third hurdle, but I forget what it is now. Point is, Simon ends up dating the woman behind the support company finding a kindred spirit.
"I remember reading how the majority of AOL's income comes from people who forgot they never canceled the service and just keep paying the credit card bill every month."
I imagine that happens even more these days with so many things on 'auto-pay'. Your pay does straight into your bank account and a whole bunch of bills get paid and people never look at what's going on after a while. This is why it might take two or three notices to some company that you want to cancel your service. You mark it down that you notified them and trusted that the billing would cease. They knew that a lot of the time people will just assume the billing will cease so if they just keep charging the account/CC, it might not get noticed. Even if you have the time to spend getting the company to refund the overage, provided you can prove when you cancelled, they've had use of the money for some time. Multiply that little scam a few thousand times and at any given point, the company has millions in 'float' funds.
I pay my bills manually and make sure I'm being mailed statements. Every once in a while my host will go down and emails will bounce. If a company gets a bounce notice they will sometimes strike the email address and I won't get bills that way. For all the abuse the post office gets, they do seem to deliver all of the bills promptly even if it feels like checks take ages to get. Part of that is the envelopes the bills are sent in exactly conform to the standards to be machine read and will often have routing bar codes.
Working for a very small ISP, I can say these things are still common today. When I took over running the place, I noticed there were a number of lines that never had any activity, and I thought I'd call up the customer and enquire whether they're satisfied with the service provided. But without a "Bernard" at the other end, these calls never went anywhere meaningful.
Got a cancellation last week for one that had been inactive at least 5 years. No request for refund. I'm surprised Bernard's provider agreed to refunding over 7 years worth.
In my locale, both privacy legislation and people's expectations of privacy makes it a bit of a grey area to look at statistics of when and how much a customer uses the service (unless you have specific limits and do it in order to enforce those limits), so that's why I worded it more vaguely than right out saying they're not using something they're paying for.
Really? If privacy legislation truly prevents you doing something like that, then I think it's gone badly wrong. My ISP and mobile provider both periodically email me about my usage. (My mobile provider tells me whether it thinks I'm on the right tariff or whether I should change to one that would be cheaper for me. Whereas I don't know what my ISP is trying to do - possibly just trying to shame me to use less.) And I can't see anyone will sue/complain to the regulator if you save them money.
But, if if the lawyers really are jumpy, you could write a script to retrieve those with zero usage so you don't see others' usage.
Or, better, you could write a script that retrieves a user's usage and emails it to them without a human ever seeing it - possibly filtered to those with zero usage. That means, in principle, no human ever knew their readings. (Maybe I'm being uncharitable to my ISP and that's what they're doing - reminding me I have a contract.)
"But folk find privacy and health & safety legislation (which they usually know very little about) convenient excuses for not doing their job properly."
Well, all of that legislation is so hopelessly confusing the way they wrote it.....
If it were much more common sense it would be easy to follow. If a person is on a building site where they are just starting with ground works, why would a hardhat be useful? Why does a surveyor's assistant holding a pole in the middle of a field need safety glasses and steel-toe boots?
"If a person is on a building site where they are just starting with ground works, why would a hardhat be useful? Why does a surveyor's assistant holding a pole in the middle of a field need safety glasses and steel-toe boots?"
Don't know where you are based, but in the UK and NL the health and safety legislation basically requires you to have a safe method of work and to wear the relevant PPE. So the legislation doesn't require you to wear a hard hat all the time. But you company safety procedures probably do require full PPE all the time - that makes it easier to work and avoids discussions about what is or isn't required.
Incidentally, if they are busy with the groundworks there is a risk of being struck by a digger bucket. And if you trip on that site or in the middle of a field and your head strikes a lump of concrete you might die (I've translated two accident reports about such accidents) - a hard hat with chinstrap (so it doesn't come off during the fall) might save your life.
In the middle of a field you might step on a nasty bit of rebar - those safety boots usually also have a steel plate in the sole to protect you against that. Apart from which modern safety boots are just as comfortable as walking boots (I often use mine as such, as they are actually more comfortable than my traditional walking boots) so there is no reason not to wear them.
Back in the days when I worked in high street retail we had a self service point with a phone so customers could contact head office customer service. It was a bare handset that simply dialed a head office line when lifted.
Some customers realised, however, that if they unplugged that handset they could plug their own phone into the line and dial any number they liked, and it all went on our bill.
Eventually a refit removed that facility, but nobody thought to turn off the phone line, and so certain customers would still come in and use the line.
During one of our managers lengthy absences I drew the short straw and so it was my turn to become "executive officer for the week"
Out of sheer boardom one day I decided to check the phone bills and one instantly jumped out, because only two phones in the building had unrestricted international dialling, and this wasn't one of those lines.
It turns out when the people using the line were phoning relatives abroad, and some where dialling premium rate chatlines, the monthly bill was in the high four digits for a line that should have been disconnected years previously.
After a panicked call to head office blame was shared equally between the head office bod who was supposed to disconnect the line, and the absent store manager who clearly never checked through the monthly phone bills.
This happened to a friend of mine in Australia. Someone discovered an unrestricted phone line that could make international calls. Loads of people took advantage of it to ring relatives in blighty (note to younger readers, there was a time when the cost of your call was relative to the distance. Long distance calls were eye watering expensive)
When the management found out they offered an amnesty and to pay for the call cost. They then called all the numbers and asked whoever picked up "Hi I'm calling from Sydney, Do you know anyone here" (Again for younger readers. It was more innocent time when Phishing involved a hook and a worm)
Anyone who caught and not taken opportunity of the amnesty were summarily fired
There was another situation when the university computers we hooked to a landline. The number went via a gray box, with the number encoded via a set of dip switches. It did not take long for people to realize this could be easily reprogrammed to any number
"note to younger readers, there was a time when the cost of your call was relative to the distance. Long distance calls were eye watering expensive"
Ah ah, so true. Heck, in France, it was already very expensive outside of your very town ! International dialing was a bit like going to the moon ...
Heck, in France, it was already very expensive outside of your very town
It was worse in the US. After the AT&T breakup allowed competition for long distance we had a situation where what regulations called "long distance" (I think it was all calls outside your area code) had competition which allowed for lower prices. But those "local long distance" calls which might be one town over or perhaps a few hundred miles away for large area codes in sparsely populated western states did not have competition, they remained the exclusive domain of the "Baby Bells" - the seven regional "local" phone companies that resulted from the breakup.
So there was a period where it was cheaper to make a long distance call 3000 miles from one coast to another than it was to make a call 30 miles away to the next city over, simply because there was competition to serve that longer call but a monopoly remained for the shorter one. In fact, since there was competition for international calls I don't know but I'll bet some calls to other countries (Canada and Mexico certainly, but perhaps places like the UK also) were cheaper than that 30 mile call.
Los Angeles was quite balkanized decades ago. A phone call a couple city blocks away could be "local long distance" at a good multiple of normal cost. There was always a hesitation to call a strange number, and then asking "where in LA are they?" And the locals never seemed as embarrassed about it as their out-of-town visitors.
"After the AT&T breakup allowed competition for long distance we had a situation where what regulations called "long distance" (I think it was all calls outside your area code)"
"Long distance" was for calls outside of your LATA or "Local Access and Transport Area". LATAs didn't (often? usually?) correspond with area codes.
Yes, intra-LATA calls may or may not be in the same area code. Such calls were billed by your local carrier, rather than the LD company you had chosen. One way around this was the use of "dial-around" numbers. This was a short dialing code that would route the call through your LD carrier. IIRC for AT&T it was 1-0-ATT-0 followed by the number to call.
When I was at school in the 90s, a call between there and home was intra-LATA. But, even in the days of cellular roaming and per-minute charges, calls within the provider's nominal service area did not incur any LD charges. This included forwarded numbers. So when I would return to school, I would forward the mobile calls to my landline (cell service was terrible anyway), my parents could call my local-to-them number, and no LD charges applied. And as a forwarded call, no airtime charges either.
Our favourite trick was to dial one of the 'magic'* test numbers to invoke a ring-back test on each payphone passed during whatever booze-fueled journey between drinking establishments was underway.
If they were quick enough, the stragglers could catch up...if they weren't, or passerby answered the ringing phone, then they had to do a bit of searching. Two jokes for the price of one.
A long while ago, I used to have a 'Mercury' account (Dial '131' + account number) - which you could use to put calls from payphones onto your bill.. I nearly got done over in the local pub nearest work....people saw me making calls from the pub coin-operated payphone without using any dosh.. and got pretty shirty when I refused to share the 'secret number'.
* Far fewer of them now, but back in the day there were multiple ways of triggering a remote test. We had inside knowledge of them all :-)
In 1965 I called Adelaide direct dial from Orange County, Taxifornia and the phone charges were $20 USD for three minutes but the taxes added were considerable (excise tax on excise tax because the temporary war tax was still in existence). The call went to a Central and a runner was dispatched
"The number went via a gray box, with the number encoded via a set of dip switches."
The early Mobile Telephone Service (MTS) in the use had the phones 'programmed' by using jumpers. A person I know that worked at a company that installed them fitted one of the phones in his car and tried settings until he got something that worked. When it stopped working, he fiddled about again until it did. He didn't have a way to let people call him, but he could have them call his pager and he'd use the phone to call them back. (I've really dated myself with this little ditty).
A 100-year-old company making specialist, rotating machinery for a wide variety of industries, including equipment for the armed forces, especially the Navy during WWI &II. As trainees, later in the same century, we used to get sent all over the factory to find old records and files etc. It was interesting in its own way and you developed techniques to retrieve information from long ago. The records were all paper or perhaps micro-fiche. One such repository was underneath an old building which I found out had been reinforced during WWII. The basement had been significantly extended, used as an air-raid shelter and the roof as an observation post. It was deep, dark, dusty and musty in places. In my wanderings in this dungeon, behind some old bookshelves, I found an old doorway into a small room which had clearly been out of use for decades. Curious, I moved the shelving...... and opened the door..... An old light-switch powered a single bulb to reveal a small desk and chair with a single telephone: vintage rotary type with twisted cord to the handset. You will have seen these in old wartime films.... On lifting the handset, it gave a dial-tone but I was too nervous and junior to try a call. I imagined it was a direct line to Biggin Hill....
I reported this information to our switchboard (three telephonists; it was a long time ago), then the bosses..... "It could be associated with the confidential/secret nature of some of our business." I was instructed to leave it alone. No-one else went to see it; "Urghh! It's dirty down there."
The building was demolished twenty years ago but it wouldn't surprise me if underneath the new housing estate, sometimes there is a ghostly ringing sound from a room that was built over.
The 80's saw a boom in the UK for people (mostly) in financial services taking a percentage cut of the money they made for their clients.
It was an interesting contrast to see those, such as engineers, who saved large sums of money for their company, surviving with a pat on the back.
But I'd still rather be an engineer.
I've probably told this one before, so I'll keep it short:
One time I was checking a maintenance contract and realised we were being billed for systems that no longer existed to the tune of £500K wasted a year. I informed my manager, our procurement manager and the vendors account manager, but nothing changed over the months.
At a briefing session from a big boss, he mentioned that they were looking to cut costs. I raised my hand and asked if he'd be interested in saving £500K. Which got me a glare from my manager, and definite interest from the big boss. Another manager [Let's call him "Jim", for that was not his name] was given the task of dealing with the contract updates, and Jim co-opted me for the details.
At the end of the year, Jim got employee of the year, the procurement manager had been promoted sideways, and I got a black mark on my file for causing trouble and neglecting my core duties.
And indeed Bob. Have you had the misfortune to see any recent episodes of Fireman Sam, Postman Pat, Bob the Builder and (possibly the worst of the lot), Thomas the Tank Engine? If those puppets ever developed sentience they'd be out on strike immediately. In the case of Thomas there are fan-produced episodes that have better stories and higher production values than the "official" ones.
Bring back Johnny Morris I say!
This approach has worked on the past.
The question is asked Is xxx still used?
No replies forthcoming…
Raise a change request to turn box off, include all correspondence trying to find out if it is use still.
Get change approved
Turn box off
Wait for screams of panic….. then turn it back on
As long as the change was approved no fallback
Worse are the machines you find under the comms room floor or inside a ceiling void that nobody remembers being put there.
I've been on the other end of this. When you're in an organisation with an at best opaque internal hardware department, and you've been using a service that is provisioned by someone else, for someone else, who has left the organisation, or moved to another department, and an email goes round saying "is anyone using machine ABC-XYZ-01?" you don't recognise that this is the machine running your vital DevOps deployment service, or whatever.
At least our place doesn't actually dispose of the hardware right away once it has been switched off, so we do get a chance to work out what was running once something disappears, and provision a new server running a version of Windows that hasn't been discontinued to replace it.
I haven't used paper labels or embosser-style Dymo labels for many years, as the glue always fails and the labels fall off. Instead, I use Esselte labels. These are thermally-blackened by the labelmaking machine, and use a glue which in my experience has not yet failed. (Caveat: if you try to remove a long-ago-applied Esselte label, it leaves gummy residue behind. Alcohol or acetone will help you clean that up.)
Brother makes similar labels+labelmaking machines, but I haven't tried them.
I've got a Brother one, which also uses thermally blackened tape. My wife uses it extensively to label various folders and storage boxes, and I use it to label my work laptop with the machine name, so I can remember the SQL Server instance name more easily.
I don't think we've had a label come off anything yet, unlike those lumpy Dymo ones which fall off the moment you turn your back on them.
"As long as the change was approved no fallback"
The problem as I've experienced it is that, if they don't understand it, they don't approve it. The change request can be written and sent for approval, but you'll hear nothing. You can bring it up manually to people who can approve it and they'll all say something like "I don't know what that is, and maybe it can be turned off, but wait for someone else to confirm that". Getting the change approved can be difficult. If it's hardware, I have an alternative. Accidentally disable the network port or unplug the network cable. It's an easy fix if people complain because you didn't even turn it off, but you can put it down to unknown failure of something old that wasn't monitored and use that to justify updating it.
Been there... domain controller decomms.
I got RO access to the firewall so I could read traffic logs. without bothering anyone else.
I was decomm'ing a bunch of DCs, came to one of the last ones, I could see traffic to it from a third party vendor.
"OI - you are still using DCxxx! Point your system at DCyyy please and thank-you!"....
days pass - still traffic going to the old DC.
Cue another email, this time I get a irate repsonse "we have, you silly man!!!!"...
ok, fine. Shutdown DCxxx - wait 5 minutes, sure enough Service Desk phone light up like a Xmas tree.....
"waaaaah, our system has stopped working".
So I power on DCxxx. Calls to the SD stop. Cue yet another email..... "are you sure?????".... repeat this about three times before they worked what was going on :(
I'm a windows guy, so I didn't fully understand the final root cause - but something about their system was "oracle something" and there's a couple of places you have configure DNS - one in the overall OS, and one in the virtual app running on the OS. Something like that. And they really only knew about the app layer.
" there's a couple of places you have configure DNS - one in the overall OS, and one in the virtual app running on the OS. Something like that. And they really only knew about the app layer."
This is one of those places where it could be good to have a big paper system map on the wall that shows routing, addresses, etc.
When I was working in aerospace I couldn't get approval to have a big print made of the systems of one of the rockets we had. I eventually had one made on my own dime and brought in it whereupon everybody was using it as a reference and penciling in updates as things changed. When they hired a new engineering manager, I finally got paid back and new prints were made with the updates added. It was so much easier to use than digital files and even better when working with several people to be able to point to things. When I left, I heard that nobody was assigned to keep it going and they had chased out the best engineering manager the company ever had by treating him like dirt. It was all back to digital files and eventually bankruptcy. I still have some of the files and as I recall this, I'm thinking I should get some prints made (smaller as I don't have the same amount of wall space).
20 plus years ago I worked for a small telco. The senior managers ensured that no-one lasted more than 18 months through ineptitude, constant reversals of decisions, corruption and nepotism.
It became a game to hide servers in the exchanges - the number of critical workloads on 1u boxes sitting under bundles of cables in roof voids and floor tiles was amazing. Morale was just so low in the technical and operations teams that no one gave a monkeys. Suffice to say they no longer exist…
Yeah it is tricky for stuff like that, which is another good reason why anything connected with fire service should be put in metal conduit. Then at least you know "this isn't a simple phone/network wire".
Of course that doesn't help for dealing with legacy stuff, but at least you can make it easier on the gen Z+1 ers who will be the ones looking at "legacy" stuff installed in 2023 decades from now. Yeah yeah, it should be labeled too, but few people use the kind of labels that stay affixed and stay legible decades on.
"I've begun posting placards in my server rooms that 'any new installs not labelled with name and function will be disconnected without warning.'"
It also doesn't hurt to have a notebook that lives in each rack with the details of that rack and a running service ledger. It's one of those things I picked up in aerospace. Being able to review service items easily can be very good for troubleshooting. If a problem keeps having to be fixed, it might point to a non-obvious cause.
Similar I am also of the turn it off and see who screams.
After a 24 month project of decommissioning a data center, and trying to identify a bit over 1000 servers or VMs, there were about 20 that no owner could be located.
So we waited until almost everything else was either moved or decommissioned.
So it fell to one of the PMs to head into the DC and shut down the hardware.
So of the approximately 20 servers with no known owner, we only got 3 screams. When asked why they had not responded over the last 2 years, it was usually i did not know the server or application name.
I just click on an icon....
Turns out the servers in question had long since been abandoned, with out updating the CMDB, which in itself was a "shock"...
You were shocked the CMDB was not accurate and updated correctly? You must have been new at this at the time.
One of the things I've done in past consulting gigs (as a side project when I notice the processes are totally broken, never as the main focus) that gets the most upper management attention was redesigning processes so that it was impossible to commission or decommission servers, networking equipment, etc. without the CMDB being accurately updated. The front line people and business owners scream bloody murder at you because it makes their lives more difficult and/or slows down implementation schedules, but being able to have a 'final' audit of the CMDB and knowing it will be correct from then on pleases the upper management folks (i.e. the ones signing my checks) to no end.
If people would do their damn job I wouldn't have to make it such a painful process, but there are always people who like to take shortcuts and think they will "remember to do it later".
Many years ago, I was on call for ICL, and one of the Camelot severs we looked after went down unexpectedly, about 8pm. I got an automated monitoring page.
I could not access the machine remotely, and couldn't get anyone on site, I had to escalate it (because, you know, national lottery and all that) and after an hour of nothing turning up, i was being told by my superior that whilst we were remote support only, in this casr, i'd have to drive to some place I'd never been to, to see what was wrong. Whilst she was trying to work out how I'd get physical access to the site, she finally got hold of someone on site... "Oh, that server? We no longer use it, so I switched it off an hour or so ago"
"If people would do their damn job I wouldn't have to make it such a painful process, but there are always people who like to take shortcuts and think they will "remember to do it later"."
Still during my early days at CFM/ICL a call came in of a server rebooting (fortunately for me - but not screaming users - during office hours, not when I was on call!)
Had a look at the logs, it was a clean scheduled reboot. Checking further, no root user was logged in at the time. Checked cron. There was a yearly scheduled reboot in cron.. WTF?
I asked around my colleagues, I asked our customer contact, no-one had any knowledge of this.. I end up removing the cronjob, and the call was closed, without finding out why...
Fast forward a few months, a server needed a scheduled reboot one afternoon. I can't remember why, but I think they decided to do it in the daytime so people would be around if there were any problems.
Anyway, the documentation for this client was dusted off. This was a procedure guide written by an ex-staffer here who used to mainly deal with this client, before he left the job.
"Check with admin".... check...
"Notify users".... check....
Then: "As it's a scheduled reboot, instead of relying on someone here rebooting it on time, set a cronjob for the specific time. Don't use "*" for the month/day/date parameters, because if you forget to remove it, the machine will reboot again the next day! Instead, set the month and date also - that way you won't have problems if it takes you a day or two to remove it"
You can guess the rest.
In the coming years of doing that job, the same thing happened to other servers on at least a further 2 occasions.
I suggested if we had to use cron, to intentionally set it to a short period, and don't bloody forget.. And if you do forget, you'll be reminded by an unscheduled reboot you can understand, rather than dumping it on some clueless sap a year later!
Referred to in many organisations as the Scream test. Desktop phones and network points should elicit a loud shrill noise in the immediate vicinity, ones connected to servers emit a choral arrangement.
After spending many years emailing, polls, even the old memos, the scream test was the most effective in finding out what was in use or wasn't.
Also a little book/spreadsheet for things that ABSOLUTELY shouldn't be disconnected proved invaluable. You can't trust people, not even yourself
Building Services tried that in a couple of remote buildings that had large comms cabinets that didn't appear to be showing activity... unfortunately they then discovered that they couldn't exit the building as the electronic door locks had stopped functioning!
Only last year I spoke to a guy who was trying to scrounge any leftover 3.5" disks for a piece of kit (X25 pad) he had been asked to look at and he mentioned it was in a in a cabinet that sat in a roped off corner, covered in 'DON'T EVEN THINK ABOUT TOUCHING THIS' stickers
Another way is to expire all the password from the group of unhelpful people - and see which userids get their password reset.
Someone else looked at the cost of some servers, and decided to charge back to the managers. The manager getting a bill for "£50,000 for support of unused server" quickly took action. For those that didn't take action - a month later the manager's manager got a copy of the bill ( £500,000) for support of unused servers.
I found quite a few like this working for local authorities.
The funniest one was a Megastream (2Mb/s pipe for those who don't remember them) going all the way to the county council office, costing back in 2000 about £8k/year. When I looked at it, I recognised the name of the office as one that had been demolished ten years ago...
Just out of interest, was that on the ACE or RENACE side of things? You might know me if you were involved.... I was on the sidelines during the VAX-->ALPHA changeover when it was an 'A&TN' job, and did some of the testing on the later NTEs. Remember the room at NCP with the ACE models? That's where I lived also :-) 'Twas much easier when things were truly Circuit Switched..
Friend of mine has a shop in an ex-library, which had the rack with the comms links to the local schools in it.
He asked if he could remove it, because of the power it was using. Council replied if he took it out he'd disconnect lots of vital services.
I'd already checked it, and found none of the kilostream+megasteam units were actually connected to anything.
So he reply to the council was "really? Because we checked that before we turned it off 2 weeks ago, didn't you notice all those vital services being disconnected?"
It's when you see it billed as:
Former XXXXX Health Centre
Car Park XXXXX Health Centre
The site in question, the original building was knocked down and rebuilt a few metres away.
The site of the of building became the car park...
Investigating lines at a now defunct NHS organisation many years ago, they were being billed for a number used by a Fish & Chip Shop.
I was working as a temp worker in the accounts payable, for a UK energy provider, at the time "Dual Fuel" was the next best thing.
I as given a pile of invoices from the mobile phone provider and was tasked with "sorting them out", about 200 different numbers.
About 10% were for phones that people did not use / had never used - lines cancelled
Another 10% took about 2 months to find the "owners", after leaving multiple voice mails that their phones were going to be cut off.
Some of them were very indignant when their phone stopped working.
"It was a frequently used emergency line", which had actually not made or received a call for over a year.
This was passed up the manglement chain and the number was usually cancelled.
One line was for a provider of frozen foods...that had nothing to do with providing gas or electric to the good (and bad) people of the UK.
I rang the number and spoke to the "owner" - a sales person, they were as confused as I was.
Both organisations were paying for the same number.
My brother worked in Births, Deaths and Marriages. They used to generate various regular reports eg number of people born in a each district by age, for various other departments and government ministers. He was given the job of seeing which reports were still needed. The conversations invariably went something line:
[recipient] No - we just throw them in the bin
[recipient] No - please stop sending them to us. We keep asking you to stop sending these reports but you still keep sending them! PLEASE STOP SENDING THEM!
[brother] err, OK - who did you ask to stop the reports?
[recipient] ***name redacted***
[brother] Oh - he hasn't worked here for years...
"[brother] Oh - he hasn't worked here for years..."
Had a customer on a supposedly fixed IP address. the ISP, Virgin Media, had done a network re-jig in the area and it turns out that, at the time at least, the "fixed IP address" was just a very long lease on DHCP so half their connections to various servers at HQ were failing because the "known IP address" was part of the security system. Once I'd realised what might be the problem, we contacted VM who said they'd been sending emails to all affected customers for months beforehand warning of the changes. They even told us who the contact was. I reported that to the customers and, of course, it was a named person address at the company for someone who had left years ago. So both client and ISP to blame there.
This was not a question of broke, this was a question of administrative bungling.
It's always good to take a sharp knife to administrative issues that nobody has a hand on. Sends the cockroaches scurrying when their comfy nest is disturbed.
Now the telco, on the other hand, might have been the subject of a review from the fire department, i.e. a stern reevaluation of its usefulness and an evaluation of possible replacement candidates (which are likely not numerous).
Then again, it's a fire department. Not like they can afford to have phone issues for a week (or ten).
I worked for a company that had a job running every morning at 1am to execute an SQL statement, store the results in CSV and then copy the CSV to a director's PC (he had a dedicated share on his machine that the job would write to). The system from which the data was being extracted was replaced with a new and improved version and consequently, the job no longer delivered new data.
Director did not notice that the data stopped updating after a date in late 2009 but he was happy because he got his daily CSV.
One day, in a clean up exercise, all unused databases were shutdown and this stopped the CSV from being generated. This did raise alarm bells with said director because now he wasn't getting his daily progress update. This was raised as a system down issue and had to be fixed immediately.
Trying to tell director that the underlying system is no longer used and hadn't been used for 3 months lead to complaints to HR that we were not handling his system down issue with the required priority.
Eventually, the last CSV was copied back from his share and the job changed to copy that CSV to his share every day.
This worked wonderfully until he got a new PC and raised a new system down issue that his new PC was not receiving the file ... I was working my notice period so I don't know what they did to fix it.
My suggestion of sending him a CSV filled with "Source","application","was","shutdown","in","2009","and","we","cannot","magic","data" may have been implemented ...
I worked for over eight years at a Managed Hosting provider in Downtown Atlanta and one of the perks they offered employees was a free hosted server. They were nothing fancy, whatever the cheapest 1U model that they had in inventory at the time the employee requested it.
About three years after I started there, they had me go through the Data Center to do audits of all the equipment racked on each of the rows. I found more than a few ex-employee servers that had been left running for a sizable number of years after the ex-employee left or got fired. The fun part was, I was tasked with doing disconnects on all the zombie servers I found, and more than a few complaints came in from those ex-employees. Once our support team was informed that these were ex-employee servers, the rest of the complaints got sent to the billing department to discuss terms to set up payment to keep the hosted servers online. They should have back billed for the time they were left powered on and connected, but they didn't.
IMHO, telco billing is the 3rd ring of hell.
I would wager that they have categories in yearly reporting of services they know are supposed to be cancelled, have cancel orders pending that never clear, or are outright abandoned that are a huge line item on their list. But of course gets covered up because they can't show such blatent scamming of customers.
Back when I had to do that, it was cancel service. Check the next n months; complain to telco monthly that they haven't disconnected the circuit yet. Once it is finally off the billing 8-14 months later, then you start the process to grab back the money paid out (unless AP is on the ball and is discounting it's payment and dealing with the dunning notices because the telco is going to be putting you to the debt collectors for money it shouldn't even be collecting).
This wasn't an isolated incident, nor any one company.
It was _every_ single time. Every telco company known.
I'd estimate that at least 30% of telco revenues come from disconnect services that just keep auto billing and being auto paid.
...knowledge of each box's function fades into legend...
I still get a phone call once in a while about what some things do, or should do, I always answer and help my old teammates out in a pinch. I don't think any call has lasted longer than 10-15 minutes. Happy to be the archive, I did pass on my previous managers hand written notes for AD to the poor soul that would pick have that dumped on his plate...
OK, so while working for a telecoms company in the UK, I was on a contract to provide telecoms lines to various 'institutions' (client confidentiality and a strong personal desire not to annoy any of the people at these 'institutions' forbids me from referencing them in any detail). Well, we installed said communications line at a site. And charged the client for it. Only, well, after a while no-one could find the customer end. The customer site people didn't know were it was, and the engineers didn't remember were they'd 'put' (OK, installed) it. (I had nothing to do with installations, being a mere IT security consultant person.)
For all I know it is still there.
I wonder if that might be $County_Council somewhere in the midlands? I was on site for some other totally unrelated reason and overheard a discussion between what seemed like senior managers and techs and a BT rep or engineer. It seems they had a live and direct phone line, not routed through any switchboard, for which they had identified the number, but neither current nor previous phone supplier could identify it or confirm if it was their line and if it was being billed for. Apparently, no amount of digging through contracts or bills could find any reference to his number at $County_Hall.
Eons ago, I supported a shop floor automation system at Boeing. It would automatically check in and update itself over our internal network. But since it had been born as a dial-up system, we maintained a modem and phone line at the server. Which came in handy on the occasion that our (flakey) network went down.
But we had some accounting types in IT that assesed charges in internal "funny money" (just inter-departmental budget tracking) to account for things like heat and light. Every few months the nework would go down. And our backup process of "just plug in the phone line and hit Retry" would fail. Because accounting figured that they could save some of that fake money by disconnecting phone lines that showed no activity. It was an internal PBX system, so no actual funds went off to a telco somewhere.
Strangely, accounting never figured out that our primary profit center involved building airplanes. And downtime on production equipment (and the resulting paying people to stand around and wait ffor lights to resume blinking) cost real money. My solution was to call the modem once a month from my desk and make sure the modem whistled back.
make sure the modem whistled back.
"You know how to whistle don't you? You just put your lips together and blow."*
Mine's a black raincoat with a Borsalino Fedora** in the pocket.
*To Have and Have Not, Lauren Bacall's character: https://m.imdb.com/title/tt0037382/trivia/
(Not a Paris Hilton icon as she's not a patch on Ms Bacall (IMHO).)
Now I really am jealous of you. Congratulations.
Ms Bacall is also excellent in the Albert Finney 'Murder on the Orient Express' film, (awesome cast: Ingrid Bergman, Richard Widmark, Colin Blakely, the truly vivacious Vanessa Redgrave, Wendy Hiller, Sean Connery and the wonderful Richard Rodney Bennett score).
I've had the opposite problem. Had to go to some effort to get a private connection run into a remote disaster recovery site. It was worth paying for the connection to be maintained because in the event of a disaster happening, it was going to be needed immediately.
It was the devil's own job stopping the hawk eyed bean counting bastard networks colleagues from stumbling across this never used link and cancelling the contract, irrespective of how many "hands off important!" stickers wrre on it. Every DR exercise would reveal that, yet again, an ill educated jobsworth had thought they knew better and sought to save some pennies.
In these cases, I have found that the occasional "Dry Run" ( say once a month, about the end of the month when accounts are trying to generate their important reports for the C suite) can be invaluable in pointing out that these lines should be maintained regardless of the cost. Illustrates to Accounts that they are important and regardless of the cost, necessary, if only to keep the accounts department running in case of a disaster.
I had that with some X25 circuits into a regional office. Stuff just got buried on the monthly BT OneBill CD which kept just getting paid every month via Direct Debit. No-one understood it from a Financial Control POV. It was just a (largely) fixed cost of doing business.
A review of it - using the analysis software BT provided with it - shaved £150k annually off the telecoms bill. Unused circuits, unused lines, call analysis and some arm twisting on call behaviours.
think of all the random unused services running in AWS, Azure, GPC or any other random cloud platform. Unnoticed VM's running services that aren't used anymore spunking cash with gay abandon. All too easy to lose stuff down the back of the sofa in the cloud, stuff that would be easier to spot and wouldn't cost as much cash if they were unnoticed running onprem
About 6 months ago, I needed some cash suddenly (it was more than my daily ATM limit), so I looked up online and found an AMEX cash station about 2 blocks from my house, and it was at a bank that has been around for t least 30 years. I walked down there and went inside the bank and looked around for several minutes, I approached the security guard and asked him to point out the machine and he looked puzzled and suggested I the bank people. I approached one, and she told me to sit down, and someone would be with me. After a few more minutes, I again explained that AMEX claimed they had an ATM here. He told me that he had worked there for 20+ years and had never seen one. I call AMEX, and they assured me that there was one there. I gave up and walked home and called AMEX to see if there were any other real stations around. I went to 4 out 5 five, and not one of them existed. I did find one, but it had a 100.00 limit.I went back home upset and was able to track down the person for the cash stations in the Chicago area, and I started to ask if any really existed. This was the first time in my life that AMEX fell down on the butts. One thing I did learn from this was not to trust AMEX.