
The interneet
This story reminds me so much of the IT Crowd and 'The 'Internet' episode.
Thanks for the chortle.
Friday is here, and perhaps your temper is a little frayed. Which is why The Register always opens the last day of the working week with a fresh installment of On Call, our reader-contributed tales of exciting incidents on the front lines of tech support. This week's tale comes from a reader we'll Regomize as "Bruce" because …
Tech on the phone with CFO
We need a firewall.
Isn’t that expensive?
Yes
Then no. We don’t need a firewall.
Can you open a command prompt an type ipconfig
Ok
What is the ip address?
18.160.181.34
One ping bomb later,
Uh, I think we need a firewall
Yes sir, we will get it ordered and installed immediately.
Unfortunately this is an actual conversation and those old enough will be able to date it by the ping bomb reference.
We had an issue with the f$^#^ckers.
We spray painted some wifi routers with dazzle flouro paint. Red, Yellow, Green, Pink (for upper manglement area) on some pretext, but really so it was super obvious to everyone which router serviced which part of the company.
After a month a big printout of top sites and usage by router color went up on the lunchroom noticeboard.
First thing they did was try to move off the pink router....
But after a while their use of job hunting and porn sites came down to more acceptable levels.
Quite a few years ago, I used to be the IT Manager for my company. One day I got a call from the CEO saying that all of his e-mails had disappeared from the server, and I needed to get it sorted out PDQ. This was a completely unexpected problem and I had to investigate the root cause of the problem before I could fix it.
The root cause turned out to be easy.
The CEO also owned a flat in Majorca and had gone out for a week or so of R&R; however he had taken a brand-new laptop with him so that he could stay in touch electronically. While he was out there he had set up the email client on the laptop, but had decided selected POP3 as the protocol instead of the (mandated by IT) IMAP (we did not use Microsoft's e-mail server having had a bad experience with it). not surprisingly, the POP3 client then sucked everything down from the server, deleting the server-side copies as it went. When the CEO came back from holiday, he left the laptop in Majorca so that he did not have to carry it back and forth.
Once I had realised what had happened I (a) restored the CEO's mailbox from a backup taken before he went off on holiday, noting that anything that had come in afterwards was stuck on his Majorca laptop; and (b) told the CEO what had happened, whose fault it was (his), and gave him strict instructions on how to not repeat this screw-up. Thankfully he listened!
Once had a call where all the email for everybody had mysteriously vanished. Turns out that whoever had setup their AV hadn't put exclusions in for Exchange server. Sophos had decided that the Exchange server database matched a signature for a really old virus and deleted it.
To add insult to injury, they also hadn't been testing their backups. The DB level restore failed. Fortunately, they had also been doing brick level backups which worked, but were S-L-O-W. Had to do a dialtone fix to get them back up and running with an empty database, then restore mailboxes from the brick level backup. Prioritized the most important mailboxes. Took about a week to get everyone back.
Okay see, the real fail here is not that POP3 was able to be selected at all (sometimes you really do need offline copies of emails), but the fact that the POP3 protocol itself defaults to "immediately nuke the server copy" rather than the fail-safer "keep remote server copy until I manually delete it from the remote server".
This is the first setting I always change when setting up a local email client.
As a charity we used to have board and trustee meetings where it became apparent people weren't paying attention (in the course of one meeting I received an email stating we had policy which contradicted what IT had recommended and I knew hands on leadership were pushing for and as well hadn't been voted at that point.) Very awkward for us getting the job done and telling recalcitrant volunteer workers to comply. From that point on I got the GDs permission to have a wifi outage during meetings. Attention increased, and under table chat was curtailed.
I understand why you sat in the back row. People would look askance at you when your head and/or body jerked left or right to avoid the rocket headed your way. I must also commend you and your compatriots' abilities to suppress the usual FPS smack-talk.
Shouting out, "Ride my flaming-hot rocket, you shit-eating monkey-fucker!!" while the CEO is in the middle of saying something is definitely a career-limiting move.
Not just IT. If you want to get a problem solved there are two options:
1) Persuade manglement the solution you need is their idea, or
2) Make the problem theirs.
Either way, they’ll end up pushing for the solution to be implemented and, unless you manage to really screw up, you’ll earn Brownie points.
I think you meant NZ, but they made a very good film about those sheep going baaad!
I once encountered a tie-wearer using a calculator add up a column of figures in an OpenOffice.org (yes, it was that long ago .....) spreadsheet. And not even a proper scientific one; just a four-function idiot-calculator, with oversized keys and probably drool-proof coating.
I had to leave the room for a moment for both our sakes. But then I had an evil idea. When I returned, I gently moved away the calculator and told him OpenOffice.org Calc had some extra features over and above what was available in the Microsoft Excel he probably was used to, and suggested he prepare to have his mind blown. Then I deleted his manual entry (which I had already clocked as incorrect, because he had entered one field incorrectly into the calculator), and shew him how to create a formula to add up a group of cells. And the computer put in the right answer.
I just hope when he started his next job, he insisted to use OpenOffice.org with its built-in formulas instead of Microsoft Office .....
All the agencies I've worked through send me timesheets where you have to manually add everything up. I create a template from them with the correct formulas in the relevant boxes, down to Monday=manualentry, Tuesday=Monday+1, Wednesday=Tuesday+1, etc., hoursworked=end-start, etc.
Project Manager finally grumbling after spending half the morning editing all the alpha numeric prefix out of the serial numbers I had collected as requested via a barcode scanner into a Excel spreadsheet.
Wish you hadn't done it like this?
Whut?
Wish you hadn't done it like this I'm about two thirds done changing all the serial numbers.
Why dont you use search & replace?
Whut?
Eh, sounds just like my mom.
As much as she got me into computers, her go-to program (we don't use the word "app" around here) back in the 80s was Lotus 1-2-3. For letter writing (her excuse was that the cells made it really easy to position text when it comes to writing formal letters and such. This was when 95% of printers were dot-matrix and uses a fixed monospaced font, and only exactly 80 characters could fit on a single line on a sheet of A4 paper, so you learn to keep track of things like that), for managing her budget... pretty much everything.
She switched to Excel in the late 90s, but the excuse stays the same. She doesn't use Word, only Excel. It's the only thing that mattered to her where productivity is concerned.
Now I managed to switch her to LibreOffice Calc. So far she seems to be fine with it, but she only uses LibreOffice Calc.
...how a company like that survives
What's there not to understand about a graph that says "usage trend, maximum capacity, expected ISDN delivery time"
That just seems like the most basic business planning task ever.
When this bank is hemorrhaging money, do they tell finance to come back when they run out?
This was my thought too.
The OP wrote The CIO forwarded the request to the bank's Executive Committee, which promptly rejected it because it was clearly wasteful to buy an extra link when the current one wasn't even half-used!
But those highly (over) paid executives are employed to do "strategic thinking"."Thinking strategically" was even part of the annual review of managers when I was in education, so I am sure it must be in commercial spheres. If not what are they for?.
Nothing. People love to pretend that people who haven't had stuff explained to them, and which they can't be expected to understand without explanation, are the ones at fault.
Obviously this is actually a story about someone who couldn't present a business requirement clearly enough to be understood, but that doesn't fit the alt-right slant the reg likes to lean at these days
UK public sector survivor here
Most (like about 90%) of the senior post-holders I worked to could actually distinguish between an average load and a maximum. And they could appreciate the need to plan capacity for the maximum.
That was in the context of students, classrooms, chairs and desks, corridor widths, canteen seating and number of staff though. Perhaps easier to visualise.
https://www.jobtestprep.co.uk/free-psychometric-test
Nice tests! They actually make me think compared to those other typical online tests which are "binge-purge syndrome". But it is late in the evening and my native tongue is German, so I'll postphone after hitting the wrong button on math question three although I had the correct value in the calculator right in front of me :D.
If you liked those then I'd recommend the British Army Recruitment Battery (BARB) tests, especially the symbol rotation test.
There used to be a free sample test which could be accessed without any form of registration. I had good mileage out of that one used on an interactive whiteboard or tablets and with half a dozen teenagers trying to work out the questions. Alas, all the samples seem to be registration only now which is a loss.
Icon: Calculator? [raises eyebrow]
Obviously this is actually a story about someone who couldn't present a business requirement clearly enough to be understood
That can only be written by someone who hasn't met the type of person who starts with "what's the price ?" and responds "I'm not spending that on anything" regardless of the business case. I've personally witnessed that - a myopic business person who knew the price of everything but hadn't a clue (didn't want to have a clue) about it's value to the business. To add an IT angle, he wasn't prepared to pay us to do his network cabling properly and bodged his own complete with badly fitted plugs, regular Cat5e running outside (and even buried), and consumer grade switches hidden wherever he felt was easiest for him - and of course it was then our problem when his EPOS system (which was a pile of rubbish anyway as one of the "I don't pay that for anything" things was an EPOS upgrade to something that would work in his business) kept falling over due to network failures. We had the satisfaction of charging him more for fixing the faults than it would have cost to do the job in the first place - and he eventually stumped up for an EPOS upgrade (again, costing more than if he'd done the right thing in the first place) when he finally realised what it was costing his business not to. His approach to property maintenance, H&S, pretty much anything was the same. It was a business that needed a licence to operate, and eventually licence renewal was refused as long as he had any involvement in the business.
Then with another work hat on, I recall us constantly being told by the manglement to be proactive - to look ahead and (effectively) do stuff before it becomes a problem. But needless to say, any time we went along and (like this story) said "it would be a really good idea to upgrade the network because ...", we just got told "let me know what that's happened and the network can't cope".
TL;DR version - some people are 110% immune from the most expertly crafted business case if it doesn't align with what they thing they should spend (or not spend) their money on.
Decades ago my late dad was was running a factory on behalf of bosses who had that attitude. He spent more of his time keeping old (sewing) machines working than he did managing. And some stuff needed an outside technician brought in.The cost of keeping those old machines running far exceeded the cost of a systemic, scheduled upgrade programme. And that's before counting the cost of delays, lost business and contract penalties.
Which is why two factories went down to one - the least efficient of the two, but nearer to where the bosses lived and, and then that one in turn went bust- dad found himself out of a job. He'd been loyal to the bosses, because they'd promoted him and stayed with them even though he'd long known which way the wind was blowing. He should have got out earlier.
Some bosses just seem to begrudge the cost of doing business. If it's the choice between a new machine for the factory or a new Jag it's the car every time.
Once upon a time I was a young repair tech at a now defunct retail chain. I worked at a "special" store that was not customer facing and was located more or less literally right in the middle of a distribution warehouse. The whole building, with it's metal frame, was like a giant faraday cage. Soooo... naturally the company decided that they'd just give us a cellular hotspot device for an internet connection. This was right around the time Apple started making it possible to restore the OS from the Internet so they could stop bundling DVDs and flash drives with the OS image on them with new computers. The hotspot would generally fall over if you tried to do much more than view a web page, and sometimes even that was enough, so the idea of downloading a 4-5GB OS image was completely out of the question. Other techs in the same store would need to be able to do things like download Windows updates for customer systems, or download updated drivers, etc.
My immediate supervisor and his manager, to their credit, both saw the issue and kept trying to push for something more reliable, but like our story's hero, were constantly rebuffed. By the time I left, it never did happen, and I heard they shut down that store shortly after I left, making everyone go work in a customer facing store. So, I felt the hero's pain as it echoed down through the ages.
Bruce wasn't by any chance "big bearded Bruce"?
Beards, after all, aren't that rare with IT folks. Bruce being Australian would also fit the bill
https://www.youtube.com/watch?v=BpfGlO6ScJ4
... unfortunately, the official YouTube channel of the band doesn't feature that very song. Please give The Beards some love, bearded love to be concise.
-> I'll see myself out for the weekend
Late 1990's. OldCo was acquired. The very authoritative NewCo decided to backhaul all the Internet traffic across country, rather than implement a split tunnel or something like that. And it sucked.
Silly me runs PING with a -t and -l switch. The -l sets the size, and I set it to the max 64k. And I let it reach out to the gateway IP address which was across the country. Ran a few extra CMD windows...and then did it on a few more machines.
Evidently IT was paying for a variable speed communications line, and when traffic increased so did the bandwidth. Talking to peers across the office they all noticed the Internet was magically getting better. I was generating enough traffic to trigger the next tier where there was a big step up in capacity.
My mistake came when I left the CMD windows running overnight. Previously it just appeared as a spike in daily traffic during business hours. When IT saw it go 24x7, they knew something was up. Traced back to the network port & Win98 PC at my desk.
The next morning I got hauled in front of HR. What were you doing, job threatened, that type of thing. I remember telling the HR droid... "If the PING command is so dangerous, why does IT leave it on our computers?"
They didn't have a good response, and I got a 'Well, don't do it again'. IT did increase the baseline bandwidth though.
Dr Al Bartlett does a good job explaining the exponential function. He puts 100% at midnight and then works back so 50% is at 11:59 and 25% is at 11:58, and so forth. If it takes a month to provision more bandwidth, at what time do you have to start worrying if your needs are growing quickly? It's a much shorter time frame than a suit might guess.
We had this site that kept annoying us yearly and before the end of financial year. The site would ask us to provide a quote to put WiFi in the building. We would come up with a plan and provide a BoM. The most senior person in that building would always reject it. Generating plans and BoM is time consuming and after 4 consecutive years of getting knocked back, we decided to go straight for the jugular.
We told the senior person that we are going to put a WiFi outside her office (and ONLY her office) because we want to do some "testing". Within a few days, WiFi users started appearing in our stats. Week after week, the number of WiFi users increased. After 8 months, we announced that we were disconnecting the WAP because testing was over.
One early morning, we turned off the radios (but the WAP remained) and within 20 minutes we got an angry call from her demanding the WAP to be returned and made functional again.
We refused and told the senior person the WAP is earmarked for a different building. "I paid for that," she said hotly. And when she said that, we sprung our trap.
Our director calmly reminded her that "'someone' in your building kept rejecting the BoM for the last 4 years. That WAP is not yours and definitely not yours to 'keep'. Hand it back." and sent her a well prepared email, with attachments of previous rejection emails with her signature block along with a new quote.
The BoM got approved before lunchtime.
yes, used terms to confuse a manager regarding an issue I'd forgotten to fix. She always thought I was so good and helpful. I never abuse this, I only did it because this manager was known to be a cunt. When I told her staff after see left, they all found it funny as they didn't like her either.
I was asked to set up a care in the home business in the early 1990's. I relished the challenge, particularly being involved and caring for more mature members of our society.
The IT bit was down to me, we had Tandy and a small outfit called "Viglen". I chose Viglen fo the PC choice and natrally a dot matrix printer, I loved the sound and all the paper spewing out of it.
We banked with Lloyds Bank in the UK, the business manager suggested "internet banking" - yes you heard "internet banking"!
Well as younger upcoming man who could resist?
I discussed and sold the idea to my late bosses, the answer was yes.
A week or so later a small parcel arrived from the bank, containing 14 floppy disks (not sure of the size now). Maybe 4" could be wrong, size didn't matter in them days.
With a dial up modem and PC loaded with banking software that took an hour to load, the business opened!
My lovely staff were to be paid straight into their bank account, a novel idea, you had to do the wage run before 4.30
Before 4.30 PM on the Wednesday before the Friday payment to the lovely staff.
So the first BACS (payment transfer) payment went out, at 56K speed to the bank. It totalled some £4 K or thereabout.
Everybody happy I thought.
How wrong I was, on the Friday the 'phones got hot, not with clients but with staff "where is my money?" being the question. By the end of the day, 30 cheques (checks) written and handed to the staff I went home.
The bank never solved the problem, we went back to the old system, pen and paper.
They refunded the £4K
"The bank never solved the problem, we went back to the old system, pen and paper."
When new tech is hot off the press, always think "that sounds great, but does it work?" Another good thought is "is this really necessary?" You don't want to be in the beta test group. If the service works, the vendor should have a few customers that can provide references.
If you have studied your XKCD properly, you will remember a chart that calculates what the breakeven is for something you set up to save time to see if it's worth the effort. The same thought process is valid for many things. If you buy this new shiny, will customers pay you anymore for your finished product? Will is save enough time to see a return in a reasonably short period of time? How many people will die if it F's up? (metaphorically). So, how much more time did it take to cut physical checks with the proper deductions after you switched systems? How much longer would it take a couple of months down the road when you've forgotten how and have to re-learn the process? When I switched to a PR service and people were getting paid via direct deposit, it was as common as muck to do it that way. I don't recall having any issues, but by then the company would have seen most of the most common faults and know how to patch it all up since they were on the hook to pay penalties if the tax authority wasn't sent the withholding on time.
Yes, that's what Borkzilla & Co have taught us after three decades of borking first releases. But back in the day of the 56K modem, we were all young and starry-eyed when faced with another new shiny, because new and shiny was all around and we all thought, surely the company wouldn't advertise a product that didn't work ?
Now we know that yes, a company can very well advertise a non-working product because "there will be a patch soon". Patch which may solve the problem, and might even do so without creating other issues - maybe (but don't bet on it).
So yeah, now we know to wait a few weeks (or even months) for the initial bug set to be resolved before rushing to purchase a new shiny. Because we've learned the hard way (well, for those who have learned, at least).
This probably also applies to all those multi-squillion quid admin systems that go horribly, expensively wrong ( like Birmingham recently).
How much additional savings will there be, if any, by including any given function into it rather than just keep doing this as a separate process? And if the answer is little or none- just don't include it.
And of course when someone later wants to add in an extra section, because it sounds like a good idea with resulting significant extra costs - the same question applies.
There is a cost to adding complication and I have nasty suspicion that it's an exponential one.
"When new tech is hot off the press, always think "that sounds great, but does it work?" Another good thought is "is this really necessary?" You don't want to be in the beta test group. If the service works, the vendor should have a few customers that can provide references."
I always say that and am always dismissed as "negative". I'm sorry that I can't stand smarmy sales peeps who sell shit that clueless managers ALWAYS believe over their own staff.
Fucks me right off!
The response time of a resource like a communications link or a CPU goes up *exponentially* after it reaches 50%. The suits apparently did not know that.
In the early days of development of a certain nameless operating system the developers thought it would be clever to make use of the hardware's CRC32 instruction to generate hashes for account passwords. Some more math-oriented members of them team thought this was a very bad idea, but were rebuffed. So the math-nerds thought a demonstration was in order. They wrote a program that read the password files on every computer running this software (which they could do, being on the development team, and the files were not as protected as they might have been) and set about cracking the credentials. They had realized that it was not necessary to come up with the correct password for an account, but only come up with a password that *hashed to the same CRC32 value*. And that the payback of this effort would be increased by doing it for the entire network at once, which was hundreds of accounts. They essentially cracked all the passwords at once.
The next day everybody on the in-house network, developers and managers alike, received an email saying that here was either their password or one that would work just as well. I remember getting one of these emails.
The hashing algorithm was changed immediately.
Once upon a time my wife was going to visit her parents by train and I wanted to park at the railway station in Derby to see her off.
The parking area was full but there was a guy sitting in his car, obviously about to leave.
Several times I tried to make eye contact, but he ignored me and just sat there.
So I parked behind his car, blocking him into his parking space, and my wife and I got out of the car.
The reversing lights on the other guy's car came on instantly.
"Oh, sorry, did you want to leave? Let me move my car for you..."
I'm just not ok with the compromise of business integrity on this one. I would have simply kept making the recommendations, continuing to be more emphatic, until the managers approved the additional bandwidth. In my opinion, IT and security are in the categories that it's mandatory to trust what they say and do as absolute fact. Otherwise, it's a clown show. If the network was on the road to bandwidth starvation because of oversubscription, then so be it and let it happen but document the requests for a solution.if I was the IT manager or above the IT department and found that they had manipulated bandwidth, I would have cleared house with terminations,
so be it and let it happen
Problem with that is the fallout. It's no good saying "I told you so" if your biggest customer withdraws business and puts your job at risk, or your bank gets the sort of poor reputation that sticks around for years and leads to takeover, re-branding and "cost saving synergies". Let's face it, it's never the PHB who gets downsized for a preventable mistake.
Rabbits out of hats are specialities of many of us 'engineer' types and just occasionally we get a box of chocolates, but more importantly we get to keep paying the mortgage.
Then again, speaking as a little Dutch boy with finger-in-dyke desperately hoping someone will pay heed to my pleas to 'do something' about a fleet of equipment, some of which dates back 19 years and some of which is running OSes which went out of extended support ten years ago, I am beginning to wonder if I should have been building an Ark rather than getting a cold, wet, stiff digit.
Where I used to work we had 3 sites triangulated using 1Gb BT LES circuits this was about 10 years ago,
All was well for a couple of years until the company got bigger and we started running the links at 90% or more all the time.
Eventually we got the budget to go to 10Gb links instead.
So we got them delivered and one long weekend set about turning them on.
A few hours later we were finished and headed home, the next week the previously 1Gb link running at 95% has transformed into a 10Gb link running at 50%
We found that a lot of replication and batch jobs suddenly had there run time reduced dramatically- wonder why….
Anonymous because they plonker will instantly recognise himself if he reads it.
Wrote a DOS based label printing routine for bar codes for boxes big one on one for the king side little one goes in the end.
Real easy really cheap on paper ran it through a large office printer printed 52 pages a minute stormed through the the work.
Boss man tells us about new thermal printers that can print on anything and we could use them to print labelling for the clothes in the boxes AS well as the box labels.
Tried to explain about the costs, but using his superior brain and management training he made the executive decision.
Printers ordered new stationary organised and new silk labels ordered.
3 months in he had spent the entire years budget on stationary and printer repairs.
Overheard him whinging to another boss man about the stupid costs of IT.
Ah I left the office that evening smiling and with a spring in my step.
Justice.
I could upvote you for the tale, but also downvote you for the ending - so I've done neither.
The problem is that the person responsible doesn't associate the costs with his decisions. Now he's told others that it's your fault. So having had your reputation sullied, you leave with a spring in your step ?
Had you included that you'd kept records of the earlier exchanges where you'd warned of the costs, and "accidentally" let others see that (especially his boss), then you'd have had an upvote.
All plants were connected to to HQ with a lousy T1 yet he had 100 M/bps connectivity. He got tired of the complaints and said 1.5 M/bps was good enough for anyone. Since he had a DHCP reservation we used QoS to throttle him to T1 speed. After three days he threw in the towel and approved the connectivity upgrades.
Way back when, I worked for a boss who thought a little too much of himself. In the early days he was open to our ideas and was a seemingly congenial fella. Though I learned from others about some of his penchants and history for dishonesty toward vendors and customers. As time moved on, he became dismissive and sometimes just flat-out rude about our requests, and the dishonesty turned more inward against the company. We figured out that if we planted the seed in his mind about something we needed then forged a campaign of customer requests and complaints, within a few weeks he would bring up this amazing idea that absolutely needed to be implemented and in short order. Even though these are often last minute, we were always at the ready to get it done and put in the extra time if necessary. For the good of the company.