Techie studied ancient ways of iSeries machine, saved day when user unleashed eldritch powers, got £50 gift voucher What's that coming over the hill, could it be Friday? Hurrah! Time to settle down for a reminder about the dangers of over-modifying a system past its prime in today's instalment of On Call. "Ed" contributed today's story, which takes us back nearly a decade to a trading system running on an iSeries. The software itself was …
I once found myself in the 'joyous' situation where projects that were behind schedule would be transferred to me to 'finish off'. The culprit would then be given a fresh project that would later on also end up behind schedule. And I would be left trying to get the project out as close to the deadline as possible.
And then come review time I would get slagged off for the number of projects that I delivered late! I made sure that I got out from under that PHB, and resisted every attempt from them to get me back.
Same here. The director of IT knows my number. My immediate manager (we've had several over a couple of years) is given my number with the extremely firm reminder that it's only to be used in an emergency. Two of my colleagues that I trust also have my number. That's it.
That worked very well up until my particular section of IT ended up under Marketing, the Head got my number and I ended up getting a call a scant two weeks later while I was on holiday to try and clear up one of their messes.
getting a call a scant two weeks later while I was on holiday
That alone is already triple over-time just for picking up your phone, quadruple if you are in another time zone. And yes, it also happened to me and I made the idiot pay for it after I came back from holiday and I made very sure it came out of his department's budget (causing him to go over budget, which really hurt). I also left the company shortly after citing the above as primary reason, making some more people unhappy with the above mentioned idiot.
Company doesn't pay overtime at all, and I got reprimanded when I offered to work overtime to deal with a significant company workload.
My compromise there is to only work out of hours when it's absolutely essential, but sometimes it's hard to determine that on the spur of the moment. In this case, the work was only half an hour or so, so I resolved to do it and save the arguments for later.
Yes, I really loved the expression on some PITA middle manager when I explained that the problem was out of scope and we would be charging their budget at the rate of £85 per hour or portion thereof.
OK, it was a long time ago now and the rates have probably doubled or tripled by now.
"That worked very well up until my particular section of IT ended up under Marketing, the Head got my number and I ended up getting a call a scant two weeks later while I was on holiday to try and clear up one of their messes."
There are times when the pain of changing a number is worth it. Or have separate work and life numbers and leave the life number at home when you're on holiday.
The people in my team, and also another team that works so closely with us that we are almost the same team, all have my mobile number. Certain other colleagues outside those teams also have my number.
A few years ago, one of our users phoned me on my mobile, while I was at lunch. He said he had an urgent problem. I listened to the problem, told him the other technicians would be able to resolve it, and that I was at lunch. He said sorry, and I asked where he got the number. One of the other technicians gave it to him. Not surprisingly, I was rather angry. I went back to the office, and told the technician in no uncertain terms that it was my mobile number. I paid for it, and I felt if the company wanted me to provide support over a mobile phone, they should provide that phone.
He also apologised.
For this very reason, I am extremely selective with who at work knows my mobile number
At one prior job, my wife had (un)helpfully provided my cell number to a team lead, who wanted me to log into a web conference while I was an hour away from home. I said I'd see if I could, presuming I got home in time. For some *strange* reason, the traffic home was *exceptionally* bad.
For the remainder of my time at that job, I wouldn't answer my cell unless it was a number already in my phone book. (and no one from work would EVER be in my cell phone book).
First thing I would have asked would be to delay that 11am scheduled task, would give more time to fix the main issue and could be someone else to look into so can be done in parallel.
2nd thing would be to see if there was a cache or log of what was pending and see if that could be used to restore the queue.
3rd would be to recreate everything, as we know that’d be resource and time intensive.
Ed is sure a hero here, still unsung despite the voucher.
Most likely with the 11am would be the deadline by a bank or other organisation to get the data over so it would be processed.
I have seen this with credit card transactions, the days file has to be there by a certain time or nothing goes through and there is a giant hole in the accounts.
I had one (Non) manager who having heard the Zebra printers needed a regular roller clean following a jam & this was covered under the support contract, suggested that I do a regular sweep to save costs.
Then he nominated this as a project idea he had implemented & got an award\gift card for making the suggestion, while I went across two sites trying to find every fucking Zebra in every production area* in a blazing summer heatwave (Icon).
Didn't save any money, as we still needed the maintenance contract & after my token sweep of the printers I could find, never went near them again & simply waited for the next reported jam typically due in 8 months time.
*For some reason I don't recall finding to many.
Back when, I worked on a billing system which ran daily, and which couldn't be allowed to go past midnight.
And as is often the way with such organically-evolved-over-time things, it was a bit flakey - it was essentially lots of processes which had to be ran in a specific sequence before a final process glued all the bits together and issued the bills.
Thankfully, to speed things up a little, this final process had been retro-fitted with a "multi-threaded" mechanism. Which generally worked pretty well, but the more you cranked it up, the more likely it was that individual threads would blow up and would need manually cleaning up and restarting.
There then came a day when we had triple the usual number of bills to process than usual, and as a result were getting worryingly close to the witching hour.
So, since we were already manually monitoring, I simply cranked up the threadcount ten-fold and cleaned up the failures on the fly, while the finance team hovered nearby and ordered the occasional pizza.
I think we scraped past the finish line with around 30 minutes to spare. Or about 13 hours later than on a BAU day ;)
I used to look after a Europe-wide IT Asset management and Billing system for an account of one of the big Outsourcers of the time.
The monthly billing run used to take several days, so we'd normally kick it off on a Friday and it would be finished (usually, if nothing went wrong) by the following Tuesday or Wednesday.
Inevitably, the systems were so flaky (the main billing system was coded in PowerBuilder and ran on the Mainframe so we didn't have access to the application itself - only our little part of the interface) that I used to simply take a sleeping bag into the office and bed down under my desk during the billing run since we had no remote access or laptops in our team back then, and have an alarm set to wake me up every four hours or thereabouts so I could check whether it was still running or not so we wouldn't lose a whole nights processing.
Was glad when I eventually pressed the eject button and left for better climes.
Over time you acquire a hushed awe of shit code that's still somehow bulletproof.
As long as the bullets are small enough. Unfortunately, that larger bullet will come around some time, just pray to the deity or saint of your choice and Murphy that it isn't on your watch.
I have one like that.
I was working a contract in the south of England in a shop very contractor-heavy. Sperry mainframe.
To send a file to a printer you use a command of the format "@SYM <filename>".
Came an afternoon when no-one could print because one consultant had sent 9 copies of a very large file to the printer.
The increasingly angry office staff were demanding what was going on, glaring at the consultants, and finally the guilty party owned up. Much groaning and gnashing of teeth but no ideas.
So muggins says, "Why on earth didn't you ask for it to be printed on three-part?" Sounds of an officefull of people headslapping.
So she cancelled the last six prints and rescheduled two on special stationery.
Had we been working on older equipment this wouldn't have happened, but Sperry made the "obvious" way so much easier.
I see the same thing these days with DBAs using Toad and turning what should be quick operations using the Oracle data dictionary to build queries into manually intensive operations on Toad.
"It'll take hours to match the test server object list status to the production server."
"If you do it the old way it'll take a few minutes to set up, and a few more to enact".
As it was on an iSeries aka AS/400, chances are very good (about 80%) it was RPG, which is a disaster to read if you aren't extremely familiar with it. If he was lucky, it was either RPGLE or COBOL, both of which are a bit easier to read and understand.
Originally RPG (the programming language) stands for Report Program Generator, RPGLE stands for RPG Language Environment, where the Language Environment is a way for programs on the iSeries to be made up from components written in various languages to be bound together in one executable.
Re the "disaster to read if you aren't extremely familiar with it" - before IBM introduced "free-form" support in RPG IV, all RPG was formatted in a fixed column style, so Operand1, Operand2, Opcode, Conditional indicators, and Result indicators were always in the same place on the screen. This created a sort of tall tree with side branches pattern when you scrolled through it. Maybe I was "extremely familiar" with it, but this sort of regular pattern (as compared to free-form, Pythonesque spacing, arbitrary curly brackets etc.) often made it easy to find problems in the code due to an incongruity in the expected pattern, particularly if you used the option that wrote the indentation level into the comments field. Just sayin'.
The sad thing is, I deal with shit like this TODAY on an almost daily basis.
Systems that are decades old that the customer failed repeatedly to upgrade. Daily escalations. Daily finger pointing and grumbling. Daily me pulling out emails from years ago where I point out the problems and what needs to be done to fix them.
Some PHBs just never learn.
Been there, got the scars to prove it.
I ended up with an e-mail folder where I kept copies of every e-mail sent pointing out just how close they were to a major disaster (generally one very, very small step), along with the read receipts and responses (the later were few and far between). That folder was very frequently used!
Despite being helpful, useful and liked, you could get told the above. Despite, no managers being in one specific day and helping theatres take in much needed money that day. Where an update at the outsourced payment company, had killed their server. So they needed someone to go round and update all the chip and pin machines for them. No one was around but me. I said just tell them to send me what they need to do and I'll do it. All I got was a thanks then about a year later all was forgotten and my contract wasn't renewed.
Cunts.
Ah, I got told that the one and only time I was a contractor.
The contract was handed over to a new company and the new manager decided that he wanted to convert all the contractors to permies. A lot of my team left immediately with the old company, the rest of us stayed to see what the new lot proposed. It turned out that the new manager wanted us contractors to go permanent and as an incentive proposed to give us a 30% pay cut. I declined the offer and was soon out the door with several of my colleagues, which for me was the best thing that ever happened. I got a beaut of a job in the Republic of Ireland where me and my senior analyst were the two best paid people in the place! Pity the bastards decided we were too expensive and turfed us out but that is another story. All I will say is that they got what they paid for which was peanuts on the new contract. And you know what you get when you offer that sort of money.
Yep. In a place I was once at, the "peanuts paid contractors" (I'm not condoning what they did but it was predicable due to how they were treated), decided that about 20 of the over 100 PCs they were in charge of rolling out, wouldn't make it to the offices they were marked for. But instead, would be distributed amongst themselves and end up in their homes.
1. Because the audit travel of the kit was PISS POOR it was someones pet project and they just wanted it rushed.
2. They knew because auditing was so PISS POOR, the PCs could disappearing without much notice and if they were eventually noticed, they'd be long gone.
3. Because at the start of the roll out they were all called into a meeting with a proper cunt of a manager who said "You're all here to do a PC roll out. You're all monkeys and can be replaced so if you don't turn up or turn up late, you're out and will be replaced by the next monkey".
I'll say again, he was a cunt!
Where I used to work we'd periodically get a new bit of software, sometimes from a third party, sometime brewed up in house. In my department I always insisted (demanded) the kits run in parallel for at least a week before unplugging the original. The IT manager always refused claiming it would cause double the work load for my staff & his. Since a lot of my staff took their loving time to get anything done I didn't really care if their load doubled. Needles to say I'd always lose that discussion. So guess what happened one day? Yup, the new system crapped out completely. It had to be reinstalled, reconfigured, data re-entered (yes, even with backups) and I seem to remember disc level diagnostics had to be run. Yup, running parallel would cause too much trouble for everyone.
This is the insurance problem. AKA "We don't need insurance, we've never had a (named disaster)"
Old joke: Chap on a hike finds the path goes past the edge of a very sudden drop.
When he gets to the next village he says to the inn keeper, "That's a scary drop back there. I'm surprised there's no warning sign".
"Oh there was", was his reply. "But it got damaged and since no one had ever fallen we took it away".
...that was of a similar "ported forward from a minicomputer, running on a green-screen" type construction. It was comprised of a number of forms built out of multiple pages of text entry boxes - some of them 20 or 30 pages long. As you filled in these forms the only time you could save them was on the last page. You also couldn't go _back_ once you'd filled a page in except by pressing on, saving the entry, and then re-opening it for modification.
All of that was just plain bad UI design, but far worse was the way they handled buffering keyboard input - which is to say, they didn't - and if you were a proficient enough typist you could cause the thing to corrupt the current page by simply typing too fast. The "fix" for this that was implemented when we reported the problem was for them to add a keyboard shortcut that would flush all unsaved data and reload the current form.
Imagine typing fast enough to corrupt it on page 19 of a 20 page form that you've not saved yet and the only thing to do is press alt+ctrl+q and lose the last 20 minutes work...
God I hated that thing.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
