back to article Techie studied ancient ways of iSeries machine, saved day when user unleashed eldritch powers, got £50 gift voucher

What's that coming over the hill, could it be Friday? Hurrah! Time to settle down for a reminder about the dangers of over-modifying a system past its prime in today's instalment of On Call. "Ed" contributed today's story, which takes us back nearly a decade to a trading system running on an iSeries. The software itself was …

  1. KittenHuffer Silver badge

    What do you get given .....

    ..... if you prove to be good at shovelling sh*t?

    A BIGGER shovel!

    1. Evil Auditor Silver badge

      Re: What do you get given .....

      ...and more shit.

      1. KittenHuffer Silver badge
        Joke

        Re: What do you get given .....

        I thought that was quite clearly implied in the original post .... but then I realised that any posting by an Auditor was bound to be obvious and unnecessary! </tongue in cheek>

        1. Evil Auditor Silver badge
          Thumb Up

          Re: What do you get given .....

          I fully agree with you! But please, don't tell my clients. I make quite a good living with that.

          Btw, I mostly see that you only get more shit but no bigger shovel.

          1. Donn Bly

            Re: What do you get given .....

            In my experience they take away your shovel, give it to someone else, and expect you to use your hands.

            1. Robert Carnegie Silver badge

              Re: What do you get given .....

              It's an excellent shovel. It did a fine job.

      2. EVP Bronze badge
        Thumb Up

        Re: What do you get given .....

        Meanwhile, his boss got a shitload (his boss’s boss a fuckton) of stock options... for saving the day and stuff.

  2. My-Handle Silver badge

    However, he also got a reputation...

    "...as being the chap to go to when everything went wrong."

    I have a certain suspicion that this is where a lot of El Reg commentards live. For this very reason, I am extremely selective with who at work knows my mobile number

    1. KittenHuffer Silver badge

      Re: However, he also got a reputation...

      I once found myself in the 'joyous' situation where projects that were behind schedule would be transferred to me to 'finish off'. The culprit would then be given a fresh project that would later on also end up behind schedule. And I would be left trying to get the project out as close to the deadline as possible.

      And then come review time I would get slagged off for the number of projects that I delivered late! I made sure that I got out from under that PHB, and resisted every attempt from them to get me back.

    2. anothercynic Silver badge

      Re: However, he also got a reputation...

      Ditto. Although these days I am responsible for critical infrastructure, so they *have to* know my mobile number. Ugh.

      1. My-Handle Silver badge

        Re: However, he also got a reputation...

        Same here. The director of IT knows my number. My immediate manager (we've had several over a couple of years) is given my number with the extremely firm reminder that it's only to be used in an emergency. Two of my colleagues that I trust also have my number. That's it.

        That worked very well up until my particular section of IT ended up under Marketing, the Head got my number and I ended up getting a call a scant two weeks later while I was on holiday to try and clear up one of their messes.

        1. A.P. Veening Silver badge

          Re: However, he also got a reputation...

          getting a call a scant two weeks later while I was on holiday

          That alone is already triple over-time just for picking up your phone, quadruple if you are in another time zone. And yes, it also happened to me and I made the idiot pay for it after I came back from holiday and I made very sure it came out of his department's budget (causing him to go over budget, which really hurt). I also left the company shortly after citing the above as primary reason, making some more people unhappy with the above mentioned idiot.

          1. My-Handle Silver badge

            Re: However, he also got a reputation...

            Company doesn't pay overtime at all, and I got reprimanded when I offered to work overtime to deal with a significant company workload.

            My compromise there is to only work out of hours when it's absolutely essential, but sometimes it's hard to determine that on the spur of the moment. In this case, the work was only half an hour or so, so I resolved to do it and save the arguments for later.

          2. nematoad Silver badge

            Re: However, he also got a reputation...

            Yes, I really loved the expression on some PITA middle manager when I explained that the problem was out of scope and we would be charging their budget at the rate of £85 per hour or portion thereof.

            OK, it was a long time ago now and the rates have probably doubled or tripled by now.

        2. John Brown (no body) Silver badge

          Re: However, he also got a reputation...

          "That worked very well up until my particular section of IT ended up under Marketing,"

          Have you considered a career change? Maybe as a nursery school worker where the level of intellectual conversation might be a bit higher than currently?

          1. My-Handle Silver badge

            Re: However, he also got a reputation...

            Have actually worked in a nursery school :)

            And yes. Plans are very much in progress.

          2. Donn Bly

            Re: However, he also got a reputation...

            Only after the entitled parents leave

        3. Doctor Syntax Silver badge

          Re: However, he also got a reputation...

          "That worked very well up until my particular section of IT ended up under Marketing, the Head got my number and I ended up getting a call a scant two weeks later while I was on holiday to try and clear up one of their messes."

          There are times when the pain of changing a number is worth it. Or have separate work and life numbers and leave the life number at home when you're on holiday.

        4. Stuart Castle Silver badge

          Re: However, he also got a reputation...

          The people in my team, and also another team that works so closely with us that we are almost the same team, all have my mobile number. Certain other colleagues outside those teams also have my number.

          A few years ago, one of our users phoned me on my mobile, while I was at lunch. He said he had an urgent problem. I listened to the problem, told him the other technicians would be able to resolve it, and that I was at lunch. He said sorry, and I asked where he got the number. One of the other technicians gave it to him. Not surprisingly, I was rather angry. I went back to the office, and told the technician in no uncertain terms that it was my mobile number. I paid for it, and I felt if the company wanted me to provide support over a mobile phone, they should provide that phone.

          He also apologised.

    3. Anonymous Coward
      Anonymous Coward

      Re: However, he also got a reputation...

      I also have a reputation as the person to go to when things go wrong.

      ... but that's usually because I'm deemed to be the one responsible...

    4. jelabarre59 Silver badge

      Re: However, he also got a reputation...

      For this very reason, I am extremely selective with who at work knows my mobile number

      At one prior job, my wife had (un)helpfully provided my cell number to a team lead, who wanted me to log into a web conference while I was an hour away from home. I said I'd see if I could, presuming I got home in time. For some *strange* reason, the traffic home was *exceptionally* bad.

      For the remainder of my time at that job, I wouldn't answer my cell unless it was a number already in my phone book. (and no one from work would EVER be in my cell phone book).

    5. Tom 7 Silver badge

      Re: However, he also got a reputation...

      Apart from having an OS that is on your side Linus Torvald came up with the best excuse ever; "I haven't used Windows in years but I can dual boot your system if you like!"

    6. Prst. V.Jeltz Silver badge

      Re: However, he also got a reputation...

      all you pussies should let everyone know your number , in case its needed ,

      and then get all outraged if they take the piss.

  3. Anonymous Coward
    Anonymous Coward

    Maybe get a process to delay that 11am thing too

    First thing I would have asked would be to delay that 11am scheduled task, would give more time to fix the main issue and could be someone else to look into so can be done in parallel.

    2nd thing would be to see if there was a cache or log of what was pending and see if that could be used to restore the queue.

    3rd would be to recreate everything, as we know that’d be resource and time intensive.

    Ed is sure a hero here, still unsung despite the voucher.

    1. Giles C Bronze badge

      Re: Maybe get a process to delay that 11am thing too

      Most likely with the 11am would be the deadline by a bank or other organisation to get the data over so it would be processed.

      I have seen this with credit card transactions, the days file has to be there by a certain time or nothing goes through and there is a giant hole in the accounts.

      1. Anonymous Coward
        Anonymous Coward

        Re: Maybe get a process to delay that 11am thing too

        Depending on the system, there are also very big fines for missing the deadline.

      2. James Anderson Silver badge

        Re: Maybe get a process to delay that 11am thing too

        11 a.m. - 6 hours == 5.00.

        Probably opening of business at some NYC based trading platform.

      3. Olivier2553 Silver badge

        Re: Maybe get a process to delay that 11am thing too

        Go rent a JBC and dig a big hole in the communication infrastructure. By the time the network will be restored, it will be after 11.

    2. Anonymous Coward
      Anonymous Coward

      Re: Maybe get a process to delay that 11am thing too

      If you don’t ask you don’t get,

      Could they just run 2 lots of data the following day.

      Often automated systems have a manual fall back or some other contingency.

  4. Evil Auditor Silver badge

    Scary shit indeed. I can only imagine how "well-tested" that SQL was. In my experience they mostly come with side effects, usually severe ones. Which, of course, may well be because I'm crap at SQL.

  5. Boris the Cockroach Silver badge
    Unhappy

    Its

    worse when the manager gets the voucher for us plebs hard work at averting a disaster.....

    1. Chris G Silver badge

      Re: Its

      History shows that while soldiers may get medals, it's the Generals who get knighted or an Earldom for what the soldiers have done.

      1. The Oncoming Scorn Silver badge
        Mushroom

        Re: Its

        I had one (Non) manager who having heard the Zebra printers needed a regular roller clean following a jam & this was covered under the support contract, suggested that I do a regular sweep to save costs.

        Then he nominated this as a project idea he had implemented & got an award\gift card for making the suggestion, while I went across two sites trying to find every fucking Zebra in every production area* in a blazing summer heatwave (Icon).

        Didn't save any money, as we still needed the maintenance contract & after my token sweep of the printers I could find, never went near them again & simply waited for the next reported jam typically due in 8 months time.

        *For some reason I don't recall finding to many.

        1. A.P. Veening Silver badge
          Pint

          Re: Its

          *For some reason I don't recall finding to many.

          And you spent hours looking for every single one ;)

  6. Scotthva5

    All software needs...

    ...a "bring forth the fluffy bunnies" option.

    1. DJO Silver badge

      Re: All software needs...

      Not sure, the rabbit of Caerbannog was quite fluffy and there was only ever one holy hand grenade of Antioch.

    2. Anonymous South African Coward Silver badge

      Re: All software needs...

      El-Ahrairah approves of a "bring forth the fluffy bunnies" option...

  7. juice Silver badge

    That whooshing deadline sound...

    Back when, I worked on a billing system which ran daily, and which couldn't be allowed to go past midnight.

    And as is often the way with such organically-evolved-over-time things, it was a bit flakey - it was essentially lots of processes which had to be ran in a specific sequence before a final process glued all the bits together and issued the bills.

    Thankfully, to speed things up a little, this final process had been retro-fitted with a "multi-threaded" mechanism. Which generally worked pretty well, but the more you cranked it up, the more likely it was that individual threads would blow up and would need manually cleaning up and restarting.

    There then came a day when we had triple the usual number of bills to process than usual, and as a result were getting worryingly close to the witching hour.

    So, since we were already manually monitoring, I simply cranked up the threadcount ten-fold and cleaned up the failures on the fly, while the finance team hovered nearby and ordered the occasional pizza.

    I think we scraped past the finish line with around 30 minutes to spare. Or about 13 hours later than on a BAU day ;)

    1. Stumpy Silver badge

      Re: That whooshing deadline sound...

      I used to look after a Europe-wide IT Asset management and Billing system for an account of one of the big Outsourcers of the time.

      The monthly billing run used to take several days, so we'd normally kick it off on a Friday and it would be finished (usually, if nothing went wrong) by the following Tuesday or Wednesday.

      Inevitably, the systems were so flaky (the main billing system was coded in PowerBuilder and ran on the Mainframe so we didn't have access to the application itself - only our little part of the interface) that I used to simply take a sleeping bag into the office and bed down under my desk during the billing run since we had no remote access or laptops in our team back then, and have an alarm set to wake me up every four hours or thereabouts so I could check whether it was still running or not so we wouldn't lose a whole nights processing.

      Was glad when I eventually pressed the eject button and left for better climes.

      1. Doctor Syntax Silver badge

        Re: That whooshing deadline sound...

        "Inevitably, the systems were so flaky"

        The worst one I inherited included some jobs that used tcl/expect to run vi to write and run scripts on the fly. Distinctly "if it ain't broke" territory. Fortunately it stayed unbroke.

        1. logicalextreme Silver badge

          Re: That whooshing deadline sound...

          Over time you acquire a hushed awe of shit code that's still somehow bulletproof.

          1. A.P. Veening Silver badge

            Re: That whooshing deadline sound...

            Over time you acquire a hushed awe of shit code that's still somehow bulletproof.

            As long as the bullets are small enough. Unfortunately, that larger bullet will come around some time, just pray to the deity or saint of your choice and Murphy that it isn't on your watch.

            1. Doctor Syntax Silver badge

              Re: That whooshing deadline sound...

              I must say I was nervous about the expect stuff. It could have been done a bit easier to script with ex. Yes, probably a parametrised shell script would have done the job but --- don't fix it, don't even be tempted to take a look.

          2. Stumpy Silver badge

            Re: That whooshing deadline sound...

            Hmm. Probably less ‘hushed awe’ and more ‘existential terror’

    2. Stevie Silver badge

      Re: That whooshing deadline sound...(4 juice)

      I have one like that.

      I was working a contract in the south of England in a shop very contractor-heavy. Sperry mainframe.

      To send a file to a printer you use a command of the format "@SYM <filename>".

      Came an afternoon when no-one could print because one consultant had sent 9 copies of a very large file to the printer.

      The increasingly angry office staff were demanding what was going on, glaring at the consultants, and finally the guilty party owned up. Much groaning and gnashing of teeth but no ideas.

      So muggins says, "Why on earth didn't you ask for it to be printed on three-part?" Sounds of an officefull of people headslapping.

      So she cancelled the last six prints and rescheduled two on special stationery.

      Had we been working on older equipment this wouldn't have happened, but Sperry made the "obvious" way so much easier.

      I see the same thing these days with DBAs using Toad and turning what should be quick operations using the Oracle data dictionary to build queries into manually intensive operations on Toad.

      "It'll take hours to match the test server object list status to the production server."

      "If you do it the old way it'll take a few minutes to set up, and a few more to enact".

  8. karlkarl Silver badge

    Lucky it was an "elderly" bit of code...

    Newer codebases would take more than 2 hours to work out the stupid sodding build system and fetch all the billions of unnecessary dependencies!

    A cool article though, I could feel the tension XD

    1. A.P. Veening Silver badge

      Re: Lucky it was an "elderly" bit of code...

      As it was on an iSeries aka AS/400, chances are very good (about 80%) it was RPG, which is a disaster to read if you aren't extremely familiar with it. If he was lucky, it was either RPGLE or COBOL, both of which are a bit easier to read and understand.

      1. logicalextreme Silver badge

        Re: Lucky it was an "elderly" bit of code...

        RPGLE was new to me, so I parsed it as "rocket-propelled grenade little-endian". Which I'm not sure but think might mean holding the launcher backwards.

        1. A.P. Veening Silver badge

          Re: Lucky it was an "elderly" bit of code...

          Originally RPG (the programming language) stands for Report Program Generator, RPGLE stands for RPG Language Environment, where the Language Environment is a way for programs on the iSeries to be made up from components written in various languages to be bound together in one executable.

      2. Elfoad Regfoad

        Re: Lucky it was an "elderly" bit of code...

        Re the "disaster to read if you aren't extremely familiar with it" - before IBM introduced "free-form" support in RPG IV, all RPG was formatted in a fixed column style, so Operand1, Operand2, Opcode, Conditional indicators, and Result indicators were always in the same place on the screen. This created a sort of tall tree with side branches pattern when you scrolled through it. Maybe I was "extremely familiar" with it, but this sort of regular pattern (as compared to free-form, Pythonesque spacing, arbitrary curly brackets etc.) often made it easy to find problems in the code due to an incongruity in the expected pattern, particularly if you used the option that wrote the indentation level into the comments field. Just sayin'.

    2. Strahd Ivarius Bronze badge
      Devil

      Re: Lucky it was an "elderly" bit of code...

      just imagine if it had been an eldritch bit of code...

  9. Shadow Systems Silver badge

    "...relabelling the nuclear launch button to one entitled "bring forth the fluffy bunnies"..."

    *Jumps up & down on the button while squealing in glee like Gir from Invader Zim*

    We're all gonna die! FINE-LEE!!!

    *JumpJumpJump*

  10. OSYSTEM
    Angel

    Business as usual

    Today we call this Devops...

  11. Anonymous Coward
    Anonymous Coward

    The sad thing is, I deal with shit like this TODAY on an almost daily basis.

    Systems that are decades old that the customer failed repeatedly to upgrade. Daily escalations. Daily finger pointing and grumbling. Daily me pulling out emails from years ago where I point out the problems and what needs to be done to fix them.

    Some PHBs just never learn.

    1. UCAP

      Been there, got the scars to prove it.

      I ended up with an e-mail folder where I kept copies of every e-mail sent pointing out just how close they were to a major disaster (generally one very, very small step), along with the read receipts and responses (the later were few and far between). That folder was very frequently used!

    2. UCAP

      Some PHBs just never learn.

      Sadly, that is the definition of a PHB.

    3. Anonymous South African Coward Silver badge

      Obligatory Dilbert link coming up soon...

    4. Alan Brown Silver badge

      Some PHBs have gullible underlings who keep pulling their balls out of the oven instead of letting them bake

  12. steviebuk Silver badge

    You get told "We're not renewing your contract anymore"

    Despite being helpful, useful and liked, you could get told the above. Despite, no managers being in one specific day and helping theatres take in much needed money that day. Where an update at the outsourced payment company, had killed their server. So they needed someone to go round and update all the chip and pin machines for them. No one was around but me. I said just tell them to send me what they need to do and I'll do it. All I got was a thanks then about a year later all was forgotten and my contract wasn't renewed.

    Cunts.

    1. John Brown (no body) Silver badge
      Pint

      Re: You get told "We're not renewing your contract anymore"

      Have a pint of bitter on me :-)

    2. DS999
      WTF?

      Re: You get told "We're not renewing your contract anymore"

      Don't take a contract job if you expect that the contract is going to last forever.

    3. Doctor Syntax Silver badge

      Re: You get told "We're not renewing your contract anymore"

      That's the nature of contract work. Leave on good terms, keep in touch. Sometimes there are more contracts down the line.

    4. nematoad Silver badge

      Re: You get told "We're not renewing your contract anymore"

      Ah, I got told that the one and only time I was a contractor.

      The contract was handed over to a new company and the new manager decided that he wanted to convert all the contractors to permies. A lot of my team left immediately with the old company, the rest of us stayed to see what the new lot proposed. It turned out that the new manager wanted us contractors to go permanent and as an incentive proposed to give us a 30% pay cut. I declined the offer and was soon out the door with several of my colleagues, which for me was the best thing that ever happened. I got a beaut of a job in the Republic of Ireland where me and my senior analyst were the two best paid people in the place! Pity the bastards decided we were too expensive and turfed us out but that is another story. All I will say is that they got what they paid for which was peanuts on the new contract. And you know what you get when you offer that sort of money.

      1. Anonymous South African Coward Silver badge

        Re: You get told "We're not renewing your contract anymore"

        ...the new manager wanted us contractors to go permanent and as an incentive proposed to give us a 30% pay cut.

        Stuff that, I'll walk rather.

      2. steviebuk Silver badge

        Re: You get told "We're not renewing your contract anymore"

        Yep. In a place I was once at, the "peanuts paid contractors" (I'm not condoning what they did but it was predicable due to how they were treated), decided that about 20 of the over 100 PCs they were in charge of rolling out, wouldn't make it to the offices they were marked for. But instead, would be distributed amongst themselves and end up in their homes.

        1. Because the audit travel of the kit was PISS POOR it was someones pet project and they just wanted it rushed.

        2. They knew because auditing was so PISS POOR, the PCs could disappearing without much notice and if they were eventually noticed, they'd be long gone.

        3. Because at the start of the roll out they were all called into a meeting with a proper cunt of a manager who said "You're all here to do a PC roll out. You're all monkeys and can be replaced so if you don't turn up or turn up late, you're out and will be replaced by the next monkey".

        I'll say again, he was a cunt!

  13. Doctor Syntax Silver badge
    Flame

    "However, he also got a reputation as being the chap to go to when everything went wrong"

    Letting one of the less destructive ones go -> might be a good idea.

    Don't be seen as too infallible.

  14. Paul Cooper

    I guess most of us here are used to people poking their head round the door and saying "Paul, How do I...."

    1. DJV Silver badge

      Well, if they did that to me I'd ignore them - because my name isn't Paul!

    2. logicalextreme Silver badge

      The correct response is "…you DON'T".

      1. Doctor Syntax Silver badge

        or "very carefully"

        1. Ken Shabby

          or "if you must do, with his"

      2. Boris the Cockroach Silver badge

        My answer normally involves sex and travel, with a side hint that I'm trying to sort out a previous f*** up made by the same person sticking his head round the door....

        Maybe I'll use the E.Blackadder solution of slamming his head in the door....

      3. Alan Brown Silver badge

        "…you DON'T".

        Right statement, but emphasis on the wrong word.

  15. Kev99

    And in a similar vein...

    Where I used to work we'd periodically get a new bit of software, sometimes from a third party, sometime brewed up in house. In my department I always insisted (demanded) the kits run in parallel for at least a week before unplugging the original. The IT manager always refused claiming it would cause double the work load for my staff & his. Since a lot of my staff took their loving time to get anything done I didn't really care if their load doubled. Needles to say I'd always lose that discussion. So guess what happened one day? Yup, the new system crapped out completely. It had to be reinstalled, reconfigured, data re-entered (yes, even with backups) and I seem to remember disc level diagnostics had to be run. Yup, running parallel would cause too much trouble for everyone.

    1. G.Y.

      necessary but ... Re: And in a similar vein...

      I heard that Rice University (ages ago) changed over to a computerized system, after a year of running everything in parallel with the old clay tablets.

      Then one student cancelled a course he was not registered for. Total crash.

    2. Terry 6 Silver badge

      Re: And in a similar vein...

      This is the insurance problem. AKA "We don't need insurance, we've never had a (named disaster)"

      Old joke: Chap on a hike finds the path goes past the edge of a very sudden drop.

      When he gets to the next village he says to the inn keeper, "That's a scary drop back there. I'm surprised there's no warning sign".

      "Oh there was", was his reply. "But it got damaged and since no one had ever fallen we took it away".

  16. JimC

    Old Code...

    Digging out an old but useful utility and modding it slightly for a new task my PFY commented "when you wrote that I was in primary school"

    1. A.P. Veening Silver badge

      Re: Old Code...

      I have done maintenance on code which was originally written when I was in high school, making that code at least 30 years old at that time (and ten years later it is still running, but I don't do maintenance on it anymore as I left the company).

  17. theOtherJT

    Reminds me of some cataloguing software I once had to use...

    ...that was of a similar "ported forward from a minicomputer, running on a green-screen" type construction. It was comprised of a number of forms built out of multiple pages of text entry boxes - some of them 20 or 30 pages long. As you filled in these forms the only time you could save them was on the last page. You also couldn't go _back_ once you'd filled a page in except by pressing on, saving the entry, and then re-opening it for modification.

    All of that was just plain bad UI design, but far worse was the way they handled buffering keyboard input - which is to say, they didn't - and if you were a proficient enough typist you could cause the thing to corrupt the current page by simply typing too fast. The "fix" for this that was implemented when we reported the problem was for them to add a keyboard shortcut that would flush all unsaved data and reload the current form.

    Imagine typing fast enough to corrupt it on page 19 of a 20 page form that you've not saved yet and the only thing to do is press alt+ctrl+q and lose the last 20 minutes work...

    God I hated that thing.

  18. analyzer
    Alert

    took 3 times to read this

    Each time I suffered a deep gut wrenching feeling from times long ago and thought forgotten, I'll bet that I'm not the only one.

    1. jonathan keith Silver badge

      Re: took 3 times to read this

      PTSD flashbacks - the gift that keeps on giving.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020