back to article That script I wrote three years ago is now doing what? How many times?

Welcome once again to the wafer-thin buffer between the weekend and the workday that we call Who, Me? in which Reg readers share tales of tech tricks that didn't quite work out as hoped. This week, our hero is someone we've chosen to Regomize as "Indiana" who has reached deep into the archive of a top secret storage facility …

  1. trevorde Silver badge

    Roll your own ... everything

    Worked on a project many years ago where the best tech for our situation was Microsoft Message Queue (MSMQ). Only problem was management didn't want to spring for the license fees. The directive was for us to build it ourselves. It went as well as expected ie flaky, badly written & undocumented. The irony was that MSMQ was included (free) in the next release of Windows.

    Also ironic was that everyone their own (badly written & undocumented) string & date classes.

    1. Anonymous Coward
      Anonymous Coward

      Re: Roll your own ... everything

      It went as well as expected ie flaky, badly written & undocumented.

      So, just like MSMQ then?

      1. Elongated Muskrat Silver badge

        Re: Roll your own ... everything

        Yeah, there's a reason people use RabbitMQ

    2. A Non e-mouse Silver badge

      Re: Roll your own ... everything

      It annoys me how people forget that staff actually cost money. And it gets worse once management start penny pinching. Why the heck do you worry about spending an extra £50 on higher spec monitors for your staff when you're paying your staff five or six figure salaries?

      1. Anonymous Coward
        Anonymous Coward

        Re: Roll your own ... everything

        because the person or department who pays out the salaries is not the same person or department who pays for a monitor, probably.

        The bigger the company, the more likely silly things like this will happen. Small startups are happy to just go and buy something or licence something, if it's obvious it's going to save money compared to paying their own staff to work around the problem. But big corporations? Don't seem to have the ability to do that kind of calculation....

        1. Killfalcon Silver badge

          Re: Roll your own ... everything

          That calculation is hard, the benefit is at some nebulous future date, and the quarterly budget targets need to be hit this month.

          Far too many managers miss that "incentives drive behaviour" isn't just a phrase about setting their bonus targets. Stress and pressure give incentives. Budgets and time give incentives. You may not like them, but they're there, making your staff do dumb shit.

          1. ChrisC Silver badge

            Re: Roll your own ... everything

            "That calculation is hard"

            Depends on where you sit relative to the problem the lack of investment is causing...

            In a former employment, which had absolutely no shortage of cash to throw at things, but also had absolutely no concept of the differing requirements of R&D engineers vs anyone else for whom a PC was an integral part of their daily working life, I distinctly remember spending all of about 10 minutes jotting down a quick list of all the time my under-specced PC was costing me, and thus the company, due to having to wait for it to do stuff that it really ought to have been able to do far faster if only it'd been configured with the appropriate amount of memory for the tools I was running on it, as opposed to the appropriate amount for running the generic set of tools that everyone in the company was presumed to be using - Word, Excel, Lotus Notes (shudder) etc.

            Took the list to my manager, who looked it over for all of about a minute before approving the necessary memory upgrade.

            So it's not necessarily that the calculations are hard, it's more a question of making sure the people who actually know how to do the calculations correctly are involved in the process, rather than leaving it to someone so far removed from the end users that they have no concept of what impact their decisions are having on them.

            Working for that employer taught me a valuable lesson re the inability to assume that a company which has the ability to spend money on equipping its employees to do their jobs effectively, also has the ability to understand that its in its best interests to do so. It also taught me that working for a huge well funded employer, with loads of obvious staff perks (free meals, private healthcare etc.) isn't necessarily going to be as enjoyable as working for a smaller company who can't afford to throw money at employment packages but at least treats you more like an individual than merely another entry in their HR database...

            1. jake Silver badge

              Re: Roll your own ... everything

              This is nothing new in the IT world, and it's nearly ubiquitous, alas ... but not always. I was working on early UNIX (pre-BSD) at Berkeley when one of the professors bellyached at me for including too many comments, as in his opinion it wasted paper.

              "Yeah, well, get me a glass tty and I'll stop using the '33 ... "

              I had a second-hand VT52 on my desk the following morning. JOY! :-)

              The very same professor still demanded weekly dead-tree printouts. Naturally.

              1. RobDog

                Re: Roll your own ... everything

                But the VT52 was not released until 1974, the same year that UNIX arrived at Berkeley (both according to Wikipedia I grant you) so how does that work, the terminal being second hand? What was the professor’s name? Anyone would remember something that significant.

                1. jake Silver badge

                  Re: Roll your own ... everything

                  The first "owner" had dropped out, and I squeaked just in time to profit from it. It wasn't a VT52, it was a VT50 (brainfart, sorry). DEC had donated several to the school. DEC swapped it for a pre-production VT52 about six months later.

                  The professor's name was Elwyn Berlekamp.

                  1. John Brown (no body) Silver badge

                    Re: Roll your own ... everything

                    "The professor's name was Elwyn Berlekamp."

                    LOL, I kinda get the feeling that was intended as a trick question from someone assuming you were blowing hot air.

                    (@RobDog, please correct me if I'm wrong and I apologise now if I was)

                    Having said that, being such an unusual name (at least to me), I looked at his Wikipedia page. Looks like he was a very clever bloke who only died fairly recently. However, there's no mention of YOU on that Wikipedia page, so.... :-p

      2. RobDog

        Re: Roll your own ... everything

        You answered your own question there.

    3. PRR Bronze badge

      Re: Roll your own ... everything

      > Roll your own ... everything

      The school I worked was cheap. I even rolled our own computer case (for an 286-AT, not some micro-trainer). Rolled several wall-racks. (I even prefer wood screws if the gear won't come out very often.) Took a finally-obsolete PC-XT power supply and a superseded car radio to make a "free" office radio (two 8 Ohm 20W resistors on the +5V to stay stable). I rolled a PHP-like CMS in DOS-Basic, Visual Basic, and Perl, when PHP was still a baby (a very long time ago). Before that I rolled a signboard with text markup, a 5150 IBM PC, and three sad 23-inch Trinitron TV sets.

  2. b0llchit Silver badge

    Penny saved,...

    You get what you pay for, and then some, for not following the instructions.

  3. aje21

    Software RAID

    Many years ago we had to put in some very small "servers" which were basically PCs running Windows Server - because there was no budget for a RAID controller, we just dropped in a 2nd HDD and set up software RAID to mirror the primary drive. Not an ideal solution, but it was quick to set up, didn't need any drivers and was sufficient for the needs of the client at the time.

    1. Julian 8 Silver badge

      Re: Software RAID

      Did that in some old MicroServers from HP. Far cheaper to buy those than a NAS, put a boot drive in place of the CD and then configured a software RAID

      Took and absolute age to rebuild if a disk failed.

      At least someone hacked the bios to support hot swap

    2. Silver badge
      IT Angle

      Re: Software RAID

      I still prefer software RAID just because I hate dealing with the hardware. Everything is (or at least was) some proprietary whitelabeled Adaptec® thing that all use the same DOS programs to flash with no better alternative. When the things die (and they will die, usually at inconvenient times) you have to go buy another one and/or replace the whole backplane if it's designed stupid.

      Meanwhile ZFS can run on any system—you can yoink the array out of a FreeBSD server and plonk it in a freshly built Linux and it will mount like nothing changed. You can disable features you don't need or don't have resources for. You can scale as small or as large as you want. Sure you need better CPU and memory to support things like dedupe (if you really need it), you should be investing in flash-based ARC, and the same for a mirrored special vdev (small block writes and metadata), but at the end of the day you will have a much more flexible system that can be built and rebuilt using off the shelf components instead of some flaky controller that will eventually go EOL.

      1. Anonymous Coward
        Anonymous Coward

        Re: Software RAID

        Is there a convenient time for kit to die?

        1. Jou (Mxyzptlk) Silver badge

          Re: Software RAID

          > Is there a convenient time for kit to die?

          Yes, right after it is not your responsibility after thousands of "I told you...".

          The beancounter department knows some "good timing financial wise" as well.

          And those who want a new printer/laptop/whatever but the old one works perfect and it good enough but doesn't like cola.

      2. John Brown (no body) Silver badge

        Re: Software RAID

        "some flaky controller that will eventually go EOL."

        And we probably all remember the days of the RAID controllers that stored the details of the array config only on the card so when the card failed you were totally FUBARed because no one knew or bothered to document the config. And the client probably cheaped out on a proper backup because...RAID :-(

    3. Doctor Syntax Silver badge

      Re: Software RAID

      By way of contrast a client of mine had all the disks (a lot because back then disk sizes were small) mirrored at the controller and then mirrored the mirrored pairs in S/W. 4 physical drives for each logical drive.

      We didn't have disk failures but the tape drive for the overnight backups need changing at least a couple of times while I was on that gig.

      1. Anonymous Coward
        Anonymous Coward

        Re: Software RAID

        "the tape drive for the overnight backups need changing at least a couple of times"

        Don't tell me... they only supplied one tape

        1. jake Silver badge

          Re: Software RAID

          It was the drive that failed occasionally, not the tape.

    4. Jou (Mxyzptlk) Silver badge

      Re: Software RAID

      It depends on the quality-level of RAID you need.

      For private: Software. Be independent, done.

      For Business: Hardware. Reason: Servers and storages, all of those which are ACTUALLY business level, have an "general alarm" light, and a "SSD/HDD alarm" light for every drive in the RAID. And they are set up to send a "General problem, please act" mail or are watched in monitoring. Which everyone does since there are no lazy or bad admins anywhere. Second reason: Liability.

      1. Sudosu Bronze badge

        Re: Software RAID

        I have lost count of the number of drives I have seen in server rooms with an issue light where the comment was "its been like that for a year".

        Some enterprise systems actually do use software ZFS Raid (NetApp)

        They just build a different GUI and some custom enclosures with lights.

        They can also send emails on disk or pool issues or just about anything else you can alert on with a UNIX\Linux host.

        Check into write hole inconsistency for standard RAID models which was part of the reason I moved to ZFS many years ago for my personal storage.

        Second reason: 100% (also a reason to hire contractors from my experience)

        1. Jou (Mxyzptlk) Silver badge

          Re: Software RAID

          > I have lost count of the number of drives I have seen in server rooms with an issue light where the comment was "its been like that for a year".

          Yeah, but ignorance is not YOUR fault. But you do as I do: "There is a warning light, something is defective, must be looked after or else there will be a total stop for several days combined with very high cost". That gets them into action, up to now EVERY time. My warnings were never ignored, and they listen when I tell them what to look out for. And surprisingly, they do.

          As for Netapp: They talk home. Any drive with pre-fail or fail condition? Automatically a replacement is sent, alarm-mail at customer and the IT-company responsible for the customer. Monitoring is very important, and a low tolerance on "reallocated sectors".

          As for my home stuff: Any change in drive parameters sends me an email. So I know about two or three weeks or month in advance when a drive might fail. Sending those smart-infos with date, along with the drive, and no questions are asked. Drive is always replaced. On the other hand: I have quite a bunch of quite old 3 TB and 4 TB drives in my main machine as Storage Space Stripesets (aka RAID0-like), and they are perfectly fine. Without combining them they would be to slow to be bearable, those 3 TB drives can be as slow as 35 MB/s near the middle of the spindle. But combined I get three times measured throughput, and random access under load is even more than three times faster.

          Write-hole inconsistency is the reason why Storage Spaces beats classical RAID, part reason is due to using erasure encoding instead of simple parity. And why typical hardware storages have buffered RAM/buffered cache which is written to a flash memory with the power of a few large caps to continue where it was interrupted once the power it back.

          1. Peter Gathercole Silver badge

            Re: Software RAID

            Unfortunately, there are some disk systems with vendor support where the maintenance people know there is significant resilience built in the system, and so refuse to supply replacement disks or support until more than a certain number of disks have failed (IBM XIVs were like this, as were the GPFS Native Raid implementations - both systems with distributed RAID and auto heal capabilities that kept access and performance up to spec. even when there were failed dsks).

            Makes for difficult some difficult questions from managers when there are alerts for single disk failures visible on the monitoring systems, and warning lights on the hardware, but you're not able to raise support calls to get them addressed.

        2. John Brown (no body) Silver badge

          Re: Software RAID

          "I have lost count of the number of drives I have seen in server rooms with an issue light where the comment was "its been like that for a year"."

          I had that issue at a small client site. No on-site techies at all, they phone in to HQ if there was an issue. Anyway HQ calls our company and I get sent out, I get there and see the two red flashing LEDs on the 3-drive array and ask when they started flashing. Answer, from the non-technical user: "Well, the one on the left has always flashed since I started here two years ago. The other one started flashing yesterday afternoon and that's when everything stopped working." This is far enough back in time that not only did not everyone in an office automatically get a PC, but new starters, experienced or not, may have never used a computer before, so I definitely didn't blame the poor user who'd not been told what to look out for. (The server was on top of the filing cabinet by her desk) She was quite upset when I explained what had happened, but reassured her that she wasn't to blame for not knowing how the system worked if no one had told her.

          On another site, I was called in after hours to replace a failed drive in another 3-drive array. This time, the non-techy security guard took me to the offending beast and even he noticed that a helpful user had pulled the "faulty" drive and there were now two flashing red LEDs where there ought to be only one.

          But, in a server room, there's no excuse. People working in there are SUPPOSED to KNOW :-)

      2. Silver badge

        Re: Software RAID

        Software RAID can have the same safeguards, if not better since you can have more control of the system. I run weekly scrubs with emailed results, have automated alert emails set up with zed for any ZFS weirdness, and have SMART monitoring with smartmontools for disk health. I set up custom SNMP alarms on the BMC and anything that sends an email also sends SNMP to the management port, so any drive or array issues start blinken the lights. (I put the hardware RAID controller in IT mode for ZSH/speed increase, so it no longer sends its own SMART alerts.)

        The only part I really agree with is the liability. If you're paying for support from a vendor, then there's no reason to not hold them accountable if there is a problem. And no matter how much I would otherwise preach for homegrown solutions, it doesn't work for all businesses.

        1. Jou (Mxyzptlk) Silver badge

          Re: Software RAID

          I hope you and five others know the intimate details of that setup or at least is is very easy tracable and well documented. If not: In a professional environment a single point of failure, in that case you, is not good planing. If you drive a car it is already reason enough, any other car may crash you, you car mail fail weirdly, trees may fall, you may fail etc etc...

          1. Silver badge

            Re: Software RAID

            That was my personal setup I was commentating. Of course if it were for work it would be heavily redundant. Easy enough to have two identical servers as hosts, so the pool can be easily failed over for updates and test/prod staging. Or there's always the classic with a clustered filesystem for maximum redundancy and even cross-DC networking.

  4. Anonymous Coward
    Anonymous Coward

    We have a management system, hosted on AWS. It manages a few hundred machines, and has various scripts it can run on this machines to perform various functions. One script that I wrote zips up several logs on the machine it runs them on, then "attaches" them to the machine record on the server, where they are available as downloads.

    I developed this script, but due to structural changes in my team, never used it, and never really even tested it properly. It stayed on the server, largely forgotten about by me until our server started crashing. The hosting company said that the attachments table for the database had reached it's quota (several gigabytes), and they blamed this script.. The script needed to be triggered by a system admin, as it was triggered by a certain event that could not occur normally (basically, using the agent software, we had to trigger the event manually, and it had a custom name). The other admins denied all knowledge of the script, even though they could see it (and how to call it) in their admin consoles.

    Somehow, this script, despite generating and uploading gzip files that were around 100k on each run had filled up the attachments table.

    When I was notified of this, I disabled the script, and the hosting company purged the attachments table, but it kept happening. Eventually, I got the hosting company to delete the script, and purge the table. This is when it settled down.

    I actually believe the other sys admins when they say they didn't use it. They wouldn't have needed to use it 10s of thousands of times in a few months, we only have about 300 computers on the system and don't get tens of thousands of calls.

    I think what happened is that I was testing the script, something went wrong, and the system did not register it had run the script, so did so, repeatedly. But because it wasn't reporting this on the admin console I have access to, and made the logs available for download (as it should have), I didn't know this, and thought it had completed sucessfully.

    Thankfully, the hosting company has since agreed to give us a test instance of the server, so any future development can be properly tested before it goes to the live server. This is good, actually, because I need to write a few scripts we can run locally to automated some aspects of the management. We have the required API access for this, but I'm hesitant to let scripts I've written API access to the live data without rigorous testing.

  5. Caver_Dave Silver badge

    On the flip side

    Back in the mid/late 1980's my boss was asked to produce something for the Ferrari F1 team. (We did lots of techy things for the teams.) He decided on a TMS320 based solution (fastest DSP at the time), that I (as the only employee of the company - the boss worked for another shell) would design the hardware and software for. The problem was that the official TI assembler was about the same price as my yearly wage.

    So, I had to write my own macro-assembler for the TMS320. It was a great learning experience and meant I knew the chip inside out.

    The hardware struggled a little with noise on the dual layer PCB as 30MHz clock was quite fast at the time, but this turned out to be from the radio producing company next door! (It worked perfectly on a Sunday morning!)

    I later rewrote a Javascript interpreter that was taking too many time and memory resources on a 4MHz 8086 Internet set-top box where memory was used for screen refresh for 50% of the time.

    Getting "deep and dirty" is something everyone should do at least once to get a proper understanding of what is involved.

    (Like every driver should spend a night 'on the motorway cones' to get an appreciation of that. I helped change the contraflow at the M2 jnct 5 one night. It is bloody scary standing next to cars doing 80+ in the 50mph limits imposed in such situations! - obviously pre-speed camera days.)

    1. Anonymous Coward
      Anonymous Coward

      Re: On the flip side

      I was once chewing the fat with the heat of IT about why some of my co-workers didn't understand why what they were trying to do wouldn't work. The boss pointed out I was the only person who had any qualification in IT (I had a degree - a third rate one from a third rate uni - but it still counts as a degree). Everyone else had fallen into IT from very different carears (including the boss!) so they didn't have the holistic view of how a computer worked.

      To fully appreciate your world, you have to understand how the areas around you work. It's so easy to design the perfect system that no-one uses because you don't understand the context in which your system works.

      1. Anonymous Coward
        Anonymous Coward

        Re: On the flip side

        I used to rant at organisations that thought a few short courses in specific tools or skills could replace formal study (degree or apprenticeship, as appropriate) - and I've only stopped ranting because I've retired!

        The short course can teach the what and how, but falls short on the why. In my former profession I saw numerous instances of people trying to make a problem fit their solution. A bit like the apprentice joiner who, during his first week and having been handed a hammer, sees all problems as nails - a screw can be driven into wood if hit hard enough, but it won't stay secure for long. The first step of my career was after a BSc (in a relevant subject for that first position) but I embarked on an MSc just over 20 years later as I'd migrated into a different field. I had been approached by a local university to lecture in it but realised I had got to my current position through on-the-job learning - I declined the request to lecture and enrolled on the MSc course at another university. Those three years of study put my 20 years of practice into theory - it put context and a better understanding of why some things worked better than others and, I believe, made me far better at my job.

        When, as part of my latter career, I was advising organisations on staff competence matters, I defined competence as comprising three elements:

        1) An understanding of the theory (often best gained through formal/academic study);

        2) An appropriate skill set (often best gained through on-the-job training or specialist courses); and

        3) Experience (qualified by the experience being successful - and a demonstration of applied lessons from failures).

        Omit one part at your peril...

        1. Elongated Muskrat Silver badge

          Re: On the flip side

          Formal study isn't always necessary. I taught myself Z80 assembly language from books as a pre-teen, and my degrees are in the sciences, not in anything IT related.

          The thing that is needed is the drive to learn, and the curiosity to dig a bit deeper to gain a full understanding of things.

          I've worked with developers who don't know what two's complement is, and don't even own a computer at home. I often wonder what has drawn individuals like this into the field.

          1. BOFH in Training

            Re: On the flip side

            I have known too many people who got the paper (diploma / degree) in CS, but hated CS after a while cos things change too fast for them to cope with (regardless hardware, programming [languages / frameworks / IDE / etc] or just the general threadmill of keeping up with whats going on within your scope). They got it cos it sounded like a good way to make decent money, not cos they were interested in computing in any form.

            I rather take someone who has an interest and migrated from another industry / self taught then someone who may have the paper but not the interest. No doubt there are many who got the paper because they got the interest, but not all of the paper holders are that.

            1. Elongated Muskrat Silver badge

              Re: On the flip side

              Yeah, it's almost as if you can't draw complex conclusions about an individual from a single yes/no variable. This is despite the plethora of magical thinking that seems to be going on around us these days.

          2. TSM

            Re: On the flip side

            > I taught myself Z80 assembly language from books as a pre-teen, and my degrees are in the sciences, not in anything IT related.

            (Almost) same [I was a teenager by the time I got into Z80 assembler] and same! I did do a computer science unit in first year but that is the extent of my formal CS education. Like you said, it's the desire to understand that is needed to get you that habit of looking at context above and below and around the thing you're doing and making sure you understand it.

      2. Yorick Hunt Silver badge

        Re: On the flip side

        Never be afraid to ask end users (the less skilled the better) into your office to try to break the software you're writing. I swore by this back when I was writing point-of-sale software and always received praise for its user-friendliness.

        1. wolfetone Silver badge

          Re: On the flip side

          This - although it requires a thick skin.

          I've always developed software with my parents in mind as neither can really work a remote control let alone anything else. So I've become good at making something as simple as possible. What happens though is that you can only make something so simple, so when you come up against someone who is a proper idiot who just doesn't want to understand something (or at least is so full of fear that it stops them wanting to understand it) problems occur.

          Like last week I released software to the company I work for that I'd been working on for 14 months. It'd been tested by numerous people and all gave the thumbs up. It goes live and one of those testing has now started stamping their feet saying it isn't fit for their use. Through talking to them it appears they either didn't test it or just plain don't understand the concept of the software. When you hit people like this it makes it easy to become so fucked off with the world that it follows you the whole day. If I had met this person 10 years ago I'd have been a delivery driver for Ocado now. But I've learnt to take the punches and work the problem. Everyone can do that, but it needs to be learned and it can only be learned by doing what we do and accepting that no matter how simple we make something, it will always be too complicated for someone who can't or doesn't want to understand it.

          1. Peter Gathercole Silver badge

            Re: On the flip side

            So you point to their signature on the acceptance test documentation, and ask them to request a feature change in the next release to cope with what they see as a deficiency.

            And if they still cause trouble, you get the management involved to help wield the clue bat.

        2. probgoblin

          Re: On the flip side

          Whenever you design/implement anything, you should consider the end user as just as important a stakeholder as the actual owner. It's the only way to get to an actual user centric experience.

      3. Anonymous Coward
        Anonymous Coward

        Re: On the flip side

        Totally agree that everyone should have an understanding of the platforms & associated infrastructure - however I'm in a large company with may graduates & professionals.

        Many of the graduates coming through now have NO idea what they are working on - just how to 'code' on their trendy mac's & then don't understand why their code with all sorts of 3rd party libraries doesn't work in a locked down production Linux environment.

        As for the professionals - a number are so focused on their own subject matter they have no interest in other areas making complex trouble shooting a bit more interesting...

      4. Keith Langmead

        Re: On the flip side

        Absolutely. It helps being fairly old (24 years working in IT), so many of the core concepts people overlook today were fairly new and often talked about when I first got into computers at college.

        It's so much easier to understand what's happening in front of you, when you understand what's going on behind the scenes. I must have been half way through my career when I properly learnt the binary maths behind IP addresses, subnet masks etc. Suddenly understanding WHY a subnet is the way it is, rather than relying on remembering the 255 or / notation by rote with no real grasp of what it really meant.

    2. GlenP Silver badge

      Re: On the flip side

      Getting "deep and dirty" is something everyone should do at least once to get a proper understanding of what is involved.

      It was something they were very keen on at Newcastle Uni when I did my degree, even knowing that hand writing machine code wouldn't be something we'd likely need in our professional careers - an understanding of the fundamentals was considered a key skill.

      I met a student from a another, supposedly more prestigious, University and discovered their entire computing course seemed to consist of learning as many high level languages as possible with very little emphasis on understanding programming techniques.

      1. A Non e-mouse Silver badge

        Re: On the flip side

        I dabbled in assembler myself and learned some higher level languages. It was much later that I learned the complexities of how a high-level language is transformed into assembler. It makes be appreciate software engineering at both a high and low level way more than I did previously.

        One of my "When I'm retired and I have spare time" projects is to write a compiler for a trivially small/simple high level language.

        1. Doctor Syntax Silver badge

          Re: On the flip side

          "When I'm retired and I have spare time"

          There's a big surprise in store for you when you retire.

          1. jake Silver badge

            Re: On the flip side

            "There's a big surprise in store for you when you retire."


      2. jake Silver badge

        Re: On the flip side

        Round about 2000 I started seeing "programmers" with a newly minted college degree who didn't know what the heap and the stack are, much less how the compiler uses them. Nowadays it's normal for the youngsters to have many gaps of that nature in their education. I fear we are losing something very important that is going to prove to be almost impossible to get back.

        (In a similar vein, about five years ago I had a child CEO ask me why I'd have a need to thump a TV with a screwdriver, but then CEOs have always been clueless ... well, most of 'em anyway.)

    3. stylers

      Re: On the flip side

      @Caver_Dave would you happen to still have that macro-assembler ? I recently was looking for similar for the TMS320C25.. retro computing project jobbie.. but software seems to be unobtanium..

  6. Anonymous Coward
    Anonymous Coward

    vmware disks and oversubscription...

    I still can't tell this one...

  7. jimmythebrit

    That's a lovely piece of writing

    Professor with a hat icon, that's the closest I can get.

  8. FrogsAndChips Silver badge

    Indiana immediately owned up to … absolutely nothing

    Obviously, he didn't want to embark on a last crusade.

    1. Paratrooping Parrot

      Re: Indiana immediately owned up to … absolutely nothing

      Of course not, because otherwise he will have to do additional films, I mean jobs, for them down the line which will get worse, just like the last two films!

      1. Anonymous Anti-ANC South African Coward Bronze badge
        Thumb Up

        Re: Indiana immediately owned up to … absolutely nothing

        Indy and the Dial of Destiny's a delightful romp.

        Enjoyed it immensely.

        So much better than the Kingdom of the Crystal Skull.

        1. Sceptic Tank Silver badge
          Big Brother

          Re: Indiana immediately owned up to … absolutely nothing

          I heard some movie reviewer on YouTube call it The Dial of Dysentery – it was that bad.

          1. John Brown (no body) Silver badge

            Re: Indiana immediately owned up to … absolutely nothing

            But, like most things, it's actually relatively rare for anything to be universally derided or lauded. It's all just a matter of personal taste :-)

      2. KarMann Silver badge

        Re: Indiana immediately owned up to … absolutely nothing

        The last two? Temple of Doom wasn't great, but it wasn't that bad! And Last Crusade was lovely!

    2. Dave559 Silver badge

      Re: Indiana immediately owned up to … absolutely nothing

      Am I the only person who thought that "Indiana" was going to be a somewhat abstract reference to (depending on how long ago these events took place) SUN OS or Solaris?

    3. Bebu Silver badge

      Re: Indiana immediately owned up to … absolutely nothing

      《the last thing he wanted to do was actually go and look at the thing for himself. That's how you get your face melted off.》

      Nice touch equating the Ark of Convenant's face melting capability with this Indy confronting his past crimes against IT.

  9. A Non e-mouse Silver badge

    Not owning up to being the author of the original bodge was probably the right thing to do as it forced the company to finally fix the underlying issue.

    1. John Brown (no body) Silver badge
      Thumb Up

      There's nothing so permanent as a temporary fix...until it bite you!

  10. Peter Gathercole Silver badge

    Narrowly escaped a bullet

    Back when I worked in a support centre for one of the major computer suppliers, I did something that was, on reflection, a dangerous precedent.

    The company was bidding to replace one of the British Rail (that is how long ago it was) ticketing solutions, bidding against the current supplier. The bid team were quite desperate to sell a complete solution of hardware and software, including thousands of the company's own async. terminals, which used non ANSI X3.64 terminal sequences.

    These terminals were, to put it nicely, quirky (or in my mind, completely crap), and did not even have a complete terminfo entry for their native mode. In theory, they had all the features required, but using them was difficult. The one that the bid team were most wanting to use was scrolling regions. These terminals had the capability for a single scrolling region, but the system they were replacing used much more expensive terminals which allowed multiple scrolling regions to be set up and used separately. Desperate to find a solution the bid team came to us to help, and I ended up being told to assist.

    So I came up with demonstration code that provided functions that would accept text destined for one of the regions of the screen, and keep track of where the cursor in that region was (one feature missing was the terminal reporting the current cursor position, so it was necessary to keep track in the program), and then set the single scrolling window up to the correct region, restore the cursor to the last position, and write the message out. It was not complex code, but just illustrated how it could be done. In theory, for a single process/threaded solution, this should have been sufficient, but I was careful to say that as the writes of the buffer, including the setup and cursor restore sequences and the message, could not be guaranteed to be atomic, care should be taken to try to serialize the messages. It was also pointed out that this was only intended to be demonstration code to show a possible solution.

    Did the bid team listen? Of course not.

    What I wasn't told was that each region would be written to by a separate process, so despite my warning, the bid/implementation team included my code pretty much unchanged in their solution. The system as delivered passed the acceptance tests, but after several months and when the system started to be used in anger, the end users started to get corrupted screens, as the writes from different processes were intermixed.

    The customer came back to us in the support centre saying that the terminals were faulty, but fortunately we were able to go back to the part of the business that delivered the system, pointing out the original disclaimers (which included comments in the code itself detailing the limitations), so even though my name was against the code, they ended up having to pick up the pieces, although I did outline how they could fix it using a synchronising message server process.

    And the lesson leaned? Do not even provide demonstration code unless you either have the authority to promise support for it, or have cast-iron disclaimers understood by the people receiving the code. And also try to keep your name off of the code!

  11. Howard Sway Silver badge

    That script I wrote three years ago

    3 years is nothing. There were some of mine that I heard were still running somewhere 15 years later : fairly long winded data-munging stuff that ran every day as a cron job. Apparently it never got folded into their main workflow because the lifers there preferred to rewrite their systems every few years using the latest and greatest trend, and generally made such a mess of it and had so many problems to fix that they never had the time to get round to it.

  12. imanidiot Silver badge

    Why would you not send a CYA email in a situation like this?

    "I did such and such to make this work after explaining the situation to so and so. Agreement and permission was received from this person and the solution implemented but as outlined it is a far from ideal solution. I recommend returning to an proper RAID system as soon as possible. Data corruption from the system as set up currently is likely and irreparable. I take no responsibility for data corruption resulting from this implementation"

    If you ever get contacted again when it inevitably implodes in someones face, point to said CYA.

    1. Jou (Mxyzptlk) Silver badge

      > CYA

      That got renamed to CYB, Cover Your Bases, for reasons of political correctness principal.

      I guess you are non-US as I am? Then go on using CYA :D.

      1. jake Silver badge

        "I guess you are non-US as I am? Then go on using CYA"

        I'm in the US and use CYA fairly often. I have never even had anybody blink at me funny for using it.

        Don't believe everything you see on TehIntraWebTubes, nor things on Deal Old Telly ... unless you enjoy coming off as a completely un-traveled xenophobe.

        1. Roopee Bronze badge

          Ah, but Jake, you are a grumpy old man, and they get treated differently!

          1. jake Silver badge

            I haven't always been old. I was using the phrase with impunity at least as early as Jr. High.

            Seriously, it hasn't been an issue in this country since (roughly) the mid 1950s. Probably earlier in military towns.

        2. Sherrie Ludwig

          CYA = Cover Your Accountability, for the politically correct.

  13. jaqian

    Robocopy to the rescue

    IMany years ago Iwas involved in setting up a new government quango, it had been announced at the last minute by the Minister (trying to score brownie points with the media) and nothing had been worked out lol.

    They had an office and a handful of desktops and a pc acting as file server. There was no raid and because they weren't part of our infrastructure they weren't getting archived to tape, so I used robocopy to copy all the files back to our office (HQ) where they were then archived to tape. It ran for a few years before they got themselves organised and were spun off from our department.

    1. Jou (Mxyzptlk) Silver badge

      Re: Robocopy to the rescue

      rsync is nice for such tasks too if your line is thin.

  14. Lee D Silver badge

    As someone who's taken over any amount of legacy junk, I now operate on this basis:

    - If it's junk, I'll tell you so, and expect you to replace it.

    - If you don't replace it, I have little interest.

    I've always left behind documentation, experienced staff, etc. whenever I've had to put in a bodge, but nobody seems to care very much.

    Every time I take over a place, nothing useful is documented (and documented the weird and wonderful, including the rationale for that, is far more important than telling me the exact spec of your server, or how you organise your IP subnets). In the last place I took over there was an entire community FM radio station hiding in a cupboard that nobody really knew anything about.

    So when I take over, as I touch things, I document them. I force staff to document stuff they know, especially if it's a "Oh, look, I'm the only one who remembers how this works, let me just fix it quickly for you" kind of arrangement.

    If it's not documented, it's not something I deal with. Not until it's been documented. Force me to take it over and job #1 is documenting it... for me and future replacements. And in that process, if you're documenting and thinking "WHWHWHWHHYYYY?!?!?" then you should be telling them they need to change it, and moving down that path, and accepting no responsibility for it failing in between that moment and the moment of replacement.

    And, yes, it's the weird stuff like this that absolutely NEEDS documenting. And why it exists. That backstory is important. Why didn't you just go the simple way? I need to know. There might be a very good reason for that.

    If you're not leaving behind a Wiki / Sharepoint / Whatever full of consistent documentation, with rationale, then you've failed in your job managing that system. If the next guy has to pick up the pieces and either doesn't know it's there, only discovers it later (or when it goes wrong) and has no idea why it's doing that, you've failed in your documentation AND handover.

    It really doesn't take that long to spin up a Wiki and start bashing out a page for each piece of software, a page for each server/VM, a page for each system and its dependencies, a page for renewals and expiries, a page for suppliers, licensing, etc. and you can literally do it as you go. Every time you have to do something, think "Is that documented?" and if it's not make a blank page with a TODO message on it (I use MedaWiki and a category for "Incomplete Pages"). Then when it's quiet or you have even a couple of minutes, grab a random unfinished page and add to it. You don't need to complete it, just add to it. Every time you do that weird thing you have to do every year but always forget how to do... write a page on it.

    And at a pace of 1 small page a day, you have over 300 pages of documentation done in a year just as one guy. 300 systems, servers, quirks, softwares, etc. It's really not hard. With a team, you can blat out thousands of pages.

    And then it all pays back that one day you have to handover, train a new member of staff, or disaster strikes. "Now... how did we build that cluster, I remember we had to do something with the registry to make the RAID controller work, what was it again..." "Oh look, there it is. All written down by someone like me who accounted for all the pitfalls in the process, and knows exactly what I need to know and what order to do things in, and why we DON'T do that step first even though it appears logical to me."

    Past me is a really nice guy who helps me out a lot, and is psychic - he always knows what I'll be asking when something goes wrong, and telling me to ignore THAT menu even though it looks tempting because the option I actually want is OVER THERE instead with a similarly-named option. He's a smart guy.

    Past-predecessors are almost universally a bunch of inconsiderate twats.

    1. TSM

      That's fine, if (a) you can find the relevant documentation when you need it and (b) the systems haven't changed in the meantime to the point that the documentation is somewhere from slightly inaccurate to actively misleading. Obsolete documentation is (in my experience) both a much worse and a much more pervasive problem.

      Had this happen recently when we had to invoke a DR process (which fortunately I knew) that we hadn't needed in several years. [I know, TRWTF is not practising our DR processes regularly. TRRWTF is that we used to do this, but don't any more.] We did eventually find some documentation a few days after the fact, but it was largely inaccurate.

      It doesn't help if you change your documentation platform every now and then, and in between reorganise the IT teams periodically so that half of the documentation you need is now in some other team's area. Even apart from those issues, when you have something that involves multiple departments it's impossible to find whether the latest thing is the one on Confluence or one of the ones on Sharepoint or the ones in various Teams channels or the one in an email chain, because everyone has their own ideas on where to put stuff...

  15. Anonymous Coward
    Anonymous Coward

    Can't blame Indie

    As, after 3 years, who can trace back the data coherency after so many random updates to various different DBs !

    Better to play it mystery.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like