back to article Firewalls? Pfft – it's no match for my mighty spares-bin PC

Start your week with a warning about those temporary emergency hacks that all too often end up permanent in today's edition of Who, Me? Our story comes from "Gary" - for that is not his name - who worked for a firm that offered a variety of services to the UK banking sector. Back at the turn of the century, our hero was tasked …

  1. KittenHuffer Silver badge
    FAIL

    I could post my history of 'temporary' bodges .....

    ..... but I do need to get some w**k done this week!

    I seem to have spent much of my w**king life w**king for businesses that would prefer not to spend money on a proper solution. But are willing to accept a cobbled together 'temporary' bodge that only ends up costing them more in the long run.

    1. hutchism

      Re: I could post my history of 'temporary' bodges .....

      Don't w**k too hard....

      1. KittenHuffer Silver badge

        Re: I could post my history of 'temporary' bodges .....

        I always w**k furiously!

        1. Graham Dawson

          Re: I could post my history of 'temporary' bodges .....

          I've heard it said that a man who turns his most enjoyable hobby into his w**k will never w**k a day in his life.

          Sounds like a load of of old billhooks to me.

          1. Doctor Syntax Silver badge

            Re: I could post my history of 'temporary' bodges .....

            Or ruins his hobby.

    2. gryphon

      Re: I could post my history of 'temporary' bodges .....

      Short term fixes rarely are.

    3. Juha Meriluoto

      Re: I could post my history of 'temporary' bodges .....

      There's never time to do it properly, but always time to do it again... and again... and...

  2. Anonymous Custard Silver badge
    Headmaster

    Basic rules of engineering

    As the old mantra goes, there's nothing more permanent than a temporary fix...

    1. slimshady76

      Re: Basic rules of engineering

      Down here we call those "provisory for ever" installments...

    2. A.P. Veening Silver badge

      Re: Basic rules of engineering

      here's nothing more permanent than a temporary fix...

      Except for a temporary tax hike.

  3. Dave Null

    You can't fix that...

    "...that's *load bearing* technical debt"

    1. Anonymous Coward
      Anonymous Coward

      Re: You can't fix that...

      The key is to ensure there is a solid foundation of cynicism and sarcasm to support the load bearing technical debt and plenty of liquid maintenance to ensure the whole thing stays in one piece.

      I prefer beer for my maintenance...

  4. Anonymous Coward
    Anonymous Coward

    So relatable, I remember cobbling together a firewall and FTP server (FTP! ... so a very long time ago)

    This was also an old desktop PC, spare NIC and good ole Linux

    This was a "stop gap" solution, only for a few weeks. ha!

    My story ended up happier though, after 6 months I was able to convince the then MD that this had become "mission critical" and in fairness they invested in a hardware based firewall and a rack mount server for the FTP services

    This was a medium sized business turning over nothing like a bank, I guess the larger the organisation the harder it is to speak to someone with the hope of changing things

    1. Gazzat5

      Bodged rack mount server

      Reminds me of a bodged up server I built while working at a high school. We need a sever to run a single app (SIMS, the school database interface) via remote desktop for electronic registration due to the separate computer networks for admin and students. So we dug out a copy of server 2003 (I believe it was the enterprise flavor) and builtba server with SCSI disks in raid 5 configuration. All was well until I realised the order of the disk IDs didn't aligh with the order of the physical drives and my boss was a bit anal about things so we decided to scrub it and rebuild it in the correct order in case a drive failed in the future and a drive needed replacing at least we'd know which one it was.

      Anyway when I went to rebuild the SCSI array, it never came up again so in the end we scratched the SCSI interface and stuck a single desktop IDE HDD in the single interface the machine had and built the server off that. The server had 32GB of ram (a hell of a lot in those days) to support all the instances of SIMS it would run (one for each teacher in the school during registration) and am IDE hard drive to load the software and OS off. It worked but it wasn't pretty. The drive was hanging loose in what would've been a cd bay lol.

      1. ecofeco Silver badge

        Re: Bodged rack mount server

        I DO NOT miss SCSI.

        1. jake Silver badge

          Re: Bodged rack mount server

          Out of curiosity, what drives do you prefer for your private use?

          Yes, this is a trick question. We may learn why s/he doesn't miss SCSI.

        2. Down not across Silver badge

          Re: Bodged rack mount server

          I like SCSI. Mostly. As long as you don't accidentally mix LVD and HVD. Use of proper active terminators is a must. Internal termination (if offered) on most devices might work if if you only have couple of targets, but I'd avoid it and use decent terminator instead.

          It can also get messy when it comes to the time to perform the obligatory sacrifices.

          1. richdin

            Re: Bodged rack mount server

            I once had a situation with a PC server using 2 mirrored SCSI adapters running Netware on a token-ring network that brought down the entire corporation including the mainframe because of a magic app running a btrieve database...

            Shows how old I am... and what an a** saver the NSE was (pre-internet!).

            SCSI=SCARY

      2. wardster

        Re: Bodged rack mount server

        I wish we had a Willy Wonka Icon. In other words - "Really? Tell me more!"

    2. Terry 6 Silver badge

      I wondered about this. I've long had an impression that the higher the corporate pyramid the harder it is to get a decision on things that are actually in the remit of the middle managers. And also that this isn't their fault, but of a system that will penalise them for any cost it sees, but not reward them for any cost saving from bad stuff that's not allowed to happen.

      1. Nick Ryan Silver badge

        Tell me about it...

        In a previous life I had to fight insanely to get a back up system approved. Nothing too fancy, just a basic, working back up system for all of the office server content.

        It was rejected multiple times because of the cost, which was no more than few thousand (it's amazing what one can do on a budget) and because they didn't see the value in something that they wouldn't immediately use.

        The next proposal I included the approximate salary of all employees, applied a few metrics to the amount of "work" that they do that involved the shared server data and extrapolated this out into the value of the "work" stored on the office servers over a short of period of time and compared this to the cost of the backup system.

        Rejected, again, of course but this time with the added threat of an investigation into my estimation of the staff salaries and how this was a privacy violation! A privacy violation based on guessing average salaries based on recruitment adverts...

        About a year later two drives on the central file server crapped themselves within a day of each other taking down the entire array and preventing any work from happening in the company. We weren't allowed to keep spares of these and therefore while we immediately ordered a replacement for the first failure, before it could even be delivered the second drive failed.

        The backup was approved the same day... a day too late of course.

        1. Anonymous Coward
          Anonymous Coward

          I had this working for an educational service. Hundreds of kids' files that were potentially evidential if there were any kind of legal challenge against the LA. But no one was interested in providing for any kind of secure backup. For many years the only backups we had on or off site were ones I took myself (illegal at first because in those early days, not encrypted) with floppy discs and later CDs.

          It was only long after there'd been incidents elsewhere that the idea slowly percolated through to them, with frequent reminders from me, to provide an automatic backup, in a (relatively) secure cabinet in another room. When I retired another decade later they were still umming and ahhing about some kind of off-site backup.

  5. Anonymous Coward
    Anonymous Coward

    That reminds me...

    Of the time I built a quick and dirty Oracle DB server for the devs on a project (story I've told before, but bears repeating...) to get them out of a delivery delay bind. Making it extremely clear, in writing, that the Fujitsu desktop PC with a single PII-350 (it was quite a while ago), 4 GB RAM and non-resilient IDE hard drive (and no backups) was for development of their stuff only, I let them get on with it. As you would already have guessed time passed, I forgot all about the PC, and about a year later I was contacted by the programme manager wanting me to look into improving the performance of the "production database server as it was just so horribly slow".

    1. Piro Silver badge

      Re: That reminds me...

      This is an odd machine. Pentium II with 4GB of RAM?

      I never have heard of such a thing. Is there even a chipset that supports that combination?

      It would have been in any case, unbelievable, and I refuse to believe that any desktop PC could have had space for the number of RAM slots/sticks needed.

      1. Lord Elpuss Silver badge

        Re: That reminds me...

        Wasn't it something like 512MB max for a PII? I know some chipsets could address 4GB in a server config but only 'see' 512MB of it at any given time - or something like that. Was a long time ago...

      2. Anonymous Coward
        Anonymous Coward

        Re: That reminds me...

        https://en.wikipedia.org/wiki/List_of_Intel_chipsets#Pentium_Pro/II/III_chipsets would suggest the 450GX and 450NX supported up to 8GB. Granted, it would probably be a special kind of desktop PC to use one.

      3. Anonymous Coward
        Anonymous Coward

        Re: That reminds me...

        Could he be referring to a massive 4GB hard disk drive?

        1. Captain Scarlet Silver badge

          Re: That reminds me...

          Thats my guess, I'm assuming just a mistake

  6. Sequin

    I once worked as the IT manager at a tourist attraction, and was given a miniscule budget to keep the back office and point of sale systems working. After several staff losses, I also took on the reponsibility of keeeping the Audio visua systems running for the punters. I did this on the cheap, scrapping obsolete VHS machines and replacing them with 20 quid DVD players from ASDA.

    I used several old PCs from home for vital tasks, and an old Epson dot matrix of mine was used to print everyone's payslips. One day I was called in to see the boss to be told that I was being made redundant and should pack up my things and leave the building, serving out my notice on gardening leave.

    At this point I requested the return of all of my equipment, with the exception of a 22inch Dell CRT monitor that I personally gifted to a lovely colleague with very poor eyesight.

    Panic ensued and I cared not a jot - most of the kit I dropped off at the tip on my way home.

    1. Ken G Bronze badge

      Why didn't you either resell the equipment to your employer or contract yourself as a service to them? As IT manager you shouldn't have exposed them to that concentration of risk.

      1. Anonymous Coward
        Anonymous Coward

        Because at that point he was no longer the IT manager, and couldn't reasonably be expected to give a toss

        1. Ken G Bronze badge
          Holmes

          I mean while he was IT manager and setting this up, could he not have mitigated the risk to his employer?

          If it was not himself but a 3rd party asking the company to run critical systems on loaner kit with no support and no exit plan would he, in his professional capacity, have thought it was a good idea?

          1. Sequin

            You mean the employer thhat was using kit so old that spares had to be sourced from eBay? The ones using a Windows95 pc with file sharing set up as the host of the main POS database? The ones who bought a job lot of redundant kit from a closed down care home for £250 to find that it was 10 times better than the kit they had?

            It was a case of do anything to keep the business running and keep 50 people in work!

        2. Doctor Syntax Silver badge

          I take your point but there's an old adage "Don't get mad, get even". And one way of getting even is to get the money.

      2. Jimmy2Cows Silver badge

        Employer should've provided adequate budget to do the job, instead of leaving it to the IT manager to bring in all their own equipment. There's only one group to blame here, and it not IT.

    2. Anonymous Coward
      Anonymous Coward

      I've done that. "You're no longer needed". Ok, uninstalls all my software, removes all my data, loads car with all my equipment, locks up, pops key through letterbox.

      1. The Oncoming Scorn Silver badge
        Coat

        Last Men Standing!

        Out of our team, only two of us remained.

        We had tried a smooth transfer of knowledge & procedures to our field based colleagues, who were absorbing the back office, general support, asset tracking into their roles & dragged in to Corporate HQ.

        They didn't want to know, made all sorts of excuses of "Too busy", everything me & the other guy had scripted & tweaked to simplify the post build process into a mostly autonomous task ignored.

        At 5pm on the penultimate day, we nuked the lot, just in case of unexpected account deletions at 8am the last day,

        The last day we terminated our accounts, wiped our machines & left at lunchtime.

        1. John 110

          Re: Last Men Standing!

          As the only IT support with the necessary Lab experience to manage the IT effectively, I gave the department 2 years notice that I was thinking of retiring and a year to say I was going and that they should maybe give me back the one enthusiastic staff member who was at all interested in doing dual IT and Microbiology. (he had been taken away when the Lab Manager went off sick and the deputy (who hated "they computer thingys") grabbed my apprentice back.)

          This was ignored by the Lab Manager.

          A month before I went, the clinical director stopped me in the corridor and asked who was to replace me. I shrugged and said that she'd have to ask the Lab Manager about that, but that I knew that the others in my lab team (in a different discipline, but also tasked with IT) would help out where necessary.

          Cue large explosion in the management team.

          I did leave a textfile with hints...

    3. EarthDog

      So basically the conversation should go like this:

      You: we need hardware and software for a mission critical reasons.

      Manager: oh, we don't have a budget for that.

      You: I can lash together something from my personal kit.

      Manager: Great!

      You: Since it is my kit I'll have to have you sign a contract where you indemnify me and hold me blameless and a monthly lease of BIGNUM money.

      Manager: How much money do you need in that budget?

    4. IHateWearingATie
      Devil

      This feels like it was a missed opportunity for an ..... enhanced...... redundancy package, as well as a lucrative ongoing support contract.

      The BOFH would never have let this one slide.

    5. aerogems Silver badge
      Devil

      A True BOFH

      This would honestly make for an excellent base idea for a BOFH episode. BOFH and PFY personally pay for some new mission critical equipment, boss then fires them, only to be forced to pay truly extortionate consulting rates for their continued "services."

      1. arachnoid2

        Re: A True BOFH

        An "internet box" care of The IT crowd

        https://www.youtube.com/watch?v=iDbyYGrswtg

  7. jake Silver badge

    Eh?

    "how I don't miss LILO and needing to rerun it for kernel upgrades"

    Because it was such a chore?

    MAybe a dozen or so keystrokes ... you must have been typing your fingers to the bone, poor thing.

    1. Lord Elpuss Silver badge

      Re: Eh?

      Without having any further knowledge of his config, you're not qualified to make that statement.

      1. jake Silver badge

        Re: Eh?

        I don't need knowledge of his config. I know lilo. Changing the kernel[0] called for in /etc/lilo.conf and then running (surprise!) lilo at the command line and rebooting is hardly difficult.

        It would appear some folks disagree.

        [0] The kernel was the only thing upgraded in the OP's scenario, so the only thing needing to be changed would be the image called out by the config file (and possibly a kernel parameter or two). And probably just the one kernel, even in a multi-boot system. Unless you're foolhardy enough to change all the kernels in such a system at once, sans testing.

        1. John Robson Silver badge

          Re: Eh?

          "rebooting is hardly difficult."

          Yeah - that's the bit he probably wasn't able to do.

  8. Anonymous Coward
    Anonymous Coward

    This is every IT project

    There’s always sellotape involved in any project. I don’t even see it as a bad thing anymore. First you get it to work, then you come back and fix…

    …ideally you fix in a controlled fashion, but sometimes the system makes the decisions for you.

    Let’s face it: this was just the hack that failed. I’m sure there are dozens of other hacks in that deployment that never caused an issue and were never fixed.

    1. Admiral Grace Hopper

      Re: This is every IT project

      "There’s always sellotape involved in any project."

      Please, we're professionals, we use duct tape.

      1. jake Silver badge

        Re: This is every IT project

        Gaffer tape ... don't leave home without it.

        1. WonkoTheSane

          Re: This is every IT project

          Also a shovel, a roll of old carpet, and a bag of quicklime.

        2. Stoneshop Silver badge
          Devil

          Gaffer tape ... don't leave home without it.

          Only the best: Nichiban.

          But if the beancounters can't be swayed you just use a couple of miles of the cheapest tape around. It's usually good enough to immobilise them while you assign a proper budget to your own 'cost centre'.

          1. jake Silver badge

            Re: Gaffer tape ... don't leave home without it.

            For all around use, ShurTape P- 628 will handle just about anything you throw at it. I generally buy it by the case when I see it on sale. I also keep various versions of ProGaff on hand for specialty work (wet conditions, automotive/boat upholstery backing, etc.). Both are kinda spendy, but you get what you pay for.

      2. Ken G Bronze badge

        Re: This is every IT project

        Only to shut up people suggesting go-live of an unsupportable configuration.

    2. Will Godfrey Silver badge
      Linux

      Re: This is every IT project

      It's not just IT.

      One of 'our' guys knocked up a temporary replacement for an interface module on a cardboard box maker that had been 'reconfigured' by a fork-lift driver. That was about 2009-2010. To the best of my knowledge it's still there.

      It uses one of the original Arduinos

  9. Fursty Ferret

    I think the big mistake here is offering his former colleagues help on the phone call, other than just a bit of fake sympathy. All this has served to do is reinforce to the company that employed him that you can extract useful work from people and then kick them out, knowing that they'll still help out for nothing if it all goes tits up.

    1. Doctor Syntax Silver badge

      No. Always be prepared to go back for suitable money.

      1. stiine Silver badge

        I think you misspelled suitcase.

      2. jake Silver badge

        For large values of "suitable:

        Remember, they need you, not vice versa. Negotiate accordingly.

  10. Diogenes

    Wrote a q&d in 1991

    Wrote a quick and dirty in 1991 specced to run on Intel 286 machines that did all the things its mission critical bigger brother did and a lot more. I was made redundant in 2003, and heard that it was only decommissioned in 2018 along with mission critical big brother.

  11. Kildare
    Linux

    Guess what I found!

    When I took on the role of Network Manager at a large university hospital just beforeY2K. Amongst the other horrors i discovered was an old (even then) 386 desktop with no monitor or keyboard sitting on a desk in the main comms room. I was told that it was the DNS - I knew this to be wrong as there was another - rack mounted - box which was. No one knew the password for this box so there it sat, until my curiosity got the better of me.

    I disconnected it and waited for the call! - Didn't wait too long before I heard that people were complaing about not being able to access from the Uni with whom we had links. I connected a monitor and keyboard and found it was running some ancient version of RedHat (nothing but the best here!) , Not having the password, I rebooted the box in maintenance mode and set a new root password to get in.

    It was running Bind but it's sole purpoose was to act as a reverse lookup - i.e.provide the name from a given IP. I assume that this was someone's idea of security!

    A new zone file on the "real" DNS provided some extra space on the desk:)

    1. Anonymous Coward
      Anonymous Coward

      Re: Guess what I found!

      A packet sniffer on the wire would have been a better way of getting info on what it was doing

      1. jake Silver badge

        Re: Guess what I found!

        A promiscuous ethernet card and tcpdump worked quite nicely back then, and still is quite useful in the right hands.

      2. Gerhard den Hollander

        Re: Guess what I found!

        But honestly, isn't this way much more fun ?

    2. John Brown (no body) Silver badge
      Coat

      Re: Guess what I found!

      "A new zone file on the "real" DNS provided some extra space on the desk:)"

      It's ALWAYS DNS :-))

      (As facebook probably found out today too)

  12. ColinPa Silver badge

    executive assistance can sometimes be a good thing.

    Some executive assistance, such as hourly phone calls about a critical problem is not welcome. You spend more time on the call or the pre-call call (to make sure you did not say the wrong things to the executive).

    We had an executive planning to come to the UK and wanted to see a demo of our bright new shiny software. We were stuck trying to get two products to talk to each other, and could not fix it. In our call with the executive assistant who was setting up the visit, we mentioned the problem. Within 10 minutes of the end of the call we had a phone call from the top guru of the other product. Within another 10 minutes we were told which fix we needed, and what the bypass was.

    The executive visit was a success!

    1. wyatt

      Re: executive assistance can sometimes be a good thing.

      I had a fault on a bit of software that we supplied that I needed to escalate to our Development Team. Support was refused until it could be fitted into their 'sprints'.

      Once I managed to get them to look at it (by escalating internally), it took them <5mins to see the issue and about 1hr to fix it. To get to that point was months.

      1. EarthDog

        Re: executive assistance can sometimes be a good thing.

        IIRC Dilbert once referred to support teams as "Denial of Support Teams". Your issue will be ignored in a random manner.

  13. jake Silver badge

    Temporary hacks aren't.

    If you are reading this, you're probably using the hack that I put together in 4.1BSD (now called 4.1aBSD) for part of the TCP/IP stack to be included in 4.2BSD[0]. It was supposed to be one of those "just get us through the demo, dammit" hacks. I got 'er done over Christmas/NewYears break in 1981. Virtually every version of TCP/IP since has used it. Not too bad for a quick hack ...

    [0] Just to cut the usual pack of idiots putting words into my mouth off at the socks, no, I didn't write the whole stack. That's why I said "part of". It was only about 120 lines of C in total.

    1. IHateWearingATie

      Re: Temporary hacks aren't.

      "[0] Just to cut the usual pack of idiots putting words into my mouth off at the socks, no, I didn't write the whole stack. That's why I said "part of". It was only about 120 lines of C in total."

      Dammit, I was looking forward to that bun fight.

      Spoilsport.

  14. Anonymous Coward
    Anonymous Coward

    Not just in software. Many years ago a local bigwig had expanded his office into the adjoining garage. As a sparks I was roped in to wire it up. However, his office shared the leccy with the plumbers next door, and they would only agree to the power being off for a prior arranged 30 minutes at some weird time of day.

    So, wired up the extension, tested everything in situe, and prepared to patch into the supply. The 30 minute window opened, I patched in and tested, and everything passed. However, the linking cables passed over an area where a shelving unit was to be inserted, so I had to leave the cables and a temporary junction box which was to be replaced with a socket outlet, exposed there waiting for the shopfitter to finish off so I could then finish off.

    So I waited for an opportunity for the power to be off. and trim that last cable and put in the last socket box. And waited. And waited. About five years or so later I popped in for something and the temporary connection point was *still* there. The next time I passed the office it had been demolished!

    1. Steve Aubrey

      And somehow, somewhere, your name was associated with that demolition in a negative way. You just know it.

      1. Anonymous Coward
        Anonymous Coward

        At that point I would've happily and proudly signed my name on the bloody thing.

  15. Anonymous Coward
    Anonymous Coward

    In a previous life...

    Got called to a site of a major player in the flight/cockpit simulator business. Back in the day this thing was huge, at least 30 cabinets full of 'stuff'. It had been down for several days and the simulator engineers were stumped until they 'found' a PDP11/05 in an a cabinet they had never looked in before, and its 'RUN' light was off. Desperate searches for tech information were made to little avail except a roll of paper tape was discovered with the 'Digital' logo on it, with the hand written 'start at 1000'. I checked out the 11/05 and found DCLO was asserted, fitted a new PSU and bingo, it ran a keyed in loop. A Teletype was found and plugged into the 11/05, loaded a loader then the mysterious rather large tape. It took a good 10 minutes to load and when running the whole simulator started to come alive. I never did find out what the 11/05 did.

    1. Antonius_Prime
      Trollface

      Re: In a previous life...

      The 11/05 was probably acting as a counterweight for the rest of the kit, in case it tipped over...

    2. Zarno
      Mushroom

      Re: In a previous life...

      Hopefully not control the Therac-25 in the simulated medical bay!

    3. 2+2=5 Silver badge
      Thumb Up

      Re: In a previous life...

      > I never did find out what the 11/05 did.

      Judging by your description, it was an electronic cache of the code/data on the paper tape. :-)

      [Icon: thumb - there's always a risk of a paper cut when handling paper tape. ]

  16. chivo243 Silver badge
    Windows

    Looking forward to this!

    Even in this day and age, somethings never change. I look forward to calls like this when I leave my current post in a few months! Not so much for a cobble job, we gave that up a long time ago. But stuff I know, that won't be part of my turn over package...

    1. Anonymous Coward
      Anonymous Coward

      Re: Looking forward to this!

      In the same boat. I'm turning over every single thing I can think of, but I'm sure there will be things missed. As I'll be working for a semi-competitor, I kinda wonder if they'll call or just muddle through.

      1. Denarius Silver badge

        Re: Looking forward to this!

        same here. In previous life on arrival over the year I found decades old manual procedures that could be automated with some shell code mostly. Set up code to run over year at appropriate times or on the common admin server, documenting what code did what and reason for it. Needless to say on third year when I left, a common comment when something time expired ( remote passwords, etc) was, Oh you've been xxxxxed" . No suck-cessor (sic) read docu of course.

  17. Anonymous Coward
    Anonymous Coward

    Walking around firewalls

    Many years ago, we did set up 3 layers internet platform. Serious stuff, no routing across 2 non adjacent layers.

    The internet platform devs were utter crap, barely able to do their shoelaces. Couldn't even figure out what an IP address was, less so a TCP port.

    One called me one day asking: "Can we circumvent the firewall to have $UTTER_CRAP_APP work ?".

    Hmmmm, no mate, this is the whole point of a firewall :)

  18. Vaughtex
    Big Brother

    Powered Off

    Sounds like the desktop PC we were asked to fix that had been sat under a desk in a certain tower of Canary Wharf. Said PC's owning department had contrived to automate emails from it to all and sundry on a daily basis, containing a batch of data thrown out by their spreadsheet following overnight updates. It's failure was traced to a new group policy that changed the power schedule and automatically shut down any PC without an active user after 7PM. Needless to say much mirth on our part as it was nothing we could do anything about and a great deal of questions to be answered by the owing team as to why this PC was sat under a desk and not registered as a proper service running on a server.

    1. yetanotheraoc Silver badge

      Re: Powered Off

      They should have contrived to automate the periodic mouse movement needed to keep the PC reporting an "active user". I may or may not have seen such automation in use.

      1. FBee

        NoSleep.exe

        Kaching!!

  19. Anonymous Coward
    Anonymous Coward

    I got called in some time ago, to help a two-node Solaris cluster talk again to a two-node Windows cluster via Windows Services for UNIX. The Windows resource had left the company (not by his own will, as many others out there) and nobody could remember how we concocted that solution 10 years ago. Hell, even I was unable to remember what the heck we did. I had left that account in search of greener (and Solaris-less) pastures shortly after we got the solution working on dev/test/prod and properly documented it, and I made sure I had enough beverages to obliterate the Solaris knowledge from my brain.

    The guy on the phone sounded pretty nervous, and was trying to engage me to get back into supporting that hybrid monster, and then I recalled the exhaustive documentation me and my Windows counterpart compiled, both from the official documentation and our own painful experience. So I politelly pointed the caller to the account's wiki, and told him all he needed to know was there. It was evidently enough to have them restore the hybrid Hydra to its working state, because nobody else bothered me again.

  20. jollyboyspecial Bronze badge

    Very early in my career I learned no to suggest temporary solutions since they have a habit of becoming permanent solutions,

    So now when somebody asks for a solution I design what's needed and price accordingly. The questions "can't you do it cheaper?" and "isn't there a quicker solution?" are met with blank incomprehension.

    Do it nice or do it twice is my motto. The problem with far too many customers (mostly project managers) is that no matter how hard you explain that the cheap and nasty solution is only a short term solution you just know it will become a long term solution right up until the point where it shits itself. At which point you are suddenly expected to do it nice the second time round, but now somehow at zero cost because the project was completed and signed off months if not years ago and there's no budget for the solution that should have been installed in the first place.

  21. Anonymous Coward
    Anonymous Coward

    Been there, seen it, done it.

    Replaced a crappy Wintel solution (that failed more times than it worked, hence the emergency to do something) with a RedHat 6 server (not RHEL6 by the way, so shows how long ago this was).

    It just worked. It worked so well that I never got a call about it in over 6 years (4 years after I left) until there was a hardware failure.

  22. HkraM
    Facepalm

    Got the T-shirt

    At a previous company we once had a 7-disk RAID5 chassis fail on the main server. Ended up removing all of the HDDs from their caddies, and connecting them up with a SCSI ribbon cable on the desk and positioning a desk fan to blow some air across them. It was just a temporary fix until the chassis was replaced. I think the server stayed like that until it was retired a few years later.

  23. DS999 Silver badge

    I remember a "temporary fix"

    I didn't have to deal with it since I was consulting on another project, but my desk was near some other guys working on a different project that had recently gone live. The guy responsible for the temporary fix hoped to make sure it was only temporary during project testing and wouldn't survive to production, so he had the server that provided the temporary service scheduled to reboot every day at noon.

    You'd think that would focus them, but no. The project went live and people depending on it just got used to it going down at noon which they felt was fine since that's lunchtime, right? They were actually using the sudden disconnect as a reminder it was time to go to lunch! He wanted to push things along and wished he hadn't picked noon, but one of the higher ups knew that daily reboot was deliberate so changing it would point the finger of blame squarely on him.

    I said "hey daylight savings time is in a week, why don't you move it an hour earlier and you can claim the daylight savings shift was responsible?" He came in Monday morning after the shift and changed the reboot time to 11am, and quickly the howls were loud enough for him to get the powers that be to allocate money for the permanent solution he'd been promised!

    1. Martin
      Happy

      Re: I remember a "temporary fix"

      That, I have to admit, is ingenious!

  24. Jou (Mxyzptlk) Silver badge

    This perfectly describes why I reject those Xing etc offers

    My current employee is a service provider. Our customers want something done, we say it has to be done this way. If the customer does not want it that way to cut some corners we are not affected. It is the customers responsibility to leave such solution going on for too long or ignoring our recommendation.

    But when I am an internal IT in a company I face exactly what makes up so many stories here: I know what is right, I get overruled, and I am still held responsible for its operation and failure. Usually by those who overruled me.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like

Biting the hand that feeds IT © 1998–2022