back to article Tech support fill-in given no budget, no help, no training, and no empathy for his plight

When the weekend rolls around, nobody needs permission to do whatever they desire. Unless, of course, they're required to be available to support tech – a restriction we mark each week in On Call, the column that celebrates fine fixing feats achieved despite the footling of flummoxed fools. This week, meet a reader we'll …

  1. Doctor Syntax Silver badge

    "Sometimes it is easier to ask for forgiveness than to get permission,"

    The technical term is fait accompli which translates as "the only way to get things done".

    1. Jou (Mxyzptlk) Silver badge

      What the story does not tell: Did he ever tell anywhere else that thing existed, and if, really got forgiven? Naaaaaaaaa...

      1. doublelayer Silver badge

        Given that the IT department in question were either unable or unwilling to put in the most basic of controls that would tell them when unusual devices were connected to their network or block those devices from working until they got approved, I'm guessing that they never found out. I'm surprised they decided to forbid it in the first place rather than taking a "you try it and any problems are not our problem" policy.

    2. Korev Silver badge
      Coat

      "Sometimes it is easier to ask for forgiveness than to get permission,"

      The technical term is fait accompli which translates as "the only way to get things done".

      You're not Ron...

      1. AndyMTB

        When I was a nipper, I read everything the local library could offer. This included stuff like the Jennings schoolboy series, Biggles of course and Bunter. Didn't understand half of the latter - what, they don't go home at the end of the day? - but I do remember Billy getting upset after he (thought) he overheard one of his school mates refer to him as a fat accomplice. I chuckle everytime I read the proper phrase.

        1. PRR Silver badge

          > I read everything the local library could offer. This included stuff like the Jennings schoolboy series, Biggles of course and Bunter. Didn't understand half of the latter - what, they don't go home at the end of the day? -

          These stories are unknown in the US. I'm enjoying discovering boys-books 60 years after my boyhood. The original (there are no others!) Hardy Boys and Nancy Drews are coming out of copyright. (Beware Harriet's revisionist versions of the 60s.) I read most of the Yankee Flyer books but Biggles surpassed them all. Canadian Copyright is more liberal and https://www.fadedpage.com/ has been eagerly posting works now public domain in Canada. (I'm close to the border, does that count?) Jennings is not hard to find.

          1. Terry 6 Silver badge

            Bunter was, to the say the least, problematic. Essentially, middle class, over weight and unhappy kid sent off to a minor boarding school where most of the other boys are over-privileged bullies- but the author takes the bullies' side and brings the readers along with that premise.

            Except that for me, even as a small child, it just felt very wrong. This was clearly bullying. Later in life I understood the undertones of racism that were in there too.Not so obvious to me at the time, because the Indian boy was a stereotype, but he was also one of the rich bullies.

            1. Yes Me Silver badge

              The Bunter books were truly awful, in retrospect. There was some very dodgy celebration of flagellation too. It all made me very glad that I went to state schools.

              1. Eclectic Man Silver badge
                Unhappy

                Prejudice in fiction

                Capt. W. E. Johns, who wrote the 'Biggles' books, did come across as racist in some, but was advised to clean up his act. In, I think it was 'Biggles flies South'*, he has a conversation with a black jet fighter pilot. Biggles' companion is very angry that a black man had the temerity to force them to land, but Biggles defends him pointing out that what matters is how you fly, not the colour of the thumb on the 'fire' button. In previous books, 'easterners' and others were not treated so kindly.

                Of course Enid Blyton was terribly racist (in the 'Noddy' books, the villains were always the black ones [actual term is probably too racist to publish here]).

                One thing that does get me is that in a lot of the newer 'period' detective dramas there is invariably a 'show' ethnic minority character who is never shown as being racially discriminated against. This is historically inaccurate, but for the comfy 'it will all be wrapped up in the next one or two hours' crime shows the heroes have to be shown to be completely Politically Correct / Woke. I am not defending racism (I have been racially abused myself), just disappointed that these TV programmes do not address or even acknowledge the prevalence of historical racism in UK society. I quite enjoy 'The Brokenwood Mysteries", about antipodean police who have the requisite high body count (at least one per episode). But the contortions the plot goes through to get the token Maori character involved (he is not a cop) are often ridiculous. When a Maori police officer was introduced the other one was dropped completely, even though he had been a 'good friend' of the main character.

                Let's face it, one of Agatha Christie's best sellers was originally titled "Ten Little N*&^%$s', then changed to 'Ten Little Indians' (for the American public, but not much less racist), and eventually became "And then there were none".

                I hope I have not spoiled anything for anyone.

                * Could have been 'Biggles in the Sud', if that tome exists, memory getting bit hazy about books I read over 55 years ago.

                1. Chris Evans

                  Re: Prejudice in fiction

                  Wikipedia lists no 'Biggles in the Sud'. Biggles in Africa & Biggles in the Gobi are the closest I could see.

        2. agurney
          Joke

          "Biggles Flies Undone" was a cracker

      2. Doctor Syntax Silver badge

        "You're not Ron..."

        No, Ron was my Dad.

      3. User McUser
        Coat

        I'm not Wright either.

        1. Doctor Syntax Silver badge

          Wright was my great-grandfather.

    3. Anonymous Coward
      Anonymous Coward

      Also Gibbs Rule #18

  2. Jou (Mxyzptlk) Silver badge

    27 years later

    they found a NAS walled in still humming along. Fan was broken long ago, but it still worked.

    I love how he did not give up to get something in a working state.

    1. JulieM Silver badge

      Re: 27 years later

      Obligatory XKCD: https://xkcd.com/705/

  3. Ali Dodd

    Universities are the absolute worst for shadow IT

    Shame the IT dept. were basically not supporting the users properly but I'll bet that thing chugged along till it didn't, which was probably after this chap left and then was dumped on IT with no information whatsoever but had to be fixed.

    Seen that before (and much much worse), glad I no longer work for a Uni..

    1. Pascal Monett Silver badge

      Re: Universities are the absolute worst for shadow IT

      Agreed.

      From what I've read (here and elsewhere), universities are the absolute most dismal IT environments that exist. No budget, so-called "administrators" that have nothing but demands and no knowledge of the requirements, and no flexibility when it comes to finding solutions.

      It's obviously a lot easier to shield oneself with a half-baked procedure document than actually think about how to make things work.

      1. Keith Langmead
        Headmaster

        Re: Universities are the absolute worst for shadow IT

        When I was at Uni a friend who worked in the IT department told me much the same. Part of the problem was they had very few if any experienced and talented staff. Most got a job there as their first IT gig and did it for a year or two to get the "supported large organisation with n thousand users" on their CV, then jump to a better much higher paid IT job elsewhere. The only experienced staff were mostly those that were so crap they couldn't get a job elsewhere, and were happy to just bumble along until retirement.

      2. Dave K

        Re: Universities are the absolute worst for shadow IT

        I worked at a University from 2006 to 2012 in IT. Our department ran its own IT systems, which actually meant I had a budget for PCs, a couple of servers to administrate, plus a remote NAS for backups (as well as a tape-safe for local ones). Was a fun job overall, but reading stories like this tells me why the department invested in their own IT instead of relying on the central facilities...

      3. Arbuthnot the Magnificent

        Re: Universities are the absolute worst for shadow IT

        A few years ago I went back to Uni to do postgrad study. We were constantly nagged about saving energy, but every evening you'd see hundreds of monitors sitting displaying the login screen for the managed desktop environment in offices - sometimes even on projector screens in lecture halls. I put a ticket in to IT suggesting they at least enable display power saving. A week later the ticket was rejected as "you can't run group policy on a monitor".

    2. Lazlo Woodbine Silver badge

      Re: Universities are the absolute worst for shadow IT

      A customer of mine used to specialise in network audits, which were usually carried out prior to upgrades or introduction of new tech,

      He got a call from a large UK university who wanted to install a new networked IP camera system across the main campus building.

      When my customer carried out his survey he found out the university already had a full IP camera system, running to hundreds of cameras and multiple recording servers, that the site management had installed a couple of years earlier, failing to tell the network department.

      Because the network department hadn't been informed this system was running on their network and their switches, it wasn't even on a VLAN, and many of the students had already discovered the recording server and were merrily deleting any footage of after hours shenanigans.

      Anyway, my customer utilised the existing kit and reconfigured it all so it was on a secure VLAN, this meant I didn't get a sale, but we had a new case study to show to other Universities and large colleges to ensure they installed kit properly in future.

      1. Jou (Mxyzptlk) Silver badge

        Re: Universities are the absolute worst for shadow IT

        A classic example showing how important audits are! I have similar cases on and on... Not only regarding networked equipment, also regarding the "we have a perfect windows update statistic" where a relatively small powershell script, only scanning "Windows Server*", exposes the reality.

      2. John Brown (no body) Silver badge

        Re: Universities are the absolute worst for shadow IT

        "Anyway, my customer utilised the existing kit and reconfigured it all so it was on a secure VLAN, this meant I didn't get a sale, but we had a new case study to show to other Universities and large colleges to ensure they installed kit properly in future."

        And nicely demonstrating to future customers that you don't install and charge for unnecessary equipment. Well done! :-)

    3. simonlb Silver badge

      Re: Universities are the absolute worst for shadow IT

      Not at a University, but a few years ago (around 2007) I got a call from a user about a PC stuck in a boot loop. After a couple of minutes conversation with the user I'd established there were no identification labels on the machine - asset tags, PC id etc. - they then mentioned the make and model which was by then a defunct manufacturer which we'd never used. Further dialogue revealed that the user had been told to use this PC for their work when they'd started a year previously, and after I'd asked them the check the back of the machine it wasn't even plugged into the network. After asking them what was displayed on the screen when booting they said, "Windows 98." So this was just a random machine someone had brought in years before and started using and it had become a fixture in the office multiple people had used over the years.

      I had to end the call by telling them to speak with their boss as the machine wasn't covered under any support for either hardware, software or user.

      1. Headley_Grange Silver badge

        Re: Universities are the absolute worst for shadow IT

        In the very early days of laptops I wanted one but our IT department didn't and wouldn't support them. I was PM for a project that used a PC for a control and management terminal. The customer asked us to upgrade the CMS to a laptop, so we had to choose a laptop and then give it a part number on the company system so that it could be procured on the project. I had loads of bunce in the project so I bought myself one. IT didn't get a look in - I raised a purchase req and one appeard a few weeks later. I gleefully set it up - insert disc 3, etc. - and, in my naivety, plugged it in to the the network. Nothing happened. I messed about a bit and nothing kept on happening so I called IT. They turned up, took one look at the laptop and left. I transferred it back into stores for the project and when, eventually, laptops were rolled out generally by IT I was, naturally, one of the last people to get one.

    4. chivo243 Silver badge

      Re: Universities are the absolute worst for shadow IT

      Edu institutes are the worst for this, as someone with many years at an Edu, I understand why teachers\professors do these things. Some are being creative and want to expand faster than the slow crawl of IT budget requests for next year, it's really frustrating for them. Others just want things their way because they have chosen a noble profession, entitlement and such...

    5. doublelayer Silver badge

      Re: Universities are the absolute worst for shadow IT

      I sympathize with the IT department. It's hard to determine where your responsibilities lie when it's a separate organization that is sharing some resources. Making sure they have network connections sounds like IT's job. Fixing their computers sounds like the organization's job. From the incident described, the problem was not the resources provided by the university going wrong. Of course, IT's insistence on preapproving the hardware they connected suggests that these lines weren't as clear as they should have been, because in my suggested arrangement, university IT would have provided them with a network where the organization could connect whatever they bought. Still, I've been in too many situations where I was expected to support computers that I had nothing to do with to blame all of this on the university's IT department when the users were not university staff.

      1. Anonymous Coward
        Anonymous Coward

        Re: Universities are the absolute worst for shadow IT

        You also have problems with shadow IT in the NHS.

        They normally dress something up as medical devices but then they expect them connecting to the network and for IT to support.

        Often the devices aren't suitable or have issues but they have ordered a large number before IT have even seen them.

    6. The Oncoming Scorn Silver badge
      Pirate

      Re: Universities are the absolute worst for shadow IT

      "Shame the IT dept. were basically not supporting the users properly "

      You want to shame the BOFH, I'd send flowers, but alas the carpet & the quicklime render that a redundant exercise.

    7. DS999 Silver badge

      Long ago I was in charge of IT for part of a university

      Major top research (AAU) institution, it had an overall IT department but they did PCs and Macs, and supported some centralized stuff. My division included all the high tech liberal arts departments like physics, math, stats and of course computer science. Since those professors were always getting grants to buy weird stuff like giant Silicon Graphics machines, clusters of HP RISC workstations and so forth the central IT was happy to let us administer our own stuff.

      Fortunately my predecessor had been visionary in terms of implementing a "tax" on grants that insured some money flowed to my budget to cover overhead like upgrading servers, adding storage and so forth, and there were student fees that covered buying workstations for student labs which we also handled (since they were RISC not PCs)

      The main pain point was where things overlapped, which was primarily desktop support (we had our own email server and file server so we had to support them to access that, central IT did the rest) and network support. They provided the networking hardware, but what they were willing to provide was woefully behind what was needing when we had something like 200 RISC workstations that all needed to talk to our fileserver, and obviously this was across multiple buildings. Fortunately the fileserver was co-located with the biggest lab that also doubled as the landing area for those big SGI Onyx systems and stacks of headless HP workstations, so we had our own switches to handle that. We had 100 Mbit links between buildings which was OK for the time, but they would only provide 10 Mb switches to attach stuff and only two ports per room. When you have labs with a couple dozen RISC workstations that eventually became untenable.

      So I had IT disconnect all the labs that had lots of stuff and essentially "stole" their cat5 runs that were now dark, and installed a 100 Mb switch in each lab plus another 100 Mb switch in a few closets to tie everything together. It took them months, but eventually they noticed, but after a meeting where they threatened to pull them I managed to get a meeting with the dean of liberal arts (only time I met her rather than dealing with one of her underlings) and told her what IT was going to do and how much it would hurt us and she managed to get a "truce" arranged where we provided networking for our RISC stuff at our own expense, and IT provided sufficient 100 Mb uplink ports to connect one per lab because they REALLY did not like me installing new switches in their closets. Especially since I picked the lock to get access lol

      1. the Jim bloke
        WTF?

        Re: Long ago I was in charge of IT for part of a university

        ... all the high tech liberal arts departments like physics, math, stats and...

        what kind of university considers STEM to be liberal arts? sounds more like a seminary or flat earther hangout...

        1. DS999 Silver badge

          Re: Long ago I was in charge of IT for part of a university

          Sounds like you need to educate yourself. "STEM" was coined by the NSF in 2001. The concept of liberal arts dates back to classic Greek antiquity.

          In classic antiquity liberal arts included the trivium of rhetoric, grammar and logic, and the quadrium of astronomy, arithmetic, geometry and music. Please tell me how the four departments I listed don't belong?

          Check out https://college.harvard.edu/academics/liberal-arts-sciences/concentrations and see that all the departments I listed are represented. Are you going to claim Harvard is some religious flat earther school? Sounds like you're the one with the shitty education, because you don't even know what liberals arts are. Maybe you've been programmed by right wing media to equate liberal arts with women's studies or other stuff they look down upon?

        2. Yes Me Silver badge

          Re: Long ago I was in charge of IT for part of a university

          There are many possible explanations (for a textbook on the subject, read "Moo" by Jane Smiley) but the generic answer to anomalies like CS and Maths being in Social Sciences (which I experienced many years ago) is simply academic politics and which senior professors and Deans hate each other (or the opposite). But generally speaking, everybody hates the university IT department for reasons amply explained above, so they are bypassed whenever possible.

    8. swm

      Re: Universities are the absolute worst for shadow IT

      At RIT we had our own support group in the computer science department. They ran file servers, mail, web server etc. They were excellent. Then the university decided that all web pages belonged to them, the mail servers belonged to them etc. So now there is a crappy home page for the CS department, mail is microsoft crap etc.

      Oh well.

  4. Prst. V.Jeltz Silver badge

    Just a wild shot in the dark but i bet this could also have been fixed by taking the switches off 10T half duplex.

    (based on my own college I.T. employment)

  5. Anonymous Coward
    Anonymous Coward

    Not "Fixing", Exactly

    In uni we had a PDP-11/45 running 6th Edition Unix in the CS department. It was classed as "unsupported" by the IT "department" (mainframe people). The machine lived in an office on the second floor of the maths department building, and was in heavy use all day, 8AM-5PM. The maths dept chairman refused to allow us access to the building outside those hours, though we asked very nicely to do so, that we might make system-wide backups.

    Workaround: whoever in the cabal (four of us) was in the room just before 5PM lockup would unlatch the window before leaving the room and locking the single door to the room. Later that night, we would return, climb the exterior stairs to the 2nd-floor level, and select a brave soul by lots.

    That brave soul would climb up over the handrail, jump the three feet to a ledge ringing the outside of the maths building, croodle along it to the window, open the unlatched window, and crawl inside and make the multiple-media (RK05 cartridges) backups, the room lit only from the -45's front panel lights and a streetlight shining through the window. Backups complete and verified, the brave soul would close and lock the window, and exit the room via the only door, locking it behind him, then exiting the building via the push-bar double doors at the side of the building.

    While this was happening, the other three of us had climbed to various overlook points, equipped with air-horns, and watched for security. Security did come by, but only to check that the exterior doors were locked.

    1. Lazlo Woodbine Silver badge

      Re: Not "Fixing", Exactly

      Ethan Hunt style IT support...

      1. KittenHuffer Silver badge

        Re: Not "Fixing", Exactly

        Pity they weren't writing the backups to CD ..... then along with the Dooo da looo music you could have have the image of a burning fuse CD!

        1. Peter Gathercole Silver badge

          Re: Not "Fixing", Exactly

          PDP 11/45 running UNIX edition 6.

          A long time before CD-ROMs were available.

          I somewhat doubt that DEC or any 3rd party peripheral manufacturers ever made a UNIBUS attachable writable CD-ROM drive!

    2. KittenHuffer Silver badge

      Re: Not "Fixing", Exactly

      Are you sure you're not just paraphrasing parts of the script of Brazil?!?

      1. Adrian 4

        Re: Not "Fixing", Exactly

        https://xkcd.com/705/

        1. Roopee Silver badge

          Re: Not "Fixing", Exactly

          Read the comments from the top and you'll see why I downvoted you...

          1. Prst. V.Jeltz Silver badge

            Re: Not "Fixing", Exactly

            Read 100 comments from the top of the page for clues to why you downvoted that perfectly fitting xkcd ????

            Why not just tell us , like this:

            "I down voted your 'Read the comments' comment because nobody has the time or inclination to search for your point which may or may not even exist"

    3. Trygve Henriksen

      Re: Not "Fixing", Exactly

      All wrong!

      You should have taken the machine offline at 9am and run the backup then.

      1. lglethal Silver badge
        Go

        Re: Not "Fixing", Exactly

        Exactly. You dont humour people by going out of your way to solve their problems.

        1) Advise them the need for out of hour backups.

        2) If they refuse, then advise them that you plan to perform the backups during working time. be sure to inform them that during this time the server will be unavailable.

        3) If they refuse, make them sign in front of witnesses (your boss, there boss, if possible) stating that they accept all liability for loss of work if the server should fail, and no back ups are available.

        4) If they refuse, escalate, escalate, escalate. And make sure as many people as possible no, there are no backups and it is because of this person...

        That is the only way to move forward here...

        1. that one in the corner Silver badge

          Re: Not "Fixing", Exactly

          You missed a point with this story:

          Escalation occurs on both sides: whilst you are trying to get more authority to do the work properly, your manager is escalating the complaints about your laziness and unwillingness to work within the rules.

          However, with the Ethan Hunt backup completed, you only need to perform one restore (possibly a bit staged, couldn't comment on that). Manager presents that to upper echelons as "look, he can do it when he tries". Then you give *all* the messy details about just *how* you'd had to risk life & limb to comply. Or, at least, point out to boss that you have that in your back pocket for next time he is a prat (wording may change).

        2. A.P. Veening Silver badge

          Re: Not "Fixing", Exactly

          4) If they refuse, escalate, escalate, escalate.

          Wrong, after the first escalation, make sure the system crashes to a state needing the backups.

          And make sure as many people as possible know, there are no backups and it is because of this person...

          Elementary.

          1. Headley_Grange Silver badge

            Re: Not "Fixing", Exactly

            In the early 90s I worked at a company that was expanding and the group I worked for were moved out of the 1950s converted factory offices to a shiny new building with aircon, kitchens, and its own car park. There was a lot of jealousy about our new offices and parking. IT, who also didn't like the 200 yard walk, were particularly arsey about it and responded slowly to requests. We had an assigned IT PoC for the building - let's call him Bob. Bob fancied an office in our small palace but his boss wouldn't hear of it. One day Bob "mistakenly" introduced some of us to a command that we should never use because it crashed the print queue and it could only be reset by IT from our building. This was handy if you had a ticket that needed prompt attention; simply crash the queue and since this was a high-priority fix, Bob turned up almost immediately, re-started the print queue and could then be hassled to fix other stuff while he was there. In the end Bob was spending so much time in our building that his boss conceded that he might as well move in.

          2. imanidiot Silver badge

            Re: Not "Fixing", Exactly

            I would put purposefully crashing a system and causing data loss down as entirely unethical. Don't do that. If it happens to occur naturally it's a good lesson, but it's not one you should be causing on purpose. Because if anyone with an iota of a clue ever finds out you'll likely receive marching orders.

        3. Missing Semicolon Silver badge

          Re: Not "Fixing", Exactly

          It's only the students' data. Nothing of value to be lost.

          1. Doctor Syntax Silver badge

            Re: Not "Fixing", Exactly

            This is apt to be the situation. There was the KCL episode a few years ago in the aftermath of which, having lost a lot of data on their managed systems, IT told the rest of the college that they should be keeping data on the college's system, not on their own departmental, personal or whatever system.

        4. Anonymous Coward
          Anonymous Coward

          Re: Not "Fixing", Exactly

          > 3) If they refuse, make them sign in front of witnesses (your boss, there boss, if possible) stating that they accept all liability for loss of work if the server should fail, and no back ups are available.

          > 4) If they refuse, escalate, escalate, escalate. And make sure as many people as possible no, there are no backups and it is because of this person...

          No, no no! Don't make decisions for yourself!! Identify options and outcomes, and present those to the managers to choose!!

          Your job is to do what you're told. You were told to make backups, and you were told to make them during business hours. You can tell them about the options - outside of business hours, within business hours; and the outcomes - downtime during business hours, downtime outside of business hours -- but you are only following their commands to the best of your ability. Until _your manager_ tells you to halt the project, you have your orders: make backups, during business hours. If the other party (not your manager) doesn't like this, they can take it up with your manager. Continue your work, until you're told to not continue your work. Perhaps you can work slowly, giving them time to consider, but do your damn job.

          No, it's *not* malicious compliance. It's doing your damn job. You've informed them of the options, effects, and having done your job, you're going to do your job. Your job is what your manager says it is. Let him say "Stop," giving him time to say that, or let the job get done.

          Document your current job with higher ups, and document it *specifically with HR* (who must legally keep a copy of that documentation), and emphasize that this may become evidence in an internal investigation, and emphasize that these are your marching orders despite your best judgement. Now, do your job. If you don't like your job, find another.

          Do not contradict your manager. Do not fail to do your job. If you *physically cannot* do your job, then document that with HR, and ask them for assistance with conflict-resolution. Your manager will hate you, you have a bad job, your manager will make it worse, and you should be looking for another -- but this is how to protect yourself, legally.

          1. An_Old_Dog Silver badge
            Pirate

            Re: Not "Fixing", Exactly

            @AC immediately above:

            You've gone way off track.

            We were not employed to administer the system. We had no jobs within the CS department. There was no one employed or tasked with maintaining the 11/45. It had ZERO HW and SW support. Our cabal was merely a group of knowledgable students.

            Icon for guerilla tech support.

    4. Anonymous Coward
      Anonymous Coward

      Re: Not "Fixing", Exactly

      Now THAT is xkcd 705 level of admin!

  6. Bebu sa Ware
    Windows

    At least Ronnie backed up the NAS.

    I once had a "(very) cheap and cheerful" LaCie NAS device connected to the local network (contravening at least a couple of central IT commandments) by a staff member purely to use as a clearing house for WH&S (OH&S) inventories. I only heard* about it because the owner (literally as she putchases it personally from a local whitegoods retailer‡) was healthily paranoid about backups generally and even of the few shares containing this rather ephemeral data which I duely arranged.

    Many moons later I notice the scheduled back up of the device failed, investigating found the device had failed. The owner had long before moved the workflow to a organisation wide application so no sweat. The device was totally dead so I was given the carcass for disposal but being curious I later performed a postmortem. Bugger all in them a 2.5" SATA disk and a SOC. Extracted the disk for later examination.

    About a week later a researcher who was notoriously lacking in any computing skills admitted he was storing his files on a share he found on the network which had recently vanished. Could I find it?

    After a lengthy interrogation I discovered he had over 1Gb stored on the defunct NAS. It was originally just a backup of his PC but thought to have two copies was a waste.

    As consequence of his unauthorised use of the device his share wasn't backed up leaving him up the proverbial creek.

    Given that it was likely only the SOC that lost its magic smoke I thought it worthwhile to examine the drive.

    The patron saint of fools and idiots must be a busy soul but this researcher must have enjoyed his particular favour, On examining the drive with Linux† PC I discovered that the SOC didn't run some obscure embedded OS or use a proprietary file system but a MIPS Linux kernel with an XFS data volume which of course could be mounted on the Linux PC - some sods are born lucky!

    In Ronnie's case one has to wonder why anyone, professor or not, is running computational workloads on a file server.

    * of course it popped up on my network monitoring well before this. Hardly Normal specifically.† Redhat EL3 or a new EL4 at that time.

    1. doublelayer Silver badge

      Re: At least Ronnie backed up the NAS.

      "In Ronnie's case one has to wonder why anyone, professor or not, is running computational workloads on a file server."

      My guess is that it was intended as a general-purpose server which was being used by this organization as a file server because they didn't have one or the ability to administer one. The professor probably didn't have access to anything more powerful, including their office computer, so they used the best option available. Or perhaps the professor used something with larger disks because their workload generated more data than they could fit on their personal computer.

      1. John Brown (no body) Silver badge

        Re: At least Ronnie backed up the NAS.

        Yeah, that's my takeaway too. The article only called it a "server", not a "file server". As you say, it was probably bigger and faster than most desktops and did multiple duties and would be the obvious place to run large data thrashing or computationally expensive stuff.

    2. Peter Gathercole Silver badge

      Re: At least Ronnie backed up the NAS.

      I use Buffalo NAS devices. These are not so dissimilar to what you've described, and it is even possible to open SSH access to them (once you open them up) and run stuff, not that the puny CPUs are really up to to running anything significant. Latest one is using a Marvell ARM processor, older ones use MIPS.

      My latest one is using mdraid support to provide raid support. Nothing special about the Linux apart from the admin. web pages.

    3. David Hicklin Silver badge

      Re: At least Ronnie backed up the NAS.

      > "(very) cheap and cheerful" LaCie NAS device

      I had one of them as my first NAS devices at home, it just a single disk device. I discovered the hard way that the OS was on the disk - and the disk only!

      So loose the disk on a single disk device and it's toast unless you can manually partition a new disk and somehow get a copy of the OS on it. I think I managed to resurrect it eventually but after that I moved on to Synology where at least the OS is in firmware.

  7. Howard Sway Silver badge

    given no budget, no help, no training, and no empathy

    Welcome to working in IT!

    Once you've learned that the only way to get things working is to work around all the stupid rules that have been put in place that make that impossible, you can happily carry on doing so, knowing that nobody is going to be competent enough to actually check what you've done.

    1. ecofeco Silver badge

      Re: given no budget, no help, no training, and no empathy

      Right?

      This is just a normal day in IT.

  8. Lee D Silver badge

    For similar reasons I have in my time deployed:

    ---

    A script that activated on a detected fire alarm, which queried the access control databases, correlated them against staff databases and printed out (to two remote thermal printers) a firelist of who should be on site, who has tagged in this morning and who hasn't tagged out. It also emailed a group with the same information.

    It was so fast that it actually started printing BEFORE the alarm was audible, and the printers were so fast that by the time you stood up to walk out, you could just tear and take the entire staff list with you.

    The company that fitted our access control system tried to buy it off me, because it was infinitely better than their firelist module (which only printed MANUALLY and only onto A4 and we only had laser printers, which by the time they'd warmed up, you'd already burned to death).

    ---

    A complete transparent proxy / filter with routing, load balancing and firewalling for several connections, including multiple DSL modems that were so flaky I included functionality to activate a rely by SMS message to reset the power to the individual modems. My boss used to use it if he was at home and couldn't VPN in, he would text, it would reset the DSL modems, and let him back in.

    That operated from an old desktop PC and ran my entire workplace for about 8 years.

    (It was actually a challenge - in interview, with two strangely techy people working in non-techy management jobs - they mentioned the organisation's slow connection. I said I could probably solve that. They let me have a quick look in the interview. I realised what the problem was immediately and said I could triple their "speedtest" score overnight if they wanted, and it would only need an old computer to demonstrate how I could do it. And that it would cost nothing.

    They hired me (they liked me for other reasons) and first thing on day one, the big boss literally invited me into his office, handed me an old desktop taken from storage (that was about 10 years old and a former secretaries' desktop) and said "Go on then".

    So I did. Linux. Apache. Squid. Dan's Guardian. It transparently cached all the CONSTANT requests for the same websites over and over again coming from the network, filtered out a ton of inappropriate usage, freed up the connection, and the speedtest score (which the big boss was running every hour or so, and they got better each time) ended up something like 5 times better than before I started. I also demonstrated that if it went wrong, all they had to do was join the IN and OUT network cables on the machine together with a coupler, and everything would work like it had previously. They absolutely ADORED me for that. They thought it was just hubris and were - in the nicest way - expecting me to fail (but still do my job and have a good laugh about it all) and I blew them out of the water instead.

    Later, some obscure kernel patches (to allow multi-interface dead-gateway detection and automatic seamless network gateway load-balancing), additions of power relays, the DSL interface circuits, and a 3G stick (which acted as emergency backup but also for automated text message retrieval via Gammu), etc. as well as an upgrade to a "real" server were allowed.

    I've had dozens of projects like that.

    "This is the way to do it"

    "We absolutely can't afford that, find another way"

    <creates and deploys something that works and costs nothing>

    <becomes the main system for precisely that reason>

    <constantly tries to upgrade, replace, budget for the real thing but is always denied>

    Ten years later: "How do we manage this now that you're leaving?"

    "I don't know. Your problem. It's documented."

    Helpdesk software, asset management, access control, visitor management, digital signage, websites, CCTV, you name it, I've been "made to" bodge it because of not having any resources, and then literally it becomes THE MOST IMPORTANT THING EVER and I'm not allowed to turn it off or replace it but also I'm not allowed any money to keep it going, upgrade it, change it for something sensible and manageable, etc.

    1. KittenHuffer Silver badge

      Been there! Bodged that!

    2. Korev Silver badge
      Flame

      A script that activated on a detected fire alarm, which queried the access control databases, correlated them against staff databases and printed out (to two remote thermal printers) a firelist of who should be on site, who has tagged in this morning and who hasn't tagged out.

      I used to work somewhere with research labs that used radiochemicals. One day one of the chemist's machines developed a fault starting a rather exothermic reaction involving solvents and the Fire Brigade sent about ten fire engines, but upon hearing what was in the building they (sensibly) refused to enter until they were told what radioactive goodies were in the building. The database for this had only ever been designed to satisfy legal requirements and provide information for the disposal companies meaning one of my colleague (who couldn't handle any stress) had to get a query running ASAP...

      I believe that regular printed reports were then created and left near the site's entrances after that.

    3. Anonymous Coward Silver badge
      Thumb Up

      "bodge it because of not having any resources, and then literally it becomes THE MOST IMPORTANT THING EVER and I'm not allowed to turn it off or replace it but also I'm not allowed any money"

      That sums up IT in one line.

    4. ecofeco Silver badge

      My experience has been the opposite. Improve a thing, get fired for making everyone else look bad. Even after having letters of commendation from the executives. Or the company buying the company I work doesn't care. Many times.

      I don't even bother these days except to make my immediate job easier. No reward, ya see?

      That said, good on ya and I hope you have continued success. Sincerely.

  9. Eclectic Man Silver badge
    Meh

    Once upon a time ...

    ... I was Sys Admin for a cluster of Sun (remember them?) workstations and a couple of YP Master servers (they served different domains on the same ethernet, and managed somehow not to interfere with each other). Anyway, one day one of the super high res Sun screens was malfunctioning, so I called the support people from Sun out for a repair, and they said they had to take it away, but could provide a replacement. So I signed the dead screen out and they installed a new screen. We had a requirement that the building would not let any IT equipment out without permission. Oh SHit!! According to my bosses I was not sufficiently senior to 'sign out kit'. So I kind of informed them that I had believed that as System Administrator I had thought I was, it was only a screen (no processor or disk storage) and if I was not allowed to sign stuff out, then whenever something broke one of the people who was senior enough would have to be around to authorise works. They sort of went into a huddle, and never came back to me with a solution. Fortunately we never needed to let any kit leave the premises while I was still Sys Admin.

  10. Andy Taylor

    Once place I worked had a strict procurement policy - all computers had to be purchased by the IT department who had no budget left when we wanted a machine.

    So we ordered the components and built our own "data logger" industrial machine which if I recall correctly cost about 3 or 4 times the price of a single PC at the time.

    1. GlenP Silver badge

      I've recounted before some workarounds for UK Government procurement policies, such as buying Sound Monitoring Equipment, a PDP11/23 with a relatively low cost analogue input, as a computer had to be from ICL.

    2. gnasher729 Silver badge

      “ So we ordered the components and built our own "data logger" industrial machine which if I recall correctly cost about 3 or 4 times the price of a single PC at the time.”

      Many years ago Apple started selling a new computer, but also provided an “upgrade” for the previous model. The “upgrade” was a motherboard, open old computer, remove motherboard, insert new motherboard, close the case. Price was the same as the new computer so buying the upgrade was total nonsense - instead of a new and an old computer you got a new computer in an old case.

      Except they had customers with zero budget for new purchases and plenty budget for repairs, upgrades etc. who were more than happy to buy the upgrades.

      1. Headley_Grange Silver badge

        Budgets cause such problems, especially in government. We bid a big comms project to a government agency in a country far, far away. As part of the bid they wanted a full third-line workshop to be set up - that's diagnosis and repair at to component level. No one did this; you bought enough LRU spares to keep the kit working and sent broken LRUs back to the factory and it worked fine. We won the bid and got down to negotiations and one of our aims was to get rid of the repair facility cos it would have been a real PIA. The were hard over on it, so one night I went out with their lead engineer and we had an off the record chat. All they really wanted was to replace bench gear in their workshops - scopes, NWAs, spectrum analysers, PSUs, etc. - which were 20+ years old. There was no budget anywhere to allow them to do it, so it had to be loaded on to contracts. After that it was easy - he sent us a list, we copied it into the contract and called it something like minium maintenance test gear, requirements, priced it up and that's what we signed off on. They paid our handling and fee-rate on the kit but given it was the only way they could get it they didn't care.

  11. CrazyOldCatMan Silver badge

    We had someone do this..

    .. many years ago, when disc space was on-prem, scarce and expensive..

    They decided rather than working with us (we were "obstructionist and didn't understand them" - possibly because they didn't bother actually talking to us, just tried to get stuff by force majeure..) they'd get a local NAS and store all their (according to them, business-critical) files on it..

    And yes, the expected happened and, one hot summer day (no air-con in our office - it's a listed building..) the NAS promptly emitted the magic smoke and shuffled off this mortal coil.

    Cue much screaming by their director (the one who authorised the install *despite* it being very clearly against the company rules) that they COULDN'T WORK and IT *MUST* fix it now and restore all their files.

    Since at least one of the drives in the NAS had got fried (they'd set it up as RAID 0 because it gave them more space) we couldn't. Cue more screaming about our incompetence..

    Our senior director sat them down for a good chat (she was from a Finance background and very much approved of rules) and a much-chastened director ended up having to pay for a new server, managed (and backed up) by us. We recovered quite a bit of their data from stuff people had saved to their own PCs but they still lost a considerable amount of data.

    Said director didn't last much longer - they had utterly lost the confidence of senior management and even their own team.

    1. David Hicklin Silver badge

      Re: We had someone do this..

      > when disc space was on-prem, scarce and expensive..

      And now it is in the cloud, plentiful and an ongoing monthly expense that keeps on going up....

  12. Rtbcomp

    "Sometimes it is easier not to ask..."

    "Sometimes it is easier to ask for forgiveness than to get permission," I'll remember that, I used to work on the premise that "If you don't ask they can't refuse."

    1. PRR Silver badge
      Happy

      Re: "Sometimes it is easier not to ask..."

      > "Sometimes it is easier to ask for forgiveness than to get permission," I'll remember that,...

      https://quoteinvestigator.com/2018/06/19/forgive/

  13. Boris the Cockroach Silver badge
    Devil

    This story

    sounds like almost every place I've attended while being paid.

    From the days of the defence dept. and "we need XXXXXX NOW!" "ok.. gimme some metal or whatever and I'll make it" to the likes of likes of last week where someone has pulled out a network lead somewhere and now the entire system took a dive.

    It says robot programmer on my contract. not design engineer/IT support/ 2nd line tech assistance / beancounter remover

    The last is not an official duty nor do the manglement ask for it, but its the best part of my job.....

    1. An_Old_Dog Silver badge

      Re: This story

      President of Johnnie's Quicklime writing here, give our rate sheet a look. Discounts for buying in bulk ...

  14. Anonymous Coward
    Anonymous Coward

    Been there, and there's a high chance the poor chap ended up using his own money to get his colleagues out of the crap. Classic education.

  15. Anonymous Coward
    Anonymous Coward

    I'm confused / intrigued

    How does a NAS fix "pressing a key took 8 minutes for a character to appear on screen"?

    1. Jou (Mxyzptlk) Silver badge

      Re: I'm confused / intrigued

      1. Why as AC? It is a legit question...

      The article hints: Files are copied and accessed over a slow line which is used for terminals as well after they moved the servers.

      Move the storage locally, and the line is free for the terminals again since all those data moves are within the building.

      At night backup to main servers, 'cause no one cares how full that line is during off hours, as long as the backup is completed before work begins.

      That should sum it up.

      1. Anonymous Coward
        Anonymous Coward

        Re: I'm confused / intrigued

        I post everything as AC - I really should just change my username to be less identifiable as me in the real world.

        Thanks for the explanation. I suppose my confusion was IT went from dumb terminals where a connection was everything, to local apps where files would only be read or written upon save, to a sort of full circle today where lots of apps are in browsers and the connection is everything again.

        So my assumption was if it were the former setup, some other configuration would be needed other than just moving storage locally. And if it were the middle of my list, entering a character wouldn't be affected, only loading and saving documents.

        Whilst I got into computers in the 80s whilst at school, my IT career didn't start until the dumb terminal format was more or less gone. I did work experience in one place that still had that setup which was cool, though, so I've seen it in action. So my ignorance is borne out of lack of experience with the configuration of that kind of setup :)

  16. Sparkus

    stealth-upgrade of multiple server OSs....

    Notoriously lazy SAP Basis team who refused to update their SAP documentation and had the company stuck on AIX 3.2.55 (!!!) for a year after IBM pulled support. They were satisfied with the 92% uptime of the overall SAP installation because it was 'better' than the 85% reliability of the old SAP on System 1.

    Leaned heavily on the (then very new) AIX transparent/no reboot OS upgrade and patch capabilities. Took over three weeks to upgrade a good-sized farm of servers one at a time. Took another 6 months before the Basis team noticed both the upgrade and the suddenly improved uptime numbers to 4 nines.

    This particular story goes on for 3-4 more years and (IMHO) deserves both it's own article on El Reg as well as an SAP whitepaper and IBM Redbook.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like