back to article Pizza and beer night out the window, hours trying to sort issue, then a fresh pair of eyes says 'See, the problem is...'

The second On Call of 2021 is a warning not to mix pizza and hashes. Today's story comes from "Jon", who describes himself as "a manager of a perpetually understaffed IT team." As many will be all too familiar, this put him directly in the escalation path for pretty much any issue that could be daubed with the "it's IT's fault …

  1. don't you hate it when you lose your account

    Proof reader

    Sometimes we need a second set of eyes to see what's there not what we assume is there. Ect instead of etc is my eternal stupid. My dyslicsic brain has done that one so many times I do a search these days just to be sure.

    1. GrumpenKraut

      Re: Proof reader

      'form" <--> 'from" is my, erm, favorite. Spellchecker does not help, of course.

      1. Hubert Cumberdale

        Re: Proof reader

        That's why I've written my own Word toolbar with routines to highlight all these potential problems.

      2. My other car WAS an IAV Stryker

        Re: Proof reader

        I do some paid proofreading on the side. All documents are Microso~1 Word (pre-x format) in Courier New.

        One of the report writers for my client managed to substitute -- probably without their knowledge due to copy-and-paste from other sources (PDF) -- "fi" and "fl" with single glyphs. Word spellcheck did not catch these either, only a vague visual feeling of "that don't look right" and using the cursor to see it was indeed just one character and not two.

        Thankfully find & replace handled both of them quickly and completely (with the side effect of "how did I miss so many before I noticed?!"), but now I have to be (even more) suspicious of every file coming from that particular writer, if not the lot of them.

        1. Dave314159ggggdffsdds Silver badge

          Re: Proof reader

          For some reason - probably very interesting*, but as yet unknown - proofreading on-screen is much, much harder than proofing printed versions.

          *Now I think about it, perhaps not. It might be that we proof on-screen before printing, and it's the two stages combined that are more effective than either alone.

          1. Giles C Silver badge

            Re: Proof reader

            I do a club newsletter and always print a copy out to read to make sure it is correct before getting someone else to proof it.

            It is a lot easier to find mistakes on a piece of paper than on a screen. It is also easier to mark up changes as ideas...

            1. onemark03

              Proof reader

              As a professional translator I also do proof-reading. One trick I have learnt is to proof-read work (of others as well as my own) twice, not just once. I find it always pays off.

              I respectfully commend this method to you.

              1. anothercynic Silver badge

                Re: Proof reader

                As someone who has done magazine translation and proofreading work, I can only second onemark03 here.

                Read through it, correct, save. Close. Walk away, take a break, do something else. Then come back, re-open, re-read, carefully. You will spot more. Of course, running a spell-check in your chosen target language is the first port of call. It's the from-vs-forms and there-vs-theirs that you suddenly spot, having of course passed the spelling and grammar checks on the first pass.

                That said, despite my obsessiveness about good grammar and spelling, nothing irked me more than to get a copy of the final product, only to spot *more* glitches. My copies are full of little marks where I've found more that I should've caught the first time around.

                1. TRT Silver badge

                  Re: Proof reader

                  As did I for scientific papers. The number of people who make up their own version of BS 5261-2 is unbelievable. Anyone would think they had no desire to get things right.

              2. Michael Wojcik Silver badge

                Re: Proof reader

                For important documents, when time permits, I do the second read backward: start at the end of the document, and (try to) read the last clause, then the preceding one, and so on. It took some practice to learn to force myself to do this, and, if it's a language I'm fluent in, it's not possible to always avoid skipping further back and reading longer sections in the correct order. But I found reading backward prevented me from skipping over a fair number of low-level errors such as typographical transpositions, even though I'm pretty good at proofing during normal reading.

            2. eamonn_gaffey

              Re: Proof reader

              I nearly always find errors on reading the printed version, even after having done the same on screen.

              My explanation is those of us who grew up with paper will find proofing a printed copy fruitful, and the kids of the digital age won't ever need to look at one.

              Agree that marking up is so much better on paper, Word Review is just plain awkward.

          2. GrumpenKraut

            Re: Proof reader

            I used to hate proofreading on screen. These days I display the document very oversized when proofreading, double linear dimensions. That helps a lot!

        2. John Brown (no body) Silver badge

          Re: Proof reader

          "the side effect of "how did I miss so many before I noticed?!")"

          I used to teach IT to students back in the days when we were still using 8-bit computers, so far back that teaching "keyboard skills" was part of the course. Assessing and proof-reading is HARD. There's only so much that can be automated, especially when the layout, eg spaces and tabs, are as important in the test as the actual letters and numbers. As you say, you often see what you expect to see, not necessarily what is really there. Even more so when you are on page 4 of the 20th almost identical submission of the proof-reading session.

          1. A.P. Veening Silver badge

            Re: Proof reader

            Most of that can be done automatically by the compiler, it just used to be too expensive.

          2. sketharaman

            you often see what you expect to see, not necessarily what is really there.

            True dat. Perhaps this tendency harkens back to the roots of Information Theory. I did a course on this 35 years ago where I studied about 1, 2 and 3 dimensional arrays of the characters of the English alphabet. I vaguely remember that, if you take a given 2-letter combo, say, "ed", there's a very high probability that the next letter is a or e or u, etc. and very low probability that the next letter is b or c or x, etc. Therefore, the English speaker's eyes are conditioned to read the word, say, "edxcated" as "educated" and miss the typo during proof reading. I also remember that compression technology leverages this principle. The sender sends "ed cated", saves one character, and the receiver is easily able to replace the blank character with u on the other side. (Of course, spell check programs would easily be catch this typo).

            1. OCR

              Re: you often see what you expect to see, not necessarily what is really there.

              I once had to validate an Excel file (30k entries) against the original hard copy. I used the text to speech mode to read the Excel fike, while I marked the discrepancies on the hard copy.

        3. katrinab Silver badge
          Unhappy

          Re: Proof reader

          Another problem with sending stuff in Word format is that it changes "quotes" to “quotes”. Very difficult to spot between the difference ascii 22 and unicode 201C/201D on screen.

          1. swm

            Re: Proof reader

            I run a website with submissions from everywhere. I have a program that spots anything not in the ASCII range x20 - x7F. It finds all of the "funny" characters and escape sequences that I would probably miss.

            1. Soruk

              Re: Proof reader

              Why would you be allowing Delete? Surely the high water mark should be 0x7E?

          2. Robert Carnegie Silver badge

            Word and quotes

            You can switch off the Smart Ypants behaviour in Word, or, type " and ' then immediately press Ctrl+Z to undo the "secret" mutation of what you typed.

          3. anothercynic Silver badge

            Re: Proof reader

            Smart quotes, as the U201C/D ones are called, are the devil. Yes, it's pretty, it looks nice in print, but FGS, don't give me smart quotes when I'm working in plain text in configuration text files. Yes, certain OS vendors with fruity names, I'm looking at *you*.

          4. Anonymous Coward
            Anonymous Coward

            Re: Proof reader

            Ugh. I frequently have to write things up in Word to export to another system that only understands ASCII. Microsoft "Smart Quotes" are one of the worst ideas ever... I ended up writing a macro that changed them BACK to real quotes (single and double), and identified any other characters that the other system wouldn't be able to display.

      3. Arthur the cat Silver badge
        Unhappy

        Re: Proof reader

        'form" <--> 'from" is my, erm, favorite.

        My bugbear is "now" and "not". Spellchecker proof and instant sentence meaning inversion.

        1. J.G.Harston Silver badge

          Re: Proof reader

          ~~ed and ~~es catches me. 's' and 'd' are next to each other, and I type them without realising and my brain edits my vision so I see what I thought I types... sorry typed.

          Sometimes I even type the complete wrong pudding and even though my brain sent "word" to my fingers and the tactile feedback reported that "word" was typed, and my vision confirms that "word" was typed, I come back a few puddings later and it's completely pudding.

          1. MJB7

            Re: Proof reader

            I have a problem typing German. I'll think "und" or "oder" but my fingers will type "and" or "or" (and of course it's invisible to this native English speaker).

            1. A.P. Veening Silver badge

              Re: Proof reader

              If/when your German gets a little better, it will become glaringly visible to you as those English words won't fit the German language pattern.

              1. DavCrav

                Re: Proof reader

                "If/when your German gets a little better, it will become glaringly visible to you as those English words won't fit the German language pattern."

                I don't know about that. I know Germans who've spoken English for 30 years, and I still find 'und's in their slides. It's the same number of letters, and almost looks the same, which does not help.

            2. anothercynic Silver badge

              Re: Proof reader

              Same here, @MJB7... Writing something in German (whilst thinking in German), then re-reading the email before sending it and going "WTF is this English sentence doing here??"

              Story of my bloody life.

      4. J.G.Harston Silver badge

        Re: Proof reader

        I've lately been doing of <-> for of some reason.

      5. Anonymous Coward
        Anonymous Coward

        Re: Proof reader

        My first time coding in C in University was writing an assembler program. Because I was parsing assembler code I of course had a variable named "lable". Yes I spelled it wrong.

        And being the first time writing in C I had lots of syntax errors. After fixing most of the errors I got down to fixing "Error - lable is not defined". And I kept thinking why doesn't it tell me which label is undefined.

        1. FIA Silver badge

          Re: Proof reader

          And being the first time writing in C I had lots of syntax errors. After fixing most of the errors I got down to fixing "Error - lable is not defined". And I kept thinking why doesn't it tell me which label is undefined.

          So, if you paste a message from Microsoft Lynq1 (and a suprising other number of programs once you know to look for it) into a unicode aware text editor it will helpfully put one of the non printable unicode code points in the output.

          These, as implied, don't print, even on screen, and are zero width.........

          It took a day to work out why the pasted code snippit wouldn't compile, and it was basically down to someone noticing that the cursor didn't move right occasionally when single stepping along the line with the cursor keys.

          To be fair, the compiler tried it's best to help, but 'unexpected char' errors don't help if you can't see it. :D

          1. <Spit>

          1. Is It Me

            Re: Proof reader

            Our previous helpdesk software did this, so you would copy and paste an email address and it would fail and all sorts of other oddities

    2. Caver_Dave Silver badge
      Boffin

      Re: Proof reader

      I use a Chinese guy for reviewing my certification documents. Having grown up with a different character set, he seems to find mistakes that those more familiar with the character set miss.

      It apparently is all to do with the skill we learn as children, to read the whole word as one rather than individual letters, which makes us so bad at spotting transposition or even missing letters.

      1. Flocke Kroes Silver badge

        Re: Prelalal dnceodig

        If the fsrit and lsat lrtetes are cercrot the orehts can be in any oredr and you can siltl wrok out waht is mnaet qitue qilckuy.

        1. Flightmode

          Re: Prelalal dnceodig

          Wow; once I got past "cercrot", the rest of the sentence just flew by. QED, I guess!

        2. sw guy

          Re: Prelalal dnceodig

          Interesting, your sentence was difficult for me, as english is not my native language (nor is chinese: my native language uses latin alphabet)

          1. A.P. Veening Silver badge

            Re: Prelalal dnceodig

            I had the same problem, though once I deciphered "fsrit" and 'lsat", it was pretty clear to me, especially as I am familiar with the theory.

        3. Anonymous Coward
          Anonymous Coward

          Re: Prelalal dnceodig

          I don't quite know why, but this comment reminded me of the Morcombe and Wise sketch with Andre Previn....

          "I'm playing all the right notes, just not necessarily in the right order!"

          1. Anonymous Custard
            Trollface

            Re: Prelalal dnceodig

            Given the topic, I'd have to amend that to Morecambe and Wise sketch with Andrew Preview (as Morecambe famously called André Previn).

            They don't make 'em like that any more...

        4. Fr. Ted Crilly Silver badge

          Re: Prelalal dnceodig

          Hey taht lokos lkei the seped erding davrts taht sued to be no eht unrdegruodn ehwn I saw a yuoht itnesetrnig sey? wot nac pyla taht mgea tame! :-D

          1. Jimmy2Cows Silver badge
            Holmes

            Re: Prelalal dnceodig

            Close, but no bnanaa.

            I feel like you may have missed the subtle details behind the aforementioned theory...

            1. RegGuy1 Silver badge

              Re: Prelalal dnceodig

              What's he say?

              1. Rich 11

                Re: Prelalal dnceodig

                Dunno. It's all Greek to μου.

        5. J. Cook Silver badge
          Go

          Re: Prelalal dnceodig

          ow, that hurt to read.

          Also:

          Aye kun spel reel gud; huked ahn fonix werks fer meh.

          1. uccsoundman

            Re: Prelalal dnceodig

            It looks like you went to school in Kentucky LOL (I can say that; I'm from Kentucky)

            1. JWLong

              Re: Prelalal dnceodig

              I'm from southern Ohio and have to deal with both of you.

      2. disgruntled yank

        Re: Proof reader

        Hmm. Should we have a movie about the adventures of proofreaders, Trans Spotting? Anyone here speak Glaswegian?

        1. dak
          Headmaster

          Re: Proof reader

          Trainspotting was set in Leith, Edinburgh

          1. tuppence

            Re: Proof reader

            where you get very stange looks at the chippy if you speak with an English accest......

            1. Bdfh

              Re: Proof reader

              I love that the chippies there serve alcohol while you wait.

              and the mexican takeaway in leith does an amazing haggis buttito

            2. Anonymous Coward
              Anonymous Coward

              Re: Proof reader

              "where you get very stange looks at the chippy if you speak with an English accest......"

              In the late 1970s we were in a somewhat rough pub in Edinburgh renowned for its Scottish Independence support. One of our group was loudly expressing views on same subject - in what sounded like a very posh English accent. We hoped that the onlookers recognised it as the accent of the public school (in Fife?) that was the hotbed of Scottish nationalism.

              One evening I met some colleagues in an Edinburgh pub - almost immediately I was told by the landlord to leave. My colleagues explained later that my suit and English accent were a strong indicator of the London police drug squad - and the landlord was well aware of the dealings in his pub.

          2. eionmac

            Re: Proof reader

            Leith some time spelt by natives as Lite (old Gaelic version). PS I am a Leither.

        2. John Brown (no body) Silver badge
          Headmaster

          Re: Proof reader

          "Trans Spotting?"

          Really? That's a little politically incorrect these days, isn't it?

          Or are you being ironic considering we're talking about proof reading?

          1. Robert Carnegie Silver badge
            Joke

            Re: Proof reader

            Just a mistake, he or she or thing meant Trans Posing. ;-)

            1. Soruk

              Re: Proof reader

              That's where you shift to a higher or lower pitch...

      3. Terry 6 Silver badge

        Re: Proof reader

        Right, beer holding time.

        Despite various governments mandating Phonics as a reading method - under pressure from various lobbies who market that stuff for considerable profit- we don't actually make much use of individual letters.

        Just as we think we see all the elements in our field of vision so we think we see every letter in every word, and in sequence. We don't our;eyes fixate on key elements in words and phrases and our brain fills in the gaps, visual and semantic.When we read we are actually thinking we're seeing what we expect to be there. The behaviourist/Phonics stuff is pretty much a nonsense, forced upon education by a political lobby. And supported by an "it stands to reason dunnit" perception that that is what we do.

        In this context, proof reading, what this means is that we see what we expect to see. We think we're looking at every word/letter/symbol on the page, but we only truly see part and as long as the remainder isn't jarringly different we just perceive it as being what it ought to be. The wavy red line under misspellings may even be a potential disadvantage, making us less likely to perceive from/form errors etc. because we don't take the extra visual step of looking closely for errors. I was taught to proof read my stuff by starting with the last word on the page and looking backwards..Professional proof readers don't actually read the text they say

        Similar effects are seen in research on witness perception- like the famous one where a gorilla walks through a game of catch and the witnesses viewing a video of this can fail to see it, or the one where pursuers fail to see a a third person as they go past. And so on.

        1. Paul Kinsler

          Re: starting with the last word on the page and looking backward

          This is indeed an excellent proof reading strategy; I rely on doing at least one pass this way on everything in proof.

          1. RegGuy1 Silver badge

            Re: starting with the last word on the page and looking backward

            With (phone) numbers I first read in pairs, then check in threes.

        2. J.G.Harston Silver badge

          Re: Proof reader

          When I'm proofreading I force myself to examine and read each word in isolation to bypass the pattern-matching. I'm doing some now for a neighbour (native Chinese writer, so in addition I'm trying to wrestle it into structural English*) and she can't understand why my proofing is taking so long "I can read that page in half a minute, why's it taken you an hour?"

          *Eg:

          You've changed deluded into delusive, I want it to say deluded.

          But deluded is wrong, it's delusive.

          Deluded is in the dictionary!

          Yes, but. it's. still. wrong.

          1. Anonymous Coward
            Anonymous Coward

            Re: Proof reader

            ahh...but if she's cute the longer the better!

            I do understand though, my wife is Korean and we have endless conversations around words and phrases that as a native english speaker you would never normally try and decode. It is an interesting process, I enjoy it.

            1. A.P. Veening Silver badge

              Re: Proof reader

              I have the same problems interesting challenges with my Vietnamese wife and Dutch. As she graduated in English from the Ha Noi university, she thinks she understands European languages, making things even more interesting.

              1. Intractable Potsherd

                Re: Proof reader

                I can go one better - Mrs IP, whose first language is only barely related to English, is convinced that she knows English better than *any* native speaker.

        3. cornetman Silver badge

          Re: Proof reader

          > Despite various governments mandating Phonics as a reading method - under pressure from various lobbies who market that stuff for considerable profit- we don't actually make much use of individual letters.

          As I heard it, my bother-in-law was a victim of this lunacy. Set his literacy back months if not years.

          1. Bill Gray

            Re: Proof reader

            I have several brothers-in-law and get along well with all of them. You now have me wishing I had at least one sufficiently annoying to be referred to as a "bother-in-law". (Is there a word or phrase for a typo that's actually an improvement over what was originally intended?)

            1. Terry 6 Silver badge

              Re: Proof reader

              Maybe your bother-in-law needs to visit your mother -in-law.

        4. Jan 0 Silver badge

          Re: Proof reader

          You might not make much use of individual letters, but I was taught phonics, got on well with them and found it fairly hard to read that line with the transposed internal letters.

          1. Terry 6 Silver badge

            Re: Proof reader

            Teaching phonics will work- to some extent- and is almost essential as a part of learning to read. Kids need basic phonic decoding skills, for the occasionally completely unknown/unpredictable words,or unusual names, or for the first word in an unexpected phrase.

            Every method will work- to some extent. The way I was taught is the least effective of all (Look and Say- bloody awful method), but most kids learnt to read at that time. Our brains make the best of what they're given.

            Phonic reading/learning is inefficient, because our brains work faster at anticipation and self-correction than they do at focussed decoding, and our eyes do not normally scan linearly. also, in English the number of words that are either not phonically regular are contradictory or complex is considerable. e.g. I read a book today, I will read another book tomorrow" see also { lead dead head} v {lead mead bead}. Or post/lost/most. down/own come/some/home cone/done And so on. and so on.

            1. Irony Deficient

              Re: And so on. and so on.

              Recite the poem De Chaos for a larger sampling (note that its text was written for a 1920s en-GB pronunciation).

              1. Terry 6 Silver badge

                Re: And so on. and so on.

                Yes, there are so many of these.

                Another.

                When the English tongue we speak.

                Why is break not rhymed with freak?

                Will you tell me why it's true

                We say sew but likewise few?

                And the maker of the verse,

                Cannot rhyme his horse with worse?

                Beard is not the same as heard

                Cord is different from word.

                Cow is cow but low is low

                Shoe is never rhymed with foe.

                Think of hose, dose,and lose

                And think of goose and yet with choose

                Think of comb, tomb and bomb,

                Doll and roll or home and some.

                Since pay is rhymed with say

                Why not paid with said I pray?

                Think of blood, food and good.

                Mould is not pronounced like could.

                Wherefore done, but gone and lone -

                Is there any reason known?

                To sum up all, it seems to me

                Sound and letters don't agree.

                1. RegGuy1 Silver badge

                  Re: And so on. and so on.

                  Wow!

                2. Cheshire Cat
                  WTF?

                  Re: And so on. and so on.

                  Because English is not one language, it is at least 3 (anglo-saxon, french, norse, germanic, greek, latin, and anything else it can steal)

                  English is three languages in a trenchcoat, that follows other languages home, knocks them down in a dark alley and then goes through their pockets for loose grammar

                  (quote courtesy of Reddit)

                  1. Terry 6 Silver badge

                    Re: And so on. and so on.

                    Partly that. Partly that standardised spellings were a late-comer to the party. Early dictionary writers selecting what they thought would be the best spelling for any given word - not always agreeing and not always keeping to that choice anyway, as fashion, pronunciation and errors crept in.

                  2. Intractable Potsherd

                    Re: And so on. and so on.

                    English is the result of Norse speakers learning Latin to talk to priests, French to talk to landlords, and German to speak to barmaids!

            2. Cliffwilliams44 Silver badge

              Re: Proof reader

              The fact that English is a bloody awful language to read does not invalidate phonics as a proven method of teaching reading. We would be better served by fixing English than throwing out phonics.

              If you think Educators of today are motivated by the success and development of your children over their political ideology than you are sadly mistaken.

              1. Terry 6 Silver badge

                Re: Proof reader

                In other words, since your Phonics system doesn't work that well in the real world, change the world.

                (Hint, most people learn to read English whatever the method taught).

            3. cornetman Silver badge

              Re: Proof reader

              > Teaching phonics will work- to some extent- and is almost essential as a part of learning to read.

              I think what went wrong in this case was that my brother-in-law was taught phonic-based spelling. So he was taught a completely different written vocabulary which he then had to unlearn before he could read what the rest of us read.

              Languages are messy, and English terribly so. By and large our intuition is phonic-based and we learn all of the exceptions, but children are perfectly capable of acquiring language quickly in the time-honoured manner of seeing and hearing lots of it, in conjunction with some instruction as to the rules when they are capable of understanding them.

              It was a trendy idea that in retrospect is completely misguided.

              1. Terry 6 Silver badge

                Re: Proof reader

                Our writing is almost wholly muscle memory based. (Hint, we only try to spell words we can't write).

                The Behaviourists that mandate Phonics for reading also expect it to be used for spelling- where it works even less well since then the unreliability and inconsistency of the rules is further complicated by both differences in pronunciation one-to-many mapping. We don't all make the same sounds* for words and there are umpteen ways of encoding many sounds.

                *The only rs in "bath" are the ones that get washed.

        5. Cliffwilliams44 Silver badge

          Re: Proof reader

          Phonics is a time proven functional method of teaching reading. Whole word reading is a method of producing functional illiterate sheep!

          We bought our grandson a toy when he was 2, it was a electronic toy in the shape of a school bus with letters and an LCD display and a voice output. It taught basic phonics, By the time he was 3 he could read anything!

          1. Terry 6 Silver badge

            Re: Proof reader

            Second part of your assertion has some truth, subject to defining "whole word". As I'd commented, the "Look and Say" approach I was taught with was not good ( though it still worked for me and most others). That (false) comparison though is often used as a way to justify the Phonic method.

            The actual alternative to phonics (which is a Behaviourist approach) is rooted in cognitive psychology and is far more effective. The "evidence" for Phonics is almost all circular, i.e. teach kids to decode and test them on their decoding skills.

            Since you make your assertion in such a dogmatic way perhaps you'd lay out your qualifications.

            Mine- psychology degree, PGCE for psychology graduates specialising in the teaching of reading, thirty + years as a reading specialist with 20+ of those years receiving formal regular training in the latest research in literacy development. Deputy head teacher and literacy specialist. And post retirement was a local authority advisory teacher (reading) for a couple of years.

      4. Jon Bar

        Re: Proof reader

        My father was a design engineer for Boeing Vertol. He used to get my mother (a social worker) to proof the operators' manuals for the helicopters for much the same reason.

        Or, as my classic debugging maxim has always been (can't remember if I owe it to SKB or GLS) "It takes two people to debug any program, only one of which needs to know what it's doing."

        1. Anonymous Coward
          Anonymous Coward

          Re: Proof reader

          "He used to get my mother (a social worker) to proof the operators' manuals for the helicopters for much the same reason."

          There was a time when we had to submit our handwritten documents to the manager's secretary for typing. Invariably "modem" came back transcribed as "modern".

          There was a humorous Punch article about creating a new car brand. Someone suggested "Norris" - to be rejected as "Morris" was already taken. IIRC - the twist was that "Norris" had also once been a brand.

          1. Glenturret Single Malt

            Re: Proof reader

            My PhD thesis was largely concerned with compounds of cobalt (Co) with carbon monoxide (CO). Since this was in the days long before word processing, you asked a friendly secretary to type it for you. I am sure you can imagine the number of corrections that had to be made because of confusion between the two.

      5. FIA Silver badge

        Re: Proof reader

        It apparently is all to do with the skill we learn as children, to read the whole word as one rather than individual letters, which makes us so bad at spotting transposition or even missing letters.

        After having a detached and torn retina a year ago I've become acutely aware of how much of this the brain actually does.

        Whilst my vision is now back, it's not quite 'the same' and as my brains been figuring it all out I have has several instances over the last year where I've simply 'seen' the wrong word, in a much more prominent sense than I ever used to misread words in the past.

        Generally this is just a 'wuh' moment, but every once in a while the unintentional substitution can be a deeply disturbing insight into the inner workings of my brain that I'd rather not have. (I'm just happy that me and the folks at the local fishing club have agreet to put the incident behind us and never speak of it, or to each other, again).

        Amazing thing the brain when it comes to language.

    3. Evil Auditor Silver badge

      Re: Proof reader

      I'd be happy if I misspelled etc. My problem is that x and c are neighbouring keys that my fingers regularly mishit and me having to write security all too often. At least, it's easy to spot.

    4. Anonymous Coward
      Joke

      Re: Proof reader

      It's surprising how many people don't know that 'ect' is a contraction of 'Ectoplasm'.

      And now you'll be reading it as that whenever you see it in the future. You're welcome. ;)

      1. Robert Carnegie Silver badge
        Joke

        chiz chiz chiz

    5. Kubla Cant

      Re: Proof reader

      Before university I spent some time working as a proof reader at a company that built nuclear power stations. I still wince at the ignorant corrections I tried to make to stuff that I was too uneducated to understand, but I can't have made many serious mistakes because the power stations have all survived to be decommissioned.

      The result of nine months proof reading was that by the time I went up to university my ability to read normally was impaired, with lots of sub-vocalising and spelling out of words. (Well, that's my excuse for not doing much of the reading required for a degree.)

      1. Jimmy2Cows Silver badge
        Mushroom

        Re: Proof reader

        Missed icon opportunity if ever there was one...

    6. Pascal Monett Silver badge

      Re: Proof reader

      A second set ? On one occasion, it took a third, and fifteen minutes for somebody finally twigged that the server name had been written with commas instead of points.

      It had been staring at us for two whole hours !

    7. Gene Cash Silver badge

      Re: Proof reader

      I've got the exact opposite problem. Misspellings pop out at me and annoy me to the point I find it hard to concentrate on the text.

      I had a English Lit friend, and I used to annoy the hell out of her by instantly spotting errors in her papers.

      1. Anonymous Coward
        Anonymous Coward

        Re: Proof reader

        "I had a English Lit friend, and I used to annoy the hell out of her by instantly spotting errors in her papers"

        Not surprising if she was quoting something before the attempts at standardisation of English spelling in the 18th century. Prior to that people tended to write as they spoke. Reading 14th century Chaucer is a perfect example of the early literary use of English instead of Norman French.

        It was a monk who "corrected" English spelling in a way that ignored the etymology of some words. Faced with "should" and "would" - it seemed logical to him that "coud" was to be changed to "could".

        1. Terry 6 Silver badge

          Re: Proof reader

          This week's New Scientist quotes some -presumed algorithmic - proof read article saying a dog should have "...all 4 feet (1.2m) on the floor".

    8. leadyrob

      Re: Proof reader

      The best proof reader I've ever worked with, was completely blind.

      When you rely on screen readers and braille displays you will quickly identify all of the typos and errors.

    9. Robert Carnegie Silver badge

      Re: Proof reader

      I actually don't type any more when I can avoid it, but "helath" used to come out frequently when I meant "health".

      At the moment, I'm on a touchscreen where a reach to the left for "a" is quite likely to hit "s", for "s" hit "d", and so on. Yed, wuite likely.

  2. chivo243 Silver badge
    Go

    Doubtless with the assistance of a baseball bat peppered with rusty nails.

    I thought this was on call? Not the BOFH! Although I have to believe that Simon would be using a cricket bat? No?

    Great start to the weekend!

    1. A K Stiles
      Alert

      Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

      Ah - the trusty clue-stick! In a previous existence we had a real clue stick mounted on a rack on the wall of the IT office next to a sign labelled "In case of emergency". It was taken down and handed to new starters in the office so they could appreciate its impressive heft. That was the only time it was taken down, and it never left the 4 walls of the office, I suspect just in case the temptation to actually use it was a little too strong...

      (icon sort of clue stick shaped)

      1. Michael

        Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

        Ah, I remember visiting a former HQ building to work with the QA department to train them on some new software. On the wall was a frame with the message for repeat offenders and a small pistol. A new team leader happened to start working in the QA department that week. She picked the pistol off the wall and pointed it towards the room. Everyone but me hit the floor and the QA manager moved at high speed from his desk and pointed her hand upwards.

        Apparently nobody could remember if it was still loaded with live rounds. They had been shooting with it recently. After that I decided to avoid visiting QA. Or putting my name on code commits....

        1. Lord Elpuss Silver badge

          Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

          I'm not sure who should be fired/prosecuted/shot first - the person who decided to put a (potentially live) weapon in a workplace, or the person who decided to wave it around. Always treat firearms as loaded, never aim at anything you're not intending to shoot etc. Unless the TL actually was intending to shoot somebody; you never know.

          Either way not a place I'd like to be working in.

          1. Evil Auditor Silver badge

            Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

            You can't expect people to know how to handle a firearm. So, lock it in, ffs. And get rid of the person responsible for having it there.

            1. shedied

              Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

              A poster would be sufficient, but not everything has to be in 3d, FFS

          2. J.G.Harston Silver badge

            Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

            Even a firearm that you have witnessed being unloaded is loaded until you yourself have it in your possession and have ensured it is unloaded.

            And even then....

            1. Anonymous Coward
              Anonymous Coward

              Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

              I knew a guy who was cleaning his unload rifle. His TV died when the rifle went off.

              He was a newspaper guy so used the mishap for a story on gun safety.

              1. A.P. Veening Silver badge

                Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

                I knew a guy who was cleaning his unload rifle. His TV died when the rifle went off.

                You mean he did unload his riffle into his TV. Or was that supposed to be an unloaded riffle?

              2. Vincent Ballard
                Alert

                Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

                The previous king of Spain killed his elder brother in a similar accident. Or "accident" for those who disbelieve the official story.

          3. Anonymous Coward
            Anonymous Coward

            Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

            "Either way not a place I'd like to be working in."

            In my distant youth I had a summer stint on an kibbutz in Israel. One got used to weapons at hand in people's apartments in case of surprise attacks. However - no one waved them about otherwise. Presumably all the owners had done their compulsory weapon training in the army or even at secondary school.

            1. Lord Elpuss Silver badge

              Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

              AllI Israelis go through compulsory basic training from age 18. Doesn't cover kids obviously, but anybody there who owns a firearm should know how to use, keep and maintain it.

        2. Fr. Ted Crilly Silver badge

          Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

          Ah the Officers mess Webley...

      2. David Robinson 1
        Coffee/keyboard

        Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

        I prefer an IBM Model M keyboard. You can run them through a dishwasher to remove blood, allegedly.

        1. Symon
          Devil

          Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

          Yes, and still use it to write their obituary.

          1. GrumpenKraut

            Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

            Plus, using it with a chisel, to prep the tomb stone.

            1. Jimmy2Cows Silver badge
              Thumb Up

              Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

              All the best LARTs are multipurpose. Plus, it's always helpful to have an innocent explanation ;-)

      3. amacater

        Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

        An American colleague gave me an unusual aluminium ruler as a gift. It has the shape more or less of a palette knife with rounded corners - I've no clue what the scale actually represents because it's not obviously linear.

        Her advice was to walk up behind people at their desks and THWAP it down hard. It's non-lethal, creates a loud noise and attracts their attention. Best clue stick I've had.

        My favourite "bleeding obvious" - 1998 or so, building a Linux computer for another colleague. Build goes OK but I can't get it seen on the network. Call in the (senior) colleague to troubleshoot who smiles, reaches round behind the computer and holds up one end of an Ethernet cable: "Would this help?" :)

        1. Evil_Goblin

          Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

          Non linear rulers can be used in education I think, I vaguely remember an enthusiastic lecturer getting all "philosophy of science" with one, illustrating importance of standards, they only work if we all stick to them, accumulation of errors etc.

        2. MCPicoli

          Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

          Had the unpleasant experience of pointing to an "network expert" that the thin coax cable that came from one computer went... to the same computer, thus explainig "why" it was offline. Remember the ancient thin coax networks with those small "T" pieces and coloured terminators...

          Curiously somehow the two adjacent computer (on the same desk) were networking just fine, it just happened that in the rat's nest mess of cables behind them the coax from the first went directly to the third and the cable from the third went on to be connecting the middle one to itself.

        3. Anonymous Coward
          Anonymous Coward

          Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

          "Her advice was to walk up behind people at their desks and THWAP it down hard."

          In the Radio Club in the 1960s there was a work bench covered with sheet steel. An old timer had the habit of waiting until someone was engrossed in the innards of a receiver or transmitter - then hitting the bench surface with a hammer.

          The same person was once asked to devise a sound system for an empty church belfry. The then new technology consisted of a large amplifier and an auto-change record player loaded with a stack of 78rpm records of bell peals. On the day it went live there were crowds outside to hear this wonderful innovation. He listened with them as it started - then left as he couldn't bear the suspense. Next day he went to the record player - and on the stone floor were the smashed fragments of his unlabelled record of "Tiger Rag".

          1. Anonymous Coward
            Anonymous Coward

            Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

            A coworker once related the tale of his first 12-hour night shift. He dozed off sitting at a big metal desk. His supervisor snuck up behind him and hit the desk with a big metal ruler. "I'M AWAKE, I'M AWAKE!!!"

            I was doing some troubleshooting with a different coworker on a million-dollar DCS cabinet, which was (even at that moment) running significant portions of a pharmaceuticals manufacturing facility. Being my first time in one of these cabinets, I was understandably nervous. As my multimeter's leads touched the terminals, he clapped. One time, good and loud. I could have throttled him...

    2. Anonymous Coward
      Anonymous Coward

      Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

      Peppered? You don't pepper someone with one, you a-salt them.

      I'll get my coat, it's the one with the pockets full of condom mints. =-)p

      1. W4YBO

        Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

        Bad puns bring back memories of watching "Hee Haw" with my grandparents.

        Pi r square. No, pi r round. Cornbread r square.

        1. Anonymous Coward
          Anonymous Coward

          Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

          "Pi r square. No, pi r round. Cornbread r square"

          Heh heh heh. Now here's Roy

    3. Anonymous Coward
      Anonymous Coward

      Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

      I thought this would be a bit overkill given someone actually confessed, a rare quality of admitting fault shouldn’t be punished so harshly.

      Maybe a truncheon with some drawing pins?

      1. doublelayer Silver badge

        Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

        True, as offenses go, this is relatively minor and doesn't deserve that much punishment. That said, "major changes to config file, don't check it for viability, send directly to production, go home" is a bad enough mistake to require some unpleasant talks about how that's never going to happen again.

  3. A K Stiles
    Angel

    Nope, never, not me...

    Definitely never been responsible for amending a config file then forgetting to reload the config before a weekend / week off causing a system to crumple into a heap and refuse to start when, a few days later, it was subjected to the patching updates schedule. Definitely not me...

    (more than once per job role)

    1. GrumpenKraut
      Boffin

      Re: Nope, never, not me...

      Improvement on that one: Customer installs Linux on his machine in rack and sets up web server and so on. Several month later a power failure. Machine would not come back. Turns out everything was set up in the install system running in RAM and on RAM-disk, absolutely nothing on the disk proper.

      Took a good while to figure out what happened.

      1. hoola Silver badge

        Re: Nope, never, not me...

        Yep, had that one as well.

        Trying to figure out why a proof-of-concept just vaporised over a weekend.

    2. John Riddoch

      Re: Nope, never, not me...

      We had a process where before any major changes/patching we should reboot the server just in case something had been left for the unwary. If the app didn't start after a simple reboot, we knew there was an issue NOT caused by our change. Without that step, any issues found after the change would be attributed to our change and Our Fault with all the attendant Blame being flung our way, especially if we had to back out and it still wasn't working.

      1. Jay 2

        Re: Nope, never, not me...

        Once upon a time when I could actually escape the office to go on a course I was doing a RHEL refresher/certification and it was pretty much drummed into us that there are many ways to change things on Linux boxes, but don't forget to make the changes permanent so they they'd survive a reboot.

        We've got some kit running an in-house mutation of Linux which rebuilds itself upon boot. So for example we can add routes on the fly, but the best way to make sure is to change the config, reboot (and therefore rebuild) and check.

      2. Anonymous Coward
        Anonymous Coward

        Re: Nope, never, not me...

        "It's not tested until the machine has been restarted"

    3. stungebag

      Re: Nope, never, not me...

      I tried to be careful to make sure updated configurations were deployed as soon as possible. And I only slightly deviated from the truth when I managed to reboot the core switch during the day, taking down an entire school with its 1400 staff and students. Anyway, it was all back in ten minutes or so, and the headteacher didn't lose that much work.

      1. Giles C Silver badge

        Re: Nope, never, not me...

        Find an IT engineer who has never taken down a system inadvertently and you have either found a liar or someone who hasn’t done any real work.

        Anyone who works in IT will have encountered that sinking feeling as something stops responding as you are working on it.

        Especially true for network engineers....

        1. Red Ted
          Facepalm

          Re: Nope, never, not me...

          Anyone who works in IT will have encountered that sinking feeling as something stops responding as you are working on it.

          In my case generally followed closely by the smell of the "Magic Smoke" that has just leaked from the device under test!

        2. Giles C Silver badge

          Re: Nope, never, not me...

          Or put it another way every engineer should have a who me story in them somewhere....

        3. Anonymous Coward
          Anonymous Coward

          Re: Nope, never, not me...

          "Especially true for network engineers...."

          #rel in 10

          then do your config changes. If the network doesn't go TITSUP*, then you do a "wri mem" followed by "rel can". If your changes swamp the router or cause you to lose connection, look innocent, wait for someone to complain about the network. Use the phrase "huh, that's weird, let me check" and delay as necessary until the router reboots.

          The same mitigation steps apply if your changes didn't foul things up, but you forgot to cancel the reload.

          * Friday malady of the admin TITSUP: Thinking Is Tracked Somewhere Unrelated Presently caused the router to go TITSUP: Traffic Is Transited Somewhere Unavailable Presently.

        4. swm

          Re: Nope, never, not me...

          "Anyone who works in IT will have encountered that sinking feeling as something stops responding as you are working on it."

          And patching real time computers on the fly. Hit return and no line feed.

    4. Anonymous Coward
      Anonymous Coward

      Re: Nope, never, not me...

      I've done something similar with firewall rules....

      The First rule of firewall rules is to allow list your own IP address first.

      The Second rule of firewall rules is to save the allow list rule for your own IP address......

      That was an embarrassing phonecall to the techies at the data centre to talk them through why only port 80 and 443 was working...

      Anon to save blushes all round...

    5. amacater

      Re: Nope, never, not me...

      I have learnt from my mistakes. With practice, I can now repeat them 100% reliably and in half the time ... :)

    6. DS999 Silver badge

      NEVER

      Make a change to a config file if you can't reboot/reload it after doing so. If you just accumulate changes that will "take effect later" finding out what borked it becomes orders of magnitude harder.

      If you want to accumulate changes, you make them to a copy of the file, and then when the reboot happens part of the procedure is to put that changed file into place. If something goes wrong, you know where to look and have the old file as a fall back if you don't have time to investigate the 'diff' output between the two to see what went wrong.

  4. Anonymous Coward
    Anonymous Coward

    I can relate to that

    Ah, the heady days of my youth on the electronics test benches of a maker of control and instrumentation products.

    As part of the banter between benches, it was a common thing to walk past someone struggling to locate a particularly recalcitrant fault, pick up a PCB, eye it for a second or so, put it down and declare "oh, I see it" and walk it off.

    The thing is, so often that was actually true (not always. That would have taken the fun out if it). To the proverbial fresh pair of MK1 eyeballs, the cracked track, whisker of solder or a bit of spurious copper left by a hair on the tracking photo mask, were all too obvious.

    That was in the days before surface mount micro components, when all you needed was a decent soldering iron, a roll of tinned copper wire, desoldering braid, a solder sucker and asbestos finger tips.

    Happy days

    >Waves at anyone remembering Protech Instruments in general and Emu in particular<

    1. Flocke Kroes Silver badge

      Re: I can relate to that

      A beginner did an excellent job of etching, drilling and soldering a one-off two sided PCB but did not understand why it did nothing until I mentioned that the holes do not conduct electricity by themselves.

      I wasted half a day debugging a 2.5V circuit until I noticed the 5V version of a chip had been installed. Some helpful person had fixed an "out of stock" problem by calling the supplier. Apparently many customers did not mind which chip was supplied because the 2.5V chip was designed to work in 5V circuits.

      I busted a few prototype PCB's before I noticed the big inductor for the switched mode power supply was very obviously not big at all. Someone had helpfully dealt with a late part by authorising a different one with the same inductance.

  5. GrumpenKraut
    Facepalm

    ~/.procmailrc

    Something like

    :O:

    * ^Received: .*\.sendgrid\.net

    $SPAM

    did not work. Fresh pair of eyes said the line :O: better be :0:

    Since then I make sure to only use fonts that have a dot within the character zero.

    1. chuBb.

      Re: ~/.procmailrc

      Reduced user login support calls by 90% and started a still ongoing turf war with marketing, by ignoring the style guide and editing with extreme prejudice the style sheet that choose a wanky modern font which displayed i L 1 and O 0 the same on all form fields and password emails for a clear monospaced one....

      1. A.P. Veening Silver badge

        Re: ~/.procmailrc

        Easily won by making sure such forms and emails have the marketing approved style and font when viewed by resp. sent to the marketing droids. Their problem when they mistype the new password.

      2. GrumpenKraut
        Angel

        Re: ~/.procmailrc

        To marketing person: "Your new password is 1lıl¡1Ii!!ıl|¡|I".

        1. Aladdin Sane

          Re: ~/.procmailrc

          Great, now I have to choose a new password.

        2. Sgt_Oddball
          Devil

          Re: ~/.procmailrc

          Oh that's evil....

          I like it.

        3. Must contain letters
          Coffee/keyboard

          Re: ~/.procmailrc

          Well done, first this year :)

        4. shedied

          Re: ~/.procmailrc

          "It's a barcode, no one can crack it. You'd have to eat the email once you print it out."

          1. Inventor of the Marmite Laser Silver badge

            Re: ~/.procmailrc

            And there are people out there who WILL believe you.

        5. Jou (Mxyzptlk) Silver badge

          Re: ~/.procmailrc

          Upvote for including the | = Pipe symbol. I hate Arial/Helev for that nonsens.

          Wow, and it should have double upvote for including ı = dotless i - I just saw that and had a real "WTF why does such nonsense even exist" moment after sending that character through google to "WTF is this".

      3. Flightmode

        Re: ~/.procmailrc

        Whenever I generate a new password that I will need to type in manually (by now, that's pretty much only the AD password for unlocking the PC), I always copy it out from my password manager to a Word document and look at it in 72pt Times New Roman. That usually helps clean up any ambiguous characters before they're erroneously committed to memory.

        1. diguz

          Re: ~/.procmailrc

          i do exactly the same, but my preferred font is Calibri...

      4. Terry 6 Silver badge

        Re: ~/.procmailrc

        You have just summed up the issue that so damages so much modern technology, and not just in IT,. i.e. A functional system being totally fucked up by a marketing dept. who think it's all about the look (form over function). Whether it's the often noted ( in El Reg at least) invisible [on] switch hidden in the moulding or stupidity like the display and controls for my fridge freezer being hidden inside the fridge section, with a concealed hatch cover over the actual controls (in case it's too obvious where to find it).

        And of course the generations old one of putting the serial number in the least accessible place - even though it is going to be needed any time the manufacturer or supplier is contacted. It goes on.....

        1. MCPicoli

          Re: ~/.procmailrc

          Positive example restoring faith in humanity is Dell's Service Tags. Almost always easily readable and almost always in easy-to-see places.

          1. SuperGeek

            Re: ~/.procmailrc

            And embedded in the BIOS so no need to flip the machine over. Unless the board's been changed and the tag not transplanted :)

        2. Jou (Mxyzptlk) Silver badge

          Re: ~/.procmailrc

          > And of course the generations old one of putting the serial number in the least accessible place

          Like on top or on the side (where the rail-mount is) of the rack mounted server, which cannot be moved while operating since SOMEONE was too stupid to do it the right way in first place.

      5. Giles C Silver badge

        Re: ~/.procmailrc

        Whenever I am writing a change request with networking code I always put the code in the document in courier new. Simply because it is one of the easiest to read fonts for numbers, commas and you don’t have the i l 1 problems

    2. Doctor Syntax Silver badge

      Re: ~/.procmailrc

      Where I started out the standard for submitting work to the card punch operators was to cross the letter O rather than the numeral 0. Maybe it encouraged students to be particularly careful when they moved into the wider world.

    3. Christoph

      Re: ~/.procmailrc

      Done that with the very first program I ever wrote. The program itself was fine. But I was punching my own cards from the coding form rather than sending it in to the punch girls (yes, 'girls' - this was the 60s). I managed to try to send output to LPO instead of LP0.

  6. GlenP Silver badge

    Around 35 years ago as an analyst/programmer I learnt the importance of "fresh eyes".

    Working with DEC RDBMS (I think that's correct) I couldn't get one particular routine to run. After extensive debugging I referred it to DEC support who came back almost immediately, I'd used a reserved word as a variable name. The error messages could have been more helpful but I should have spotted it.

    Same role my boss was struggling to debug a Cobol program and asked for help. Despite the fact my Cobol knowledge was minimal (at the time my experience was Basic, Pascal and Fortran) I spotted her mistake immediately. Brownie points for me, shame for her but easily done.

    1. Tom 7

      One of the first think I used to do was to create a file of all the reserved word in a language so that something could be done with them - editor highlighting or a shell script to grep them on save. That was before it took longer to learn how to program the editor than it did several languages. Thank god error reporting has improved faster than documentation.

      1. J.G.Harston Silver badge

        That's all ok if you can *find* a list of reserved words.

  7. chuBb.

    There is a lot to be said for only using text editors configured to display comments in a different colour especially multiline day ruiners committed by whitespace phobic devs...

    1. Tom 7

      Using a version control system and Diff for anything going near production is handy too. When working from home your cat can get paid for those mods he's done while you make a cuppa.

  8. Phil O'Sophical Silver badge
    Coat

    At least he'd used the right comment character

    It's amazing how much chaos you can cause by commenting out a line with a # when the daemon in question uses a semicolon for comments...

    1. GrumpenKraut
      Pirate

      Re: At least he'd used the right comment character

      Can beat that: editing /etc/inittab with an office program. The "computer expert" of a certain company did that. He did not understand why that was a Bad Thing and I could not explain it to him (him: "I looked at it, everything was fine! Yes, I renamed the file to end in .txt, you said I should do that, right?").

  9. MrMerrymaker

    a shamefaced colleague stepped forward to confess to the deed.

    What a dunderhead this fellow was / may well still be!

    I've never done this, nope sir, nope nope nope (other sounds of protesting too much)

    1. Tom 7

      Re: a shamefaced colleague stepped forward to confess to the deed.

      Its always worth noting that even in the days of floppies etc there is almost always a way that intelligent forensics will reduce the possible perpetrators to a manageable level. I've even managed to convict myself of a Friday afternoon cock-up after I left someone I detested's leaving do early and returned to the office to wait for a lift home. I have no memory of the incident but I figured it can only have been me but no-one noticed I'd gone and it was a couple of weeks later when the system crashed and the date stamp on the file probably didnt lie. No-one else worked it out though.

  10. MJI Silver badge

    The value of not working all hours

    I have seen the effects of huge hours on productivity over sensible hours.

    Little difference in the end.

    On those 12 hours days you pace yourself (go slower), you make more errors, you brain struggle near end.

    Normal day plus two hours on a Saturday remote gets a lot more done.

    1. Tom 7

      Re: The value of not working all hours

      In my first proper job we worked on things like Vax780s and things of less power.That, in my job, left a lot of waiting time for jobs to finish and I used it to learn new skills. Because of the nature of the job you could do create some moderately accurate productivity data. Over about 6 years I worked our a continuous 37 1/2 hr week was the most productive. I could do a couple of weeks of 60hr weeks to get a job finished on time and then it would take a month of normal to get back up to normal. A month of 60hrs would lead to lower than normal productivity, largely due to recovering from errors but also from lack of enthusiasm.

      The real answer to loss of productivity in longer hours I discovered working in the US where 12 hr days 6 days a week seemed to be the norm: just dont do anything that might tire you out like work.

      1. Anonymous Coward
        Anonymous Coward

        Re: The value of not working all hours

        "[...] a continuous 37 1/2 hr week was the most productive."

        In my trouble-shooting days there were numerous problems that could be attributed to devs working a strict 9-5 day. The bug would be a piece of code that was left "dangling" at the end of their day. Next day they forgot to complete it - or didn't pick up the thread cleanly.

        My strategy worked on "breaking points". I either went home late or early - when a "clean" point in the coding/logic had been reached. I would often leave an easy bit to start with the next day - just to get myself running smoothly again.

        Like many trouble-shooters I had intrinsic mental stamina - possibly a "spectrum" ability. It made solving "impossible" problems look quick and easy - as other people didn't see the hours put in to finding/proving a root cause.

        The intellectual rewards felt good. As my doctor pointed out after retirement: irregular meals, irregular sleep, and stress are known triggers for Type 2 Diabetes - no matter how healthy you are otherwise.

    2. Flocke Kroes Silver badge

      Re: Two bankers boasting

      US division: We work more hours than you lot.

      UK division: We make more money than you lot.

    3. Mast1

      Re: The value of not working all hours

      Turn of the millenium; as a UK-based Brit on a usual 40 hour week, I did a week's stint with an LA-based company with the 12 hour days, but only 5 days per week. Noticed that they were doing a lot of fire-fighting clearing up previous mistakes.

  11. AndyMTB

    First thing I used to do when faced with a config issue was scan the file system for files modified in the last few days. Easy-peasy in unix, probably not feasible with Windaz registry, thankfully never was called to do anything serious on a click-click-drag o/s.

  12. Kevin Johnston

    Eye of an eagle

    While working on flight simulator visual systems I was supporting a senior engineer with testing some upgrades and one area was not working as expected. We traced the fault to a single card but everything about it looked good according to all the circuit diagrams etc. As we walked back to the test area to have another go we passed another engineer who shall be known as 'The Guru' as from us passing 6 feet away at walking pace he looked at the card we were carrying and said "that's the wrong chip type". Apparently one of the 40 or so discrete ICs on the board was a 74Fxxx and should have been a 74HCxxx. We changed it and all worked perfectly

    1. Doctor Syntax Silver badge

      Re: Eye of an eagle

      The trick to achieving this is to inspect the article at leisure when nobody else is about but pick your moment to reveal what you found.

    2. Boris the Cockroach Silver badge
      Boffin

      Re: Eye of an eagle

      Its more fun just to lean against the machine , and then say "Its about to blow up"

      The nay sayers will chorus, and about 20 secs later it goes down

      Impressed a machine salesman/tool seller after his 1/2hr speech about how this new tech was the dogs balls and save us umpteen 100's in production.

      Started the test job, it did 1 part and told him "it wont do 2".... "naaaayy it will do hundreds of " <BANG>

      Exit salesman stage left.

      sometimes its good to be a geek

  13. David Robinson 1

    CVS

    Policy is that all text config files are held in CVS or similar. You check out the latest version and make changes to that. Should anything untoward happen on deployment of changes, it makes it easier to roll back.

    1. Anonymous Coward
      Anonymous Coward

      Re: CVS

      <joke>

      Doc, I need a prescription so I can make an edit.

      </joke>

      (Explanation: CVS is the name of a pharmacy/chemist chain in the US.)

  14. Christoph

    I had a call that a web sales site had started giving weird errors. Got the client to send me their copy of the files, and found that:

    1> They'd added a call to an exterior function that was somehow trashing the login cookie.

    2> Someone had hashed out the line that checked that the cookie was still present (and kicked you back to the login screen if it wasn't) immediately before the call to the very complex SQL stored procedure that did all the charging.

    At a guess they'd put in the exterior call, got fed up with going to the login screen, took out the line that did that, ...

  15. PerlyKing
    Happy

    Pineapple on pizza

    What's wrong with pineapple on pizza?

    1. It's tasty

    2. It winds up Italian colleagues like nothing else :-)

    1. BenDwire Silver badge
      Facepalm

      Re: Pineapple on pizza

      I think enough time has passed for me to confess that I used to flip burgers for pocket money in the 1970s. It was a time of poor taste, and dodgy menus, but one of the offerings was the "Hawaiian Burger". Bun bottom, lettuce, cucumber, tomato,* burger, pineapple ring, bun top, in that order. We never sold many, but there were enough sales to keep it on the menu.

      * Every other type of burger (plain, cheese, egg) had a splodge of salad cream on the tomato, but this was prohibited when pineapple was present. Apparently customers would complain that it didn't taste good, so they must have had some tastebuds.

      No, it wasn't a Wimpey, but was very similar in most respects!

      1. Phil O'Sophical Silver badge

        Re: Pineapple on pizza

        Every other type of burger (plain, cheese, egg) had a splodge of salad cream on the tomato, but this was prohibited when pineapple was present. Apparently customers would complain that it didn't taste good, so they must have had some tastebuds.

        Of course. Hawaiian burgers should have thousand-island dressing, not mayo.

        1. BenDwire Silver badge

          Re: Pineapple on pizza

          I mentioned that this was in the 1970s. I didn't mention that this was in the UK, in the wilds of Essex. We never had mayo in those days (apart from a small jar at Christmas), and thousand-island dressing was still years away from the average salad dodger that ate in a burger joint.

          No-one can explain what salad cream is to people outside the UK: It's not mayo. It's not ketchup, but it's fantastic in a fish finger sandwich ...

          1. Mast1

            Re: Pineapple on pizza

            Salad cream : for something nominally regarded as bland by people outside of the UK, it runs a very close second to mayo on calorie count.

            So let's get this straight:

            bread: carbs

            fish finger : a thin sliver of fish (protein) covered in (golden) breadcrumbs and oil (fat)

            salad cream: fat

            Does one have to put butter on the bread as well in case the above is seen as the low-cal version ?

            1. BenDwire Silver badge

              Re: Pineapple on pizza

              There may well be better guidance on the internet (Google James May's instuctions on Man Lab, for example) but my preference is to use the salad cream in place of butter, as it acts as the perfect adhesive.

              Of course now I want one ... and I suspect, dear reader, you do too?

          2. shedied

            Re: Pineapple on pizza

            Some fast food joints have tartar sauce with some of the seafood/burger meals; could have been that.

      2. Intractable Potsherd

        Re: Pineapple on pizza

        The Hawaiian Burger was my post-nightclub go-to snack from the all-night burger place under the railway arches in Sheffield. A brilliant invention that I'd completely forgotten about until now!

  16. Doctor Syntax Silver badge

    One of my current problems involves reworking one of our out of print books as a PDF for our website. I scanned it to PDF and then ran ocrmypdf on it and copied and pasted from the PDF into a text file so I could correct OCR errors with vi before reading them into LibreOffice Writer. What I'd missed was that a lot of full stops had been OCRed as commas. I asked for volunteer proof readers and it was one of them spotted this. She sent me a complete list and I got through about 2/3 of it before running out of will to live. I've set it aside on the basis that I'm replacing the original blurred grey on grey photos with digital colour and need better weather for that.

    It would have been a doddle to have fixed them in vi first time round but I really couldn't face going back to the text files and having to re-do all the WP line spacing juggling to replicate the original page breaks (font metrics never really match up to the original).

    1. Down not across
      Pint

      OCR

      Oh, that reminds me all too well about when I was doing my bit for the legal export of PGP and helped with proofreading. So got my stack of photocopies and floppy disk(s).

      OCR in those days was nowhere near what it is now, so there were lots of corrections needed. There were times when losing the will to live was close. Persisting seemed worth the effort however.

      Icon: here's to you who have had to proofread

      1. Unicornpiss
        Happy

        Re: OCR

        When assisting a paralegal with how to make a scanned document editable, ran across a PDF scan that had been converted to text via OCR. Was mostly okay except where spell check had mangled "incompetent" and turned it into "incontinent", which adds a bit of a different dimension to the situation being described..

        1. Anonymous Coward
          Anonymous Coward

          Re: OCR

          My favourites OCR errors -

          "Burn" becomes "bum" and "click" becomes "dick"

  17. Blackjack Silver badge

    Evidently not done enough overtime

    Otherwise you learn to use the computer with one hand and eat using the other. Is it a bad idea? Yes. Can it end terrible wrong if you drop food or drinks in the worst place possible? Yes. Do you care after way too many hours of extra work? No.

  18. slimshady76

    This might qualify as an "On Call" piece on itself, given the tone of the actual one. About 10 years ago I got assigned a ticket for a memory leak in an AIX TSM server. The darn thing had to be rebooted every 4-5 days because it chewed up memory like crazy. We tried the locally available debug tools, then opened a PMR to the OS support queue, which then involved the TSM support one, and started collecting snaps and perfpmr reports. we went through the usual hoops of upgrading the OS and several software components as required by the OS and TSM support queues (with all the change managemeng hassle in between) and weeks flew by, with the customer's anger growing progressively because of the delays in the backup cycle every reboot introduced. Then, to add more spice to the curry, the memory leaks became more violent and we had to reboot the TSM server (and all the library managers hanging from it) every other day. TSM and AIX top guns were clueless, we had almost nothing else to change/upgrade... I even started combing through TSM manuals, reference redbooks and older PMRs to see if I could spot some lead to hang to...

    And then, one morning, I ran a simple "ls" in the TSM root directory, and found a $PREFERRED_PROFANITY copy of libc sitting there... Seems like one of the TSM support resources had been debugging a routine and a previous TSM PMR asked him to put a copy of libc there to collect some evidence. Well, turns out the TSM binaries were compiled with "." as the first argument to look in the $PATH[0], and thus the TSM instances reported their memory deallocations to that libc copy instead of doing it to the one the OS was using... So in TSM's internal records, it was marking the memory segments of the dying processes as available, but to AIX they were still allocated.

    Simply removing all the libc stuff from the TSM root directory and restarting the servers ended all the memory leaking issues, and reminded me once again to look for the obvious before digging too deep into an issue.

    [0]nice way to compile binaries eh? The 80s called, they said they wanted it back for good.

    1. Unicornpiss
      Pint

      Obvious?

      I think that was a pretty good catch, really. Many people would have glanced over that 100x before the implications of it dawning on them.

      1. slimshady76

        Re: Obvious?

        Honestly, security in the AIX world is lagging a bit (and by "a bit" I mean maybe a couple decades) and I still recall my professor from the AIX 101 courses saying us to remove "." from PATH. He also warned many applications were compiled with the local directory in they built-in path as first argument, so we should NEVER let anything nasty/exploitable in the same directory as any other application.

  19. Martin
    Happy

    The extra pair of eyes don't even have to see the problem.

    Retired now, but what always used to happen to me was this.

    Me: Dave, could you take a look at this for me? This routine is going into a tight loop and never coming out.

    Dave: OK, so talk me through it

    Me: OK - so it starts here, and this bit runs fine, and there's the output, and then I increment the counter and - oh shit, the counter increment is outside the loop, isn't it? What a prat I am. OK, that's sorted it, thanks.

    And Dave returns to his chair, not even entirely sure what the problem was.

    We used to say that we could get a lot done by just having a cardboard cutout of the best programmer in the room to explain our problems to.

    1. Already?

      Re: The extra pair of eyes don't even have to see the problem.

      That was our regular thing - if something doesn't make sense or a bit of code just won't do what it's meant to do the answer was to go and explain why to whoever was handy, even if it was the receptionist (she was a lovely woman so any excuse...).

      It evolved into explaining problems to The Big Cardboard Ear, as salient life within wasn't necessary for the obvious to land squarely between the eyes before too long.

      1. A.P. Veening Silver badge

        Re: The extra pair of eyes don't even have to see the problem.

        And currently the method is very fashionable as rubber duck debugging.

    2. Dave Taflin

      Re: What a great discovery

      I call that "code therapy." You figure out your problem whilst describing your code to someone else.

      1. Anonymous Coward
        Anonymous Coward

        Re: What a great discovery

        As a junior support programmer I was expected by the users to diagnose their program errors - in any of the mainframe's many compiler languages. My technique was to go through their code with them - saying "and what does that do?". There would come a point where they spotted their error - and thought me so brilliant for (not) seeing it.

  20. J.G.Harston Silver badge

    #

    is not a hashtag.

    It's a hash.

    A hashtag is a hash followed by a tag.

    thereby the name: Hash. Tag.

    If # was a hashtag then #hello would be a hashtagtag

    1. Martin
      Thumb Up

      How dare you come on here with your pedantry, precision and accuracy.

      Next, you'll be telling us off for PIN number.

      1. Cederic Silver badge

        I'm grateful that he pointed out this horrific misuse of terms. It saved me having to.

      2. Unicornpiss
        Coat

        "Next, you'll be telling us off for PIN number."

        --well, you need that when you're using the ATM Machine.

    2. swm

      # is an octothorpe. (or pound sign or number sign or ...)

      1. dak
        Headmaster

        It's not a pound sign. Not anywhere that uses pounds, anyway.

        1. Screwed

          USA uses pounds.

          Just not as currency. You ever looked at any of their diet and weight loss sites?

          Also for stray dogs.

          1. Anonymous Coward
            Anonymous Coward

            It's also the verb for what should be done to language pedants, particularly "that's not how it's said/written here, so it's wrong" ones.

      2. Vincent Ballard
        Coat

        I remember back in about 2000, the first time I tried to top up my pay-as-you-go phone. I duly purchased a 10 quid top-up card from the supermarket, scratched off the opaque coating over the secret code, and tried to follow the instructions. But the instructions wanted me to enter the secret code followed by the hash key, and I couldn't find the output of a one-time function applied to the secret code anywhere on the card... It took a call to the operator's support line and much mutual confusion to figure that one out.

  21. Marty McFly Silver badge
    Thumb Up

    Screwed up

    "As tradition demands," he sighed, "the support agent instructed me to retry all the steps I had already tried, to no avail…"

    The correct approach:

    "It is going to take 15-20 minutes for everything to reboot, hold on the line for me"

    -Eat pizza

    -Drink beer

    "Nope, just like the first three times I rebooted, nothing has changed"

    1. doublelayer Silver badge

      Re: Screwed up

      I always do it. Yes, I'm sure it won't help, because I've done it at least four times before I called them. I still do it. I'll mention, while I'm waiting for it to reboot, that I've done it before, but I'll do it.

      The reason? I hate it when someone tells me they've already done something, I believe them, they're lying, and it could have saved me time. That happens very frequently, so I know why they don't trust that I've done it. Now if I can make them not ask for it by providing them a lot of information in the initial request, I'll happily do that, but if they're giving me instructions I've already tried, there's always the possibility that it will turn out to be just slightly different from what I did and will actually help. It hasn't happened yet, but it might. I have to give them the same benefit of the doubt that I want my family and friends to give me when they tell me that they really have tried forgetting the Bluetooth device and repairing, but actually they just turned Bluetooth of and on.

  22. Anonymous Coward
    Anonymous Coward

    What's red and slippery and a threat to national security?

    I have been amused for years by one usenet troll who endlessly complains about "viscous criminals" and "rouge elements".

  23. Sykowasp

    Never edit production config files directly.

    Always manage it with peer reviewed changes in a change repository, deployed via a tool such a puppet.

  24. the spectacularly refined chap

    if (not - a - tautology);

    this(always, executes);

    Usually takes someone an eternity to spot the first time, but forever wise to it afterwards...

  25. T. F. M. Reader

    When proofreading does not help

    Some years ago I earned - and maintained! - quite a reputation for being able to quickly figure out why some scripts didn't work. I had observed that some colleagues were in the habit of copying files from and to Linux servers using WinSCP on their Windows workstations. Copy from Linux to one's Windows, then copy to another (or the same) Linux, possibly with a bit of editing in between, lather, rinse, repeat.

    Somewhere in the process WinSCP screwed up EOLs to the detriment of #!/bin/bash and similar. Any attempt to run such a script would result in "command not found". Perfectly correct, but not very helpful in practice in this case. [No idea if that was ever fixed.]

    No amount of proofreading could notice the invisible "\r" at the end of the sharp-bang line, but a casual "Just run dos2unix on it" became a habit. I managed a bit of showmanship, too, just pausing for a second when asked to have a look and nonchalantly continuing on my way after making the above comment, accompanied by "How the hell did it get corrupted and how did you know???"

    Funnily, the situation kept repeating itself.

  26. Peter Ford

    The fresh eyes were my own...

    In my first flirt with system admin, while doing my PhD, I was in charge of a bunch of HP workstations. My supervisor had shown me the office and said 'See those cartons? That's your computers: unpack them, set them up and manage them for the group while your doing your research on them'.

    Some time later I was upgrading the OS: HP-UX something - it was a fairly major upgrade step and needed a complete fresh install from CD. So I set it going and entered some basic settings (network, time, stuff like that), sat for an hour or so while it all copied into place and then rebooted. Nothing. Didn't want to start.

    So back to square one. Clean start, load from CD again, enter settings, wait, reboot. Nope.

    So it's getting late, I'm getting tired. I'm sure there's something I've got wrong, but I can't work it out so I run through again. Nope.stage

    Right, tired, hungry. Give up and go home for sleep.

    Next day, after a decent sleep and some breakfast, I go back in and sit down in front of the machine and start again. I'm entering those details again, and realise that every time I'd been asked for the system time in the set-up process the night before, I'd been entering the wrong year, and when the boot process checked the license the computer thought it was last year...

    1. A.P. Veening Silver badge

      Re: The fresh eyes were my own...

      Entering the wrong (previous) year is also something I regularly do wrong in January. I even once managed to screw up the year run for a bank by doing exactly that. To make it even better, nobody else caught it either (at least eight eyes on it). And still they didn't want me to automate that part, something that was very well possible and would prevent recurrence.

      1. Anonymous Coward
        Anonymous Coward

        Re: The fresh eyes were my own...

        "And still they didn't want me to automate that part, "

        Even automation makes mistakes.

        My stomach said it was lunchtime- but the wall clock said it was still two hours to go. The clock is MSF "radio controlled" and gets an update once a day. Fortunately there is another one on the table - and a quick check with a third confirmed my instinct that it was about 13:00.

        Telling the faulty one to resynchronise worked ok. Seen that happen once before on a different clock. The MSF signal encoding has very little protection against subtle corruption - as it builds the time/date from sixty messages in each minute. Presumably such domestic clocks trust a one minute data set if they didn't have any error flagged for any one second transmission.

        I use the same technology for my Arduino Xmas lights timer - but it requires three consecutive minutes' data to correlate before it will update the RTC. ..and yes - it does therefore lag slightly on DST jumps.

        1. A.P. Veening Silver badge

          Re: The fresh eyes were my own...

          Even automation makes mistakes.

          Sure, but not about date and time on an AS/400.

          1. Anonymous Coward
            Anonymous Coward

            Re: The fresh eyes were my own...

            Everything operates within constraints. All one can do is hope to minimise the probability of a known constraint being breached - and pray there are no "unknown unknowns".

            "What can go wrong, will go wrong - at the worst possible moment."

            1. A.P. Veening Silver badge

              Re: The fresh eyes were my own...

              "What can go wrong, will go wrong - at the worst possible moment."

              Murphy was an optimist.

  27. Anonymous Coward
    Anonymous Coward

    Bring relatively abstemious - I found three whiskies would slow my brain to the point where I progressively had to think harder about the code I was reading. Each stage could be the successful one. After three I was near to sleep and soon in bed. Sometimes the sleep itself produced the eureka thought on waking.

  28. Stu J

    Boiler

    Once called out a boiler repair service to fix my boiler which wasn't coming on at all. The guy flipped open the front panel, and flicked the timer control switch from "off" to "manual", the boiler fired up immediately, and he thanked me for the easiest job he'd had all day, and knocked off an hour or two earlier than he'd expected to.

    My wife and kids all denied having touched the switch. I knew I hadn't, and had assumed because it was in a cupboard behind the front panel that nobody else would have touched it. Never figured to actually check...

    1. Anonymous Coward
      Anonymous Coward

      Re: Boiler

      When you have eliminated the possible - that only leaves the impossible.

      We judge "possible" and "impossible" by our knowledge of a particular abstraction model to predict its behaviour. We may not be aware of what constraints underlie the abstraction.

      Does one regard the unexpected switch position as a root cause - or a symptom of something else?

      If it happens once - it is a glitch that only merits a quick investigation. If it happens twice it is a problem needing a deeper investigation.

      In this case it has been assumed that a human agency was required to effect the change. If the switch is merely a microprocessor momentary input - then the possibilities for a different activator are increased. eg cosmic particles will cause random effects on electronic circuitry.

  29. Unicornpiss
    Meh

    "Perpetually understaffed IT team"

    I think that is the description for 90% of IT departments everywhere. You wouldn't build a house without laying the foundation first, but businesses are so reluctant to properly staff and fund IT, which is often treated like more of a redheaded stepchild or necessary evil than one of the foundations that it is for all but the smallest businesses. And this includes not properly compensating most employees at every level.

    Why? I think because IT isn't creating or selling products but instead seems like just a money sink to leaders that are too high up to see down. "I can get my email, what do you mean we need to spend another 500K to upgrade infrastructure? Didn't we just do that?" (ten years ago)

    1. Anonymous Coward
      Anonymous Coward

      Re: "Perpetually understaffed IT team"

      There was a fashion to divide a company into units of function. Then each unit had its own P&L accounting - and their own T&Cs for employees. The only way the IT unit could make its "profit" was to charge other units for their IT services. The charges also required a "margin" and "mark up" according to business practice.

      These other units then often reacted by trying to do their own IT on the cheap. There is an apocryphal story that one unit was buying networking gear off a market stall. They caused catastrophic corporate network loops when they linked two of their locations - with a private bridged link that was otherwise unknown to the central network operations.

      The end result was internal competition and introversion - that did nothing for efficiency. Incidentally the competition between some overlapping units also extended to undercutting each other's business offers to external customers. It was termed a "robber baron" culture - with the board instigators expecting only the most ruthless units to survive and take the company to bigger profits.

      1. Terry 6 Silver badge

        Re: "Perpetually understaffed IT team"

        This is not dissimilar to the local authority insisting that IT purchases had to come via a purchasing department, which told us what product we had to buy (through them) and added a percentage to pay for their cost. But their recommended purchases were invariably more expensive than I could have bought the same items elsewhere - even before their charge. The purchasing officer had no concept of tco, so just bought the cheapest device that would do the job, even if it wasn't going to last more than a year or two, or would be inefficient to run, with our level of usage. He was dismally unaware of competing products, too. So in effect the tiny budget allowed to us by the authority, supposedly calculated on a needs basis, would be significantly diminished, because of course they didn't revise our budget to allow for the extra costs. Well they couldn't could they, because that would be an admission that the system was not a cost saving for the frontline services they were supposed to be providing. Oh and most of his purchases came from the same catalogue based supplier that we'd been told we must never use.

  30. Anonymous Coward
    Anonymous Coward

    Vcenter

    I think my response to an issue with vcenter on a Saturday would be "connect to the hosts, I will look at it on Monday."

  31. NeilPost Silver badge

    Boo-Hoo

    “ Jon has short shrift for those who gloat over accrued overtime: "Managers don't get overtime, we get one cold slice of leftover pizza."”

    Boo-Hoo. Managers get paid more and usually attract an annual bonus. My tears are dry here.

  32. SuperGeek

    Yummy!

    "The second On Call of 2021 is a warning not to mix pizza and hashes."

    Mmmm, potato hash pizza! Yummy :)

  33. Anonymous Coward
    Anonymous Coward

    We won't automate that - it's too important

    At a previous employer, I'd coaxed them into having CI build pipelines for their key products - dev, CI, staging, pre-prod.. and it all worked beautifully, but prod was one step too many for them. Nooo, management said - we need to be very careful with changes to the prod system, it must be done by a human.

    What this meant was that every 2 weeks to a month or so, a luckless dev or two would have to sit around until a few hours after close of business, then spend an hour and a half or so deploying the latest changes manually.

    We had 2 occasions of the devs being unsuccessful with deployments, spending 5 hours trying and failing to deploy the changes, then rolling them back. Both times the result? a case-sensitive string that occurred all over the place had it's capitalisation entered incorrectly by the pesky humans doing the deployment.

    Pointing out that not letting the prod deployment be automated was making the deployment less reliable, whilst also killing dev morale with the unpaid overtime, did not go down well. They persisted with the manual deployments, and I left!

  34. Anonymous Coward
    Anonymous Coward

    Config files?

    Hold my beer...

    Once drove 2 hours to site to unplug an unused SCSI cable on a Compaq server which failed to boot with a POST error* that halted the machine, on site had insisted the machine was 'blown up' after a site power down so we got the wheels in motion to courier a complete replacement machine out and for me to meet it on site to commission it.

    total time taken, 4 hours and 5 minutes (included signing in at reception).

    * The crimped on terminator had failed and the SCSI controller was smart enough to recognise that and refused to let the machine boot.

  35. This post has been deleted by its author

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like