back to article See that last line in the access list? Yeah, that means you don't have an access list

Just one more day to go – the weekend is creeping into a view. Unless, of course, you're one of those brave souls cursed to be forever On Call. It seems an awful lot of you have had a run in with Cisco hardware at one time or another. The company is, after all, almost a byword for networking infrastructure and some interesting …

  1. Anonymous Coward
    Anonymous Coward

    My boss was a twit.

    He executes the command. He specifies it to dump to a log file. He specifies a system critical filename. The command promptly shits itself as it's not able to do as it's told, the system process that has control of the file takes a crap as the system spends a few moments wrestling with itself trying to do the impossible. One hard reboot later and we have our network back, our network monitor restored, and the boss telling everyone "Don't do that last bit, ok?"

    *Face palm sigh*

    1. This post has been deleted by its author

    2. DJV Silver badge

      Re: My boss was a twit.

      Well, at least he learned something from the experience! Many PHBs fail that last stage completely.

      1. Steve Kerr

        Re: My boss was a twit.

        Or get promoted

      2. Anonymous Coward
        Anonymous Coward

        Re: Well, at least he learned something

        But will he remember it tomorrow?

  2. Pascal Monett Silver badge
    Mushroom

    "The contractors were fired."

    And I hope they weren't paid.

    1. MiguelC Silver badge

      Re: "The contractors were fired."

      In my first employment, as a junior dev, I was working with an expert just contracted for his extensive DB2 knowledge. He was supposed to set up a battery of DB tests on a Friday for me and my mates to follow up during the weekend (as this was a Y2K project and time was running short, we, the minions, would be there - but not the expensive contractor).

      The bad news is that he managed to screw up on every single test he was supposed to prepare by not allocating disk. space - and the leaving logs of it for us to see. That meant he had just wasted the whole team a Saturday By the end of the day I'd managed to redo - this time correctly - everything the expert was supposed to have done for us and our team got it's work done for others to take on next Monday.

      The good news it that the following Monday, our manager heard us arguing about what had happened (I may have been expressing in not very pleasant terms how pissed off I was about all that sorry affair) and, after investigating it, called me apart and told me I didn't need to worry about it any more as the 'expert' was no longer working there.

      Even better news is that I also got promoted a month or so later, with the way I handled that incident being a factor.

    2. DS999 Silver badge

      Legally, they can't avoid paying them

      But they can claw back all their expenses/damages from the incident, which might even be higher than their invoice, by making a claim against insurance. That is, assuming they were smart enough to check that the contractors were properly insured against errors and omissions before engaging them!

      1. Martin Silver badge

        Re: Legally, they can't avoid paying them

        Oh yes they can avoid paying them - at least in the UK.

        This is not an error or omission. This is clearly incompetence.

        They will have a contract to do xxx in return for £xxx. They have clearly not done xxx, due to their incompetence. Therefore they are in breach of contract and you don't have to pay them.

        It may be that xxx is only a part of what they were meant to do, and the rest is ok. Then fine - pay them a proportion, and subtract something for compensation for the xxx that they failed to do that someone else had to find and repair.

        It's then up to them to sue you if they still feel they owe you the money. Me, if I'd cocked up that badly, I wouldn't want to risk it in case it made the computer press.

        1. Lil Endian
          Joke

          Re: Legally, they can't avoid paying them

          do xxx in return for £xxx

          The contractor was John Holmes and I claim my £Wadd!

          1. sev.monster Bronze badge
            Gimp

            Re: Legally, they can't avoid paying them

            Oi you, shut your mouth and lookit my Wadd!

            ...Is this public indecency?

  3. Jay 2
    Pint

    "See the last line in the access list?" he told the customer. "That means you don't have one."

    That brightened my day somewhat, thanks! Therefore beer ->

    Though on a slightly more serious note, makes you wonder WTF the contractors were playing at. How can anyone with any sort of conscience fudge a firewall/ACL with an any:any (or equivalent) and say nothing about it?

    1. Giles C Silver badge

      I’ve had people come to me with firewall changes with an ‘any’ in the rule.

      Now for certain things this is fine i.e. web servers where you have inbound traffic; mail servers to smtp outbound etc

      But the only thing that rule should be used for is with a deny statement and then send everything to a syslog server.

      It isn’t hard to set up a firewall, think about it logically and check each service properly although with some of the stuff I have seen a lot of people shouldn’t be anywhere near a firewall.

      1. Anonymous Coward
        Anonymous Coward

        I once had a director ask if the firewall was needed between the public facing web server and database server because "it slowed traffic down"...

        1. Giles C Silver badge

          Used to be a common argument about why SIP couldn’t go over a firewall…

          Mind you an employer had programmers that wouldn’t use dns as it slower the program down!

          They hated it when we moved to a new site and had to replace the IP address as the old site was still operational.

          1. swm Silver badge

            In the early days we got a vax with networking and it didn't work. Looking at the code we found that the PUP (this was before TCP/IP) address of the gateway was hard coded so when they moved the vax to another net it couldn't find the gateway.

            Then there was the old routing algorithm "send to BRL". That is, send the packet to Bell Labs and let them deal with it.

            1. sev.monster Bronze badge

              I ran into an issue where this PHP installation running on Windows Server (there's your problem) ran incredibly slow if you left the database address to the default localhost, because on Windows localhost DNS lookups are painfully slow. I'm not joking.

        2. Plest Bronze badge

          Should have said, "OK, you tell me what you think will happen and then we'll put it to the test. Before we start can you just sign this waiver and have my P45 ready?"

          Of course I'd also be inclided to slot a dual-redundant IDS and firewall setup on the outside of that webserver, box the whole lot into an isolated subnet and only allow key internal boxes to be able route into the isolated websvr/dbsvr combo network to perform admin, zero traffic inbound to the primary nets. I'm not a networks bod, just an admin who's fecking paranoid, and dealt with enough pondscum from the internet who will have your webserver compromised in about 60 secs after you've stood it up if you let 'em!

          1. A.P. Veening Silver badge

            who will have your webserver compromised in about 60 secs after you've stood it up if you let 'em!

            You are either very generous or not nearly paranoid enough with those 60 seconds, 6 seconds will be a lot closer to it.

            1. ortunk

              No issues with mine, without firewall and properly configured debian, 15 years and counting

              1. jgard

                .... Which means you're a careless, incompetent admin and you operate the internet's least interesting web server. We'll done sir!

              2. Anonymous Coward
                Anonymous Coward

                I will generously assume you mean you have no hardware fw and rely on ip tables and fail2ban...

                Otherwise I suspect your the main reader of your blog

              3. Stuart Castle Silver badge

                What monitoring do you do? Would you even be aware if your box was compromised? Bear in mind there are hackers that can gain access to systems and use it for months without flagging anything to the system admins.

            2. Anonymous Coward
              Anonymous Coward

              I remember looking at the inbound traffic on my residential internet access. And realized that a freshly-installed Windows (XP at the time) box hooked directly up to the internet, with no firewall, would get pwned before it could download the updates to patch all the known security holes. A firewall is a NECESSITY.

        3. Anonymous Coward
          Anonymous Coward

          re: I once had a director ask if the firewall was needed ... because "it slowed traffic down"...

          So when someone without the knowledge asks someone with the knowledge they get sneered at? That's not something to be proud of.

          The directors job is to ask those kinds of questions, it's your job to give accurate answers. If the director knew everything you knew you'd be out of a job.

          And we wonder why people don't like the IT dept.

        4. Stuart Castle Silver badge

          Re "

          Anonymous Coward

          I once had a director ask if the firewall was needed between the public facing web server and database server because "it slowed traffic down"..."

          That's potentially a bit harsh. Unless they were a director of IT they wouldn't be expected to know much about network security, so may not be aware of the problems involved in enabling direct access to the database server via the web server.

          If they were a director of IT, then fair enough, they should be aware of the security problems, but even then they can't be expected to know everything about their subject areas, often having to rely on their teams for the specifics.

          1. Alan Brown Silver badge

            If they're not the director of IT, then they should be asking the director of IT, not the lackeys

      2. Anonymous Coward
        Anonymous Coward

        Are you calling my supervisor an idiot?

    2. MrReynolds2U Bronze badge

      Firewall rules 101

      A lot of SME and enterprise firewalls come with two implicit rules:

      - deny all IN

      - allow all OUT

      The first thing I do is change the outbound rule to deny all OUT.

      Keep that allow all crap on consumer kit please.

      1. Alan Brown Silver badge

        Re: Firewall rules 101

        the number of times I've seen thie rule save a compromised box because the script kiddies can't actually do anything with that compromise and abandon it immediately....

        If you're in a commercial environment and you don't have your web & mail servers sandboxed to the hilt (including INSIDE the network), then you're a disaster area waiting to happen

  4. Anonymous Coward
    Anonymous Coward

    was it seman's contracting dicks?

    Had a run in with "cisco certified" experts back in late 90's from siemens.

    quickly found out "cisco certified" means "can run config generator tool", to the point that they had no clue about subnets and even less clue about t1/e1 and isdn configs.

    (they later did hire someone i had worked with, made sense as he was the company idiot! who's CV was written by a fantasy writer.)

    1. My-Handle Silver badge

      Re: was it seman's contracting dicks?

      "...who's CV was written by a fantasy writer"

      Either that or it was written and the job applied for by someone from within your own company. Just to, you know, help him get on to that next stage in his career.

    2. MrReynolds2U Bronze badge

      Re: was it seman's contracting dicks?

      There was a glut of "cisco certified" chaps running around in the late 90s.

      They paid a grand for 2 weeks of instruction and came out with a CCNA or whatever it was then.

      I remember taking one out to a customer's site and he got all excited because he'd never worked on a real bit of Cisco kit before, only lab stuff.

      I preferred the Checkpoint tech. He knew his kit inside and out.

      1. John Brown (no body) Silver badge

        Re: was it seman's contracting dicks?

        "I remember taking one out to a customer's site and he got all excited because he'd never worked on a real bit of Cisco kit before, only lab stuff."

        Wow! Lucky him! The training I get means once I've passed the course, only then am I allowed to go out and play with customers kit. Being remote (not just because of COVID, been remote from the office for 20 years now), I don't get to actually touch the physical kit, the training is all online videos and exams.

        1. Alan Brown Silver badge

          Re: was it seman's contracting dicks?

          acquiring secondhand cisco kit is cheap and a home lab is a nice thing to have

      2. WhereAmI?

        Re: was it seman's contracting dicks?

        I was one of them. I did it as a part-time course over three months at night. What total waste of money and time - I have rarely been so bored in a tech class. All paper work and no actual hands-on except for the last couple of days.

        I only did it because I was expected to. Guess what? Haven't seen a piece of Cisco crit (crap? kit?) from that day to this.

        1. elaar

          Re: was it seman's contracting dicks?

          "I only did it because I was expected to. Guess what? Haven't seen a piece of Cisco crit (crap? kit?) from that day to this."

          What exactly is it you do in IT if you've never seen a bit of Cisco\Meraki kit? Cisco have almost 50% market share with their enterprise WLAN "crit".

          Working for an ISP, all of our core routers are Cisco, some of their product lines are great. Overpriced, yes. Crap, no.

          1. jockmcthingiemibobb

            Re: was it seman's contracting dicks?

            This ISP heavy lifting is Juniper. Sub 100Gb stuff is generally Mikrotik. Meraki wireless...no thanks.

            I know plenty of people in IT who've never touched Cisco or Meraki gear outside of training courses

      3. Alan Brown Silver badge

        Re: was it seman's contracting dicks?

        "There was a glut of "cisco certified" chaps running around in the late 90s."

        same as MCSE people. lots of theoretical experience and usually only enough to pass the exam

        It's critical to ask a few questions about experience and configuration during the hiring process. If someone ONLY has a MCSE or CCNA then stay far, far away

    3. Nick Ryan Silver badge

      Re: was it seman's contracting dicks?

      I had a candidate's CV come in listing CCNA and a specific entry claiming knowledge of DNS. One of the (basic) test questions I asked was "what does DNS stand for and what does it do". He couldn't answer this question. I was quite blunt with the agency that sent this candidate through...

      1. Old Used Programmer Silver badge

        Re: was it seman's contracting dicks?

        I once expressed amazement to a recruiter that *anyone* could fail the standard COBOL programming test that the client was using the filter applicants. He assured me that about 80% of the people claiming to know the language did fail it.

        Of 30 questons, 29 were trivially easy. The only one that wasn't was about a construct I never once (in 40 years of working with COBOL) ever saw anyone actually use. It was about PERFORM...VARYING...WITHIN...VARYING...

  5. elsergiovolador Silver badge

    Expensive

    Usually you get what you pay for.

    1. Antron Argaiv Silver badge
      Happy

      Re: Expensive

      No more than you pay for...

      1. Plest Bronze badge

        Re: Expensive

        Lucky if you get anywhere near not enough with some cowboys! You get cowboy builders and the IT biz is no different.

  6. Pete 2 Silver badge

    The difference between a good contractor and a great one

    > A quick call to run show tech-support had expanded into multiple days

    That sounds like a guy who knows his stuff.

    However, a truly great contractor could have turned it into a month's work.

    1. Jiggity

      Re: The difference between a good contractor and a great one

      As someone once remarked to me, "If you're not part of the solution, there's good money to be made prolonging the problem"

      1. Lord Elpuss Silver badge

        Re: The difference between a good contractor and a great one

        The mantra of the pharmaceutical industry....

  7. Paul Johnston
    Flame

    Oh yes!

    Was once told of a contractor from a "Very Expensive" outfit whose build scripts for cloud VMs included the line

    chmod -R 777 /var/data

    Warning, just because you sign the contract with a company which seemed during the tender process to have some very good people no guarantee you won't get the apprentice doing the coding.

    Obviously you need your best people getting contracts not actually doing them.

    1. Zippy´s Sausage Factory

      Re: Oh yes!

      We once told a company to pull out of tender process because they wouldn't guarantee the people who were pitching for the work would actually be doing it. They were not impressed, and finally had to be told that they were wasting everyone's time and money, especially their own.

      1. 2+2=5 Silver badge

        Re: Oh yes!

        > We once told a company to pull out of tender process because they wouldn't guarantee the people who were pitching for the work would actually be doing it.

        My employers mostly make the provision of named staff onto a project conditional on the project starting on a specified, agreed date. Despite agreeing to this, tenderers never manage to start on the agreed date.

        1. LyingMan

          Re: Oh yes!

          Hmm.. How come that's acceptable business practice but not acceptable for ir35?

      2. jezbod

        Re: Oh yes!

        Like the tender for 60 laptops we put out, one company did not even make it past the first filter.

        They had not even quoted a machine with the correct CPU spec, as listed in the tender document.

        Fail!

    2. My other car WAS an IAV Stryker Silver badge

      Re: Oh yes!

      "Obviously you need your best people getting contracts not actually doing them."

      Too often in US defense contracting, those writing the proposals are the numbskulls who will promise that $platform (often a vehicle) will do everything to the max for free. Then it takes the actual engineers months to years to design the thing AND convince the customer that it's never going to meet the proposed metrics.

      Examples:

      - Acceleration: Sure, but maybe if you switch off the aircon and as many electronics (like radios, remote weapon station, vision enhancements, etc.) as possible, paved road only, within a certain temperature range. (It also helps if the engine has the proper "military/unlimited" fuel map -- see my post history regarding that one.)

      - Alternator power output: Alternator supplier advertises output of P above certain speeds (certainly not engine idle speed with the vehicle standing still). That's using room-temperature cooling air with the control module ("regulator") close by. The CM can't handle engine compartment temps and wants to be near the battery bank, so it is, about 10X the advertised cable length from the alternator. And we have to test with HOT intake air -- with an intake ballistic grille restriction, not just a chicken-wire screen -- and even hotter compartment air. So when you get 75% * P during an under-armor test, don't be surprised.

      Defense contractors -- behind schedule, over budget, and falling short on tests, but still getting paid.

      (Note: That extra money pays for our extra time, failing some tests like the above is because the design meets the other 80-90% of tests like temperature range, ballistic protection, etc., and in general this industry is no more crooked than any other. Sigh.)

      1. Anonymous Custard Silver badge
        Trollface

        Re: Oh yes!

        ...within a certain temperature range...

        Such as hell frozen over?

        Always such a joy being expected to keep sales people's promises after they've departed off to create their next disaster area.

      2. imanidiot Silver badge

        Re: Oh yes!

        When it comes to military equipment I think the best way to find an optimal config is to just have a bunch of squadies (preferably marines) use the vehicle for a while. Leave out an angle grinder and a welder.

        I've heard of WW2 era designs getting optimized by finding what stuff was still left in the tank after it had (successfully) been through a few combat engagements. Anything not removed and tossed was important. Anything not there doesn't need to be there in new vehicles either.

    3. Anonymous Coward
      Anonymous Coward

      Re: Oh yes!

      Standard procedure for 90+% of web developers in my experience.

      First sign of something not working, chmod 777 all the files.

  8. Anonymous South African Coward Silver badge

    We once did the unforgivable and opened port 22 to * on a firewall... (a longish while ago)

    ...linux server got hacked and people was pissed off with us.

    Luckily for us we caught it fairly quickly.

    Suffice to say we now install fail2ban on every linux server we now deploy. Just in case.

    1. gumbril

      Er.. and turn PasswordAuthentication off and employ certificates only I hope. Username and password on SSH is no way to be going through life.

      1. JAB van Ree

        Or set up 2FA, these days it comes included in most Linux distributions.

  9. TeeCee Gold badge
    Meh

    Smoking gun.

    "email was meant to be channelled through a filtering company and not directly exposed."

    Aha. I'll bet that proved tricky to get to work properly as, as is usual in such cases, getting the filtering company to pony up with exactly what needs to be allowed where is like getting blood from a stone. They'll have a document that says to do XYZ which you have dutifully implemented, but it still doesn't work.

    The problem will turn out to be (as in you can safely bet your mortgage) that they haven't updated their customer config document for 18 months, or "a hardware refresh cycle and umpty-something software changes" as that's better known as.

    So, stick in any / any to get things running with removing same, once you've got to speak to someone at the third-party who can discuss technical detail without drooling, on the "to do" list. Roll on a bit, the "to do" list and the "too hard" pile merge and we are where we are.

    1. PM from Hell
      WTF?

      Re: Smoking gun.

      I'm a contract PM, my job is to get an implementation live by the agreed deadline. I will take some shortcuts when the pressure is on, minimal testing of non-core functionality, maybe defer a patch application until after go live (unless there are critical security updates)

      But I will never allow a breach in security. I often need firewall rules adding or amending to get SAAS systems working but I will always try and sit down with a firewall admin and talk through the requirement, often then progressing to testing exactly which port numbers do need to be open and for which protocols.

      Vendors seem to know less and less about what is really required every year.

      I used to be able to get a specification of all iP addresses ports and protocols from vendors before we stated implementation but now SAAS vendors will often just specify a couple of IP addresses for all ports and protocols, effectively kicking a huge hole through the firewall.

      I'd rather we had to redo some tests which failed early in UAT because rules were not enabled than find out that network security was breached after go-live because we'd compromised it that much,

      1. Daedalus Silver badge

        Re: Smoking gun.

        Vendors seem to know less and less about what is really required every year.

        Simply put, as technology gets into more and more places, the competent tech people, whose numbers are not growing as fast as needed, get stretched more and more thinly. Inevitably the bozos take the place of the competent people, especially since they tend to be cheaper. Then you've got the wannabees in lower manglement who think they can give it a shot, and there have always been the EE's who wander into programming because they aren't that good at hardware....

        We're doomed, I tell ye. Doomed.

        1. Old Used Programmer Silver badge

          Re: Smoking gun.

          That would be me (EECS, actually), except that I knew all along that I was lousy at hardware, so I used what I learned about the hardware to aid my programming. Worked, too. Used to read S/360 Principles of Operation to find new ideas.

  10. Anonymous Coward
    Anonymous Coward

    Oh, yes, contractor installs..

    I was once lead architect of a major project at a telco, and what we did was at the time totally new (think 15 years ago).

    The problem: router installs HAD to be done by the subcontractors who did on-premise work, so we compiled a list of the correct commands to implement and waited for the system to tell us it had been done.

    Of course, we couldn't reach the thing. We couldn't see it at all on the infrastructure we were building, and this was just about the most important, most prestigious location which the service was going to reach. Naturally, the dolt installing it had ignored our instructions and left it with the default install config so it wasn't even set up for the right IP range.

    Now I only know enough about networking to be dangerous (ok, it's a bit more, but router specific configs is not my expertise), but I was the only one in the team who was authorised to enter that place and I was at least 100% familiar with serial interfaces and terminals, so I was given my own list of instructions by our resident expert, and off I went.

    The net result is that I am pretty much the only person who can claim to have "hacked" a router in that specific, very interesting place, but what gets the beers is how I did it, and how the guy who escorted me in paled when he saw how little it took to change the setup.

    I used a Psion Series 3A :).

    1. MrBanana Silver badge

      Re: Oh, yes, contractor installs..

      It's pretty funny when the "security" on a system is exposed to those who think it is impenetrable. I once worked for a software company that sold an accounts package. When our own accounts department had a problem with it, some bug that wasn't correctly updating the sales ledger, I was called in to categorise the problem, and get it registered with the package vendor. In the meantime all their consolidation reports were wrong and the head accountant was tearing his hair out as he knew how long it would take to get the vendor to fix it. Oh, if that's the problem, [ I type a single SQL command to change the master ledger ]. He now seems even more enraged that it is possible to alter the system without using the official application - no pleasing some people.

    2. John Brown (no body) Silver badge

      Re: Oh, yes, contractor installs..

      "The net result is that I am pretty much the only person who can claim to have "hacked" a router in that specific, very interesting place, but what gets the beers is how I did it, and how the guy who escorted me in paled when he saw how little it took to change the setup.

      I used a Psion Series 3A :)."

      On the other hand, the "simple" kit used to "hack" the router required physical access. I'm guessing what this "interesting" place was, so I assume the had pretty rigid physical access control to the site, so your escort should not really have paled at the ease of the job. More likely, he was not especially IT or network literate and maybe thought this could also be done remotely once it was working.

      1. Anonymous Coward
        Anonymous Coward

        Re: Oh, yes, contractor installs..

        I'm guessing what this "interesting" place was, so I assume the had pretty rigid physical access control to the site

        Oh yes. You can't just walk in there. I don't think you'd even actually make it to the front door before they'd nab you, and even with authorisation you will be escorted and eyeballed for the entire duration of your presence, which in my opinion 100% as it should be.

        Personally, I don't have a problem with 100% supervision and auditing of the work I do, because it pretty much removes the chance of being accused of something I didn't do. This is also the basis on which I audit, and it removes a lot of the psychological friction you otherwise get.

        Audting to find fault is easy, any moron or consultant (but I repeat myself) can do that. Audting to improve things is harder, as it requires actual deep insight in and knowledge of what you audit, but it is in my opinion much more interesting than the former.

        And I can charge more :).

      2. elaar

        Re: Oh, yes, contractor installs..

        "I used a Psion Series 3A :)."

        Why didn't you just take a laptop?

        1. Anonymous Coward
          Anonymous Coward

          Re: Oh, yes, contractor installs..

          Where would be the fun in that? :)

          Also, in those days, laptops were not the nible lightfooted devices we have today, they were better defined as luggables. A battery life of seconds and a boot up time which blew most of that charge demanded that you also had to lug a power supply along.

          I could carry Psion Seria 3A in a pocket of my overcoat, including serial cables and a spare set of penlights to stave off Murphy's Law of batteries. No contest for this specific job :).

          (come to think of it, it probably would have required the laptop to be screened prior to connection to anything there).

  11. ColinPa

    Bluff your way to the top.

    I remember working with a very senior person in our large IT company who was the test guru in one area. I had been there about a year. He wanted a configuration change made to an operation system ready for the weekend test. I made the change he requested, but the box failed to start. I tried different flavours of the change, and all failed.

    Come Monday, I nervously went round and told him about the problems. It turns out he knew very little about anything. He had been hired a few years earlier on a huge salary - and therefore got the title that goes with the salary. I spoke to my boss who did some digging. Where his CV had said "has worked with ..." basically meant " sat in a room where other people worked on it". His next years "targets" were set for someone at his level, and he decided to take early retirement.

  12. Flightmode

    Show tech

    The fact that 'show tech-support' is usually abbreviated to just 'show tech' (especially when done verbally) has delayed the response to many Cisco support cases. Way back, the Cisco support organization used to be called the Technical Assistance Center, or TAC for short. They'd usually be the ones asking for the 'show tech' output when you raise a case. Also, more senior engineers usually get pre-PFYs (PFIEs?) to log support cases for them with the comment "oh, and go ahead and collect a 'show tech' already now, 'cos they're gonna ask for it".

    The less experienced would log on to the device and not collect a 'show tech', but rather a 'show tac', which the CLI expands to 'show tacacs' - which essentially just shows you the IP and status of the configured TACACS+ authentication servers. This is typically 5-15 lines of output - compared to the thousands you'd get from a 'show tech-support'. They'd then happily attach that output and go home for the day (because these issues only happen after 4 pm in the day), delaying any further progress until the next morning.

  13. Timo

    Stopped asking why a long time ago

    I used to ask that question of "what are you trying to do?", but that often opened an few cans of worms. I now just follow my orders even when it makes no sense. Much less friction at the ivory idiot towers and it keeps the paychecks flowing. I've given up.

    1. Plest Bronze badge
      Happy

      Re: Stopped asking why a long time ago

      While I can concur that sometimes you need to pick your battles, to me the professional attitude is always make a fuss when you know you're right. Allthough it depends if you mind your ribs rattling eating Pot Noodles for the next month while you look for a new job!

      1. Anonymous Coward
        Anonymous Coward

        Re: Stopped asking why a long time ago

        When I have to make a fuss, I also make damn sure it's documented in an email or something, so when my fuss is overridden and they start staring daggers at me, I can say 1) it's not my fault and 2) I told you so.

        1. CuChulainn

          Re: Stopped asking why a long time ago

          Upvoted.

          But I once got told to stop sending such emails because it 'made people uneasy'.

          This was in the early days of everyone having a computer on their desk where I worked, and the same time period when my manager came back from an introductory computer course one time with a big fold out picture of a keyboard so he could see where the letters were.

          My emails have never been one liners (i.e. terse) when explanation was necessary, and since I worked as a chemist in a pharmaceutical company, explanation usually was. But suddenly being able to communicate by traceable email instead of not being able to reach people on the phone for days at a time (and they couldn't operate their mailboxes either, which were often full or not enabled), seemed like a great way of protecting your back. And it worked very well for that.

          But the higher-ups didn't like the fact that it exposed their backs after a few times.

    2. Anonymous Coward
      Anonymous Coward

      Re: Stopped asking why a long time ago

      I'm one of those sales peoples so often burned in here, but the first thing I always ask is "what do you plan on doing with the hardware". Often what is planned and the hardware ($$$) they are planning on are in two different piles, both of which are catalysts for one big pile of shyte.

      I consider it my job to congratulate the customer in the end on their selection of what they needed and how astute they were, even if in the misty background they are wondering how they managed to get there and how the bean counters (be it spouse or the guys and gals with the ledgers) are going to react.

      I live in a smaller community where word of mouth is everything, and have been doing this for 20+ years. Some of us behind the brochures desk do want a totally successful transaction.

  14. Plest Bronze badge
    Facepalm

    Averted several serious screw ups by so called experts called into places I've worked as the logic goes from the PHBs, "They charge an arm and a leg for their services so they must be bloody good, obviously better that you bunch of permie monkeys else you'd be earning what they make!".

    One such event was for rigging up kit for outsourcing, expert rolls up with "bag'o'scripts" and proceeds to demand they're run on all systems through central managment tool that can run anything as admin. I decide to take a look, as you obviously should do, before running anything just handed to you. 20 mins later found last line was variations of "shutdown -reboot now" for various o/s flavours that was left in for testing. I report my finding to PHB and contractor stating that it would not be wise to execute as is, shall I edit it first? I was told to leave it and they'll be back on Monday with some new scripts.

    Much more contrite contractor arrives hands me the "bag'o'scripts" (tm) and asks if wouldn't mind vetting them before we run them. I wanted to say in my most patronising tone, "Hmm, now isn't it better we all play nice and stop being so condecending to those on less money but with way more experience?". I simply smiled and said, "Of course....but we run them on a small selection of pre-dev builds to be sure."

  15. Norman Nescio

    Firewall rulesets and defaults

    at the end of the access list: "permit any any".

    There can be a good reason for allowing any - any as the last rule in a ruleset. It depends on the firewall defaults: block, or allow.

    Back in the day when networking was simpler, one function of firewall* rulesets was to drop, rewrite, or redirect chosen traffic only, and allow the rest through.

    I have worked on firewalls where each rule in the ruleset was tested in order**, and if no rule matched, then the traffic, by default, was dropped. In this case, it was important that the last rule in a ruleset was any - any, as without it, traffic that you expected to traverse the firewall in fact would not.

    I would not be surprised if whoever worked on the ruleset expected the above behaviour - that is, the firewall by default drops/blocks everything and has to be explicitly told to allow through everything not already matched.

    Of course, if allowing any - any was the first rule in the ruleset, criticisms of the competence of the writer are fully justified.

    Note that if the 'On Call' writer didn't know this ruleset quirk, they might possibly have made unjustified assumptions about their predecessor.

    NN

    *Where firewall == router configured with some ACLs

    ** Linking to the relevant Cisco documentation on ACLs

    Cisco:Configure Commonly Used IP ACLs

    The IP ACL is a sequential collection of permit and deny conditions that apply to an IP packet. The router tests packets against the conditions in the ACL one at a time.

    The first match determines whether the Cisco IOS® Software accepts or rejects the packet. Because the Cisco IOS Software stops testing conditions after the first match, the order of the conditions is critical. If no conditions match, the router rejects the packet because of an implicit deny all clause.

    1. stiine Silver badge

      Re: Firewall rulesets and defaults

      More normally, the last rule is: any/any/any drop. If you need any any/any/any permit rule for testing, it has to be next-to-last in the list above the any/any/any drop. This is so that when you complete testing and have all of your required rules in place, you need only delete the any/any/any permit rule.

      That being said, the best way to do this is to NOT use an any/any/any permit rule, but to open ports as the application requires them, but this is much more labour intensive.

      1. Norman Nescio

        Re: Firewall rulesets and defaults

        The linked Cisco documentation makes clear that any/any/any drop is implicit. So any/any/any permit as the last item in the explicit ACL is effectively penultimate.

        I did not make clear enough that the philosophy of block/reroute chosen traffic and let everything else through was not ideal, but at the time it is what some people did, the point being that if you did not put any/any/any permit as the last processed ACL entry you could end up with not passing the traffic you expected. Cisco show this in the example ACL for denying telnet traffic:

        Cisco ACL example:Deny Telnet Traffic (TCP, Port 23)

        interface ethernet0

        ip access-group 102 in

        !

        access-list 102 deny tcp any any eq 23

        access-list 102 permit ip any any

        But it is a subtle point that may be lost by people assuming that it is obvious that you set up firewalls to allow only certain traffic and block everything else. While this is recommended practice, some people don't do that (or at least, didn't), and not all use-cases demand that.

        I cut my teeth on Wellfleet Routers routing IPX, Appletalk and IP, so I learned to question my assumptions when using Cisco kit as there were always multiple ways to do things including: the Cisco way, the Wellfleet way, the standards-conforming way, and the right way.

  16. Will Godfrey Silver badge
    Angel

    For the avoidance of doubt

    I just want to say it wasn't me.

    Will.

    1. SuperGeek

      Re: For the avoidance of doubt

      "I just want to say it wasn't me.

      Will."

      Then when the proverbial shit hits the fan you'll hear "Fire at Will!"

      "Will didn't do it!"

    2. This post has been deleted by its author

  17. aregross
    Megaphone

    Ooo, Ooo, I know this one!

    "We'll fix it later, it's time to go!" meaning the contractors had run over the allotted time per the contract and a "Supervisor" made a "Command Decision"

  18. FlamingDeath Silver badge

    Cisco is just over priced and over licensed corporate crap. They know there are businesses out there with full pockets itching to spend it on a brand, any brand so long as its popular, appears in movies, etc

    It aint even that good, has back doors etc, shoddy code, just like all the tech junk companies spew out daily, some just have better marketing

    1. elaar

      Provide some examples...

      It is overpriced (we can all agree on that), but they're workhorses. Leave them in a dirty comms room for 20 years and they'll carry on working.

      The company I work for installs probably about 20,000 bits of kit each year and the Cisco kit stands out for reliability.

      Explain how "it aint even that good".

  19. tip pc Silver badge

    One time for an audit

    Before our annual audit we where instructed to remove occurrences of ‘any’ in the policy.

    There are occasions where you might want an any, I.e you may want all clients to the proxy or to ad, dns, av, ntp etc.

    Auditors typically don’t understand the technology and back then just looked for keywords like any.

    So I put in an inverse rule, effectively permit any source other than a made up range to dst on specific ports.

    The auditor was happy with that despite it effectively being an any.

    That’s when I realised management and expensive auditors cared more about ticking boxes than actual intent behind the requirement.

    Been the same ever since.

    1. Anonymous Coward
      Anonymous Coward

      Re: One time for an audit

      management and expensive auditors cared more about ticking boxes than actual intent behind the requirement

      That is just Standard Operating Procedure for management. In general, managers are not often promoted into that place having previously been time-served in the area they will manage. There are exceptions of course, but it's a principle which applies across the board - from the Portable Appliance Tester who doesn't understand the tests he is applying, just looking for the green light and therefore (as I have seen on many occasions) will pass an item of equipment, even when his inappropriate test or finger-fumbling has just blown the input fuse, to the managers who just scan the list of equipment and check that there's a mark in the box next to each item, to the job interview panel who have a list of questions and prototype answers, and can be seen putting little ticks next to each one as the interview progresses.

      In my case the latter - for a general electrical, electronic, computer support, unblocking toilets role - involved the question "what does 127.0.0.1 mean to you?". Only one of the panellists had any idea even what the question meant and you could see the others visibly relax when I - after a short preamble during which they looked a little confused - uttered the magic word "loopback". Another question in the same interview was "how would you feel about unblocking a toilet?" Again, "tick".

      1. Potty Professor
        Holmes

        Re: One time for an audit

        When I worked for a pharmacy, we had a secure door to give easy access for deliveries, etc, which could only be opened from the inside. There was a bell push on the outside of the doorframe, which communicated by radio to an annunciator inside the vault. One day we were visited by an electrician, who proceeded to PAT test everything electrical he could find, including that annunciator. After he had left, there was a thunderous banging from the security door, our deliveryman had been pressing the bell push and received no reply. Turns out, despite the PAT pass label on the annunciator, it was no longer working. We replaced both bell push and the annunciator, and all was well until 12 months later, when the same bloke turned up and fried another annunciator. On his third visit, we told him not to PAT the bell system, as he had already buggered the previous two, but he insisted he must do it. I unplugged the annunciator and put it in the safe until he left, at which point I plugged the still working annunciator back in, and told the Manager to do the same next time the sparky arrived. I left some months later, but was told by someone who still worked there that yet another dead bell system had been added to the growing pile.

        1. imanidiot Silver badge

          Re: One time for an audit

          Sounds like a sparky that needs a good PAT testing

          1. Martin an gof Silver badge

            Re: One time for an audit

            There is also the problem of box-ticking requiring PATs every 12 months like clockwork. If you actually read the guidance it doesn't mandate 12 months at all, it merely (well, it did last time I looked) suggests 12 months as a reasonable frequency for equipment in general use.

            PATing came in while I was working at a radio station and doing the tests was usually my responsibility. Officially, all the equipment in the racks - as it had mains leads with plugs on the end - and most of the equipment in the studios was "portable", but being securely screwed and cable-tied in, wasn't really. My boss had a habit of reading the rulebook* and being responsible for that part of the station's safety and risk assessments decided that we would only test "in chain" equipment every three years or so, and obviously out of prime time. Not that doing so would put the station off-air, but bypassing such an item in order to test it would "change the sound", which usually brought complaints from the on-air talent, and the station manager knocking (the door to the cellar where we lived was on a slam-lock and needed a key to open it from the corridor outside).

            Likewise the mains-powered "OB" kit, particularly extension leads and the like, was tested much more regularly, preferably after every couple of uses and with a purely visual inspection before every use, and certainly where leads had had to be run (protected of course) across walkways or supermarket car parks.

            M.

            *he also had a habit of dismantling absolutely everything. If a sales rep. brought us some item of equipment about to be launched by some major manufacturer for us to get an early taste of it, the lid was off before the thing was even plugged into the mains and some audio. If the sales rep. was reluctant, he was either sent out of the room on some pretext or he was given a coffee and half an hour of time and then sent on his way.

      2. Anonymous Coward
        Anonymous Coward

        Re: One time for an audit

        "what does 127.0.0.1 mean to you?"

        How about answering "Home, sweet home"?

  20. Paul Hovnanian Silver badge

    It could have been worse

    They could have made everyone an admin.

  21. Anonymous Coward
    Anonymous Coward

    Oh I've fallen foul of the 'any any' rule.

    I was setting up something with Citrix, in what was supposed to be an isolated test environment. Only it wasn't, so I took out the production Citrix environment. Came out of that unscathed because I had the necessary emails clearing what I was doing. A few words were said elsewhere though...

    The other incidents have been related to successful intrusions into the network (which were raised against my team), and when we looked into it with the security team, the dreaded 'any any' was there just as mentioned in the article!

    Almost every time it was said that the rule was added because it was the easiest way to resolve an issue rather than work the problem through properly.

  22. Anonymous Coward
    Anonymous Coward

    A fuzzy website

    Early 00s I was asked to look at a website that a contractor had made for a small business as a business-card type. The business owner couldn't get in contact with the contractor and wanted to know why the website looked "fuzzy". He wasn't wrong. It looked fuzzy to me too. Hit the View Source and found very little HTML at all!

    Each web page was a single JPG image with clickable bits specified as an image map.

  23. Dwarf Silver badge

    Wirewalls

    Reminds me of a time when I looked at a firewall config that had gone in as part of an ERP design, the firewall was in front of the web tier to the business of around 4K people. There was a reasonable rules base for most, but there was an ephemeral ports type definition that was in the wrong direction as the engineer provided by the implementing 3rd party wasn't properly skilled in networks or firewalls.

    It was interesting explaining to management that their £50K/device pair of shiny new firewalls were working like a piece of wire, so I adopted the expression a Wirewall.

    It caused a brief discussion between our internal security team and the 3rd party, where the engineer was replaced with someone that understood what they were doing. The expression stuck though.

  24. Naselus

    Honestly, almost everywhere I've worked in the last ten years has turned out to have an any-any allow buried in it's firewalls somewhere. Usually added in time immemorial and requiring literally days of plodding through traffic captures to kill off.

  25. Lotaresco

    Coincidence, probably not

    I can recall exactly the same thing happening to me. I couldn't at first work out why I could route to places that I should not have been able to route to. The error only discovered after spooling to the end of the access list. I would not be surprised if it were the same contractors because the one I'm thinking about had managers who didn't know what they were doing hiring people who were cheap who also didn't know what they were doing.

    1. Lotaresco

      Re: Coincidence, probably not

      " the one I'm thinking about had managers who didn't know what they were doing hiring people who were cheap who also didn't know what they were doing."

      Come to think of it, that's all of them, isn't it?

  26. Anonymous Coward
    Anonymous Coward

    Chaos on the continent

    Intragovernmental European organisation, where most staff and contractors were in some way technical. Called in to sort out the office network, where Internet connectivity was via a firewall with the world's simplest config - a single rule for incoming traffic: "Allow Any Any". Better still, having inherited one quarter of a Class B network, someone had decided that every PC and printer would have their own fixed registered address - and in combination with the open firewall, everything was directly accessible from the Internet. None of the office PCs and notebooks had any anti-virus software, which meant everyone was blissfully unaware of the 40 or so trojans that rampaged through the office network at any given time.

    While starting the attempt to bring some order to this chaos, I detected an office PC that was generating a lot of unusual traffic, especially for a PC whose owner was on holiday. It turned out that most of the traffic was generated by bittorrent (this was not that unusual for this organisation: many employees were using bittorrent and some were even downloading huge amounts of porn at work). The user was a "software developer" who was using bittorrent to download a bunch of music, films and assorted hacked software. But in a hacked commercial IDE she had downloaded was a trojan, which had been used to launch some exploits against the servers of this organisation and several other organisations in the same country. The exploits had turned up a number of admin/root credentials and the results had been zipped up and sent to an IRC server in the middle east.

    The IT manager was away, so I emailed him report marked URGENT. He read my email three months later. My loyalty in not escalating the entire fiasco while he was away was rewarded when he got rid of me a few years later.

  27. Anonymous Coward
    Anonymous Coward

    Hackers don't submit change requests

    IT environment lead by a very bossy head of department. DNS servers got hacked as they were at least a year behind the recommended patch level and some hackers had successfully used some relatively new exploit against them.

    Boss: Who made a change that allowed this to happen?

    Me: Nobody here made any change. Some hackers used a new exploit, to which our DNS servers (at their patch level) are vulnerable.

    Boss: But if someone made a change, why did they not submit a change request?

    Me: Hackers don't submit change requests.

    I am unsure if my voice betrayed the sarcasm that I was feeling.

    Anyway, the systems guys were ordered to update their DNS servers (following an approved change request).

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021