An important system on project [REDACTED] was all [REDACTED] up Welcome once again to the horrors of Monday, dear reader. But fear not – The Register is here to cushion the blow of the working week's resumption with a instalment of Who, Me?, our reader-contributed stories of tech gone awry. This week meet a reader we'll Regomize as "Red" – short for [REDACTED]. A decade or so back (we can' …
Red/black network communications was controlled by a secure router. Only a limited set of commands for configuration, only a limited number of protocols allowed through (and could be tightened by command) and even such things as 'data diodes'. The UI was as tight as tight and would explain at length why it would not allow you to do what you were trying to do, and if possible, how you might achieve it by another method.
Supplier of said router was taken over by a VC group and new manglement brought in.
The new router that would be supplied in place of the super secure one came from a different section of the company, located on the same physical site where manglement presided. i.e. they had the ear!
New router was Linux based, requiring root access to the command line to manually modify text configuration files.
Supplier wondered why the [security conscious] customers left for a competitor. Plus all the engineers responsible for supporting the super secure router would have nothing to do with the new system and left.
Fairly sure it is the same Aerospace and Defense board manufacturer who converted all the equipment in Nitrogen storage for long term support of 30 year contracts into 2" squares (think dongles and Windows 3.1 PCs, to more modern. Chips were in another store.) And then converted the room into an Executive Office, at about the same time as the Router debarkle!
Many of us commentards are ex-employees of said company, both in the US and over the pond, but I certainly wouldn't to go on record giving the name.
I believe all of the customers who needed to know about the router change found out very quickly from a TLA.
The same conglomerate made PLC's and other controllers in a different division - this is prior to the Venture Capitalists buying the OP's area a decade or more ago. Their Achilles Heel was that at least a decade after they should have been baking it into their development processes, they didn't know anything about Penetration Testing.
[See what I did there with Achilles?]
Large organisations think that Engineers do not talk to each other, whereas we will often help each other, even across divisions. (That's how I know these stories are from the same organisation.) It's just a shame that it's only management and sales get to go to the networking events, otherwise experience could be shared much more.
Sounds like the usual mess of people who think that "security" is a case of locking something down so hard that it barely works.
Firstly, that's just lazy. Security is supposed to be a supporter and an enabler, so you design the policy fine-grained for the task at hand and those who need to use the system. Simply blocking everything is dumb.
Secondly, it can force people into desperate unapproved workarounds just to do their job. That circumvents all of your "hard work" and arguably just makes things worse.
Yeah, something as simple as the "You must change password every x weeks and no repeats" can create havoc. Especially if a reset has to be done by a named person or an IT dept.
As when someone comes back from their holidays and needs to get working,. Or best of all, when school staff get in at 7:30 or 8AM after a school holiday, on an INSET day especially, and everyone needs to get their machine up and running immediately (else why get in early). Every single password has expired and IT don't start work until 9:00 ( read 9:30 by the time they answer the phones). At which moment every teacher and TA in every single school all want to bring back corporal, or even capital, punishment, just for IT staff.
"To add to the fun, the access control software demanded ten percent of the drive be free in order for it to create its temp files – which is madness."
Madness that Microsoft copied for Exchange? Hit a fault where incoming emails were bouncing, turns out if you have less then 10% free space on the boot drive, some versions of Exchange refuses to accept incoming emails, regardless of how much space is free on the drive/partition where Exchange itself is installed.
Reminds me of one of the first versions of internet explorer for solaris (yes there was such a thing) that could only set the cache in % disk space and minimum of 1% which on a quota based unix system was not such a good idea when 1% was way more disk then you had quota for... It was promptly banned...
There's 2 versions of the pst file. The early version (limit 2Gb) and the later versions that can go larger.
A couple of versions of outlook would ask which format you wanted if you manually made a new PST.
This is how you end up with 50GB psts, and really want to slap the person who's storing that much email in one file.
Yes. The more common size at the time though would be 100/150GB to boot. The server I saw this on a lot with a client had 100GB to boot, so if it dipped under 10GB you got the refusal to accept messages. At which point you nuked the IIS logs, the CBS old logs, and the temp files, and normally end up around 35GB free.
To add to the fun exchange 2013 (yes I know it is now a month out of support) logs a cryptic error message in the windows event log, so you have to deduce this is the problem. In my.case the lack of spare disk (for exchange) was caused by another VM faulting and consuming the spare physical disk (dumps and log files…) the exchange virtual disks were mapped to…
[Site] your board is doing something strange
[Me] can you swap it for another board?
[Site] it's only coming out of the bunker in 2" squares
[Me] can you tell me what it is doing strange?
[Site] no!
[Me] can you give me a clue?
[Site] NO!!!
[Me] if I test my system doing A might it show the issue?
[Site] no
(I had a long list of companies bidding for contracts, and those who won it (if it had got to that stage and was announced), so I did have a bit of an idea about what the customer might be trying to do.)
...
[Me] if I test my system doing H might it show the issue?
[Site] partially
...
[Me] if I test my system doing H and M might it show the issue?
[Site] maybe
Add to this, [Site] is in a different time zone, so I have done a day at work, I have been contacted at home just before my bed time and this is now 4 a.m. Code is something approaching 6 million lines with approaching 90 concurrent (and to some extent interacting) threads if all the options are in use.
[Site Manager] this is costing us 1 million an hour when will you have it fixed?
[Me] did you trial this before today?
[Site Manager] do you think I'm "£$%^ (= no, of course we did not)
How I loved working on those issues!
Security Systems look ancient when you install them out of the box, let alone how long they keep them running.
The one that controls my floor has a software interface looks like it was written in TurboPascal, and I know it was installed 25 years ago...
I'm shocked the drive hasn't fried itself yet.
Every fule knows, you don't install them, they are eventually found down in the cellar (When they finally fail), with a flashlight, avoiding the missing stairs, in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard’.
Then you tiptoe away very carefully (If still working, if not find some other idiot to take on the task of fixing it).
There is a great scene in the film "Hidden Figures" (the story of a team of female African-American mathematicians who served a vital role in NASA during the early years of the U.S. space program) about dealing with [REDACTED].
https://www.youtube.com/watch?v=eEJWtAMnAlA
The answer - hold it up to the light.
The security "fix" to prevent this.... Use darker ink!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
