Firewalls? Pfft – it's no match for my mighty spares-bin PC Start your week with a warning about those temporary emergency hacks that all too often end up permanent in today's edition of Who, Me? Our story comes from "Gary" - for that is not his name - who worked for a firm that offered a variety of services to the UK banking sector. Back at the turn of the century, our hero was tasked …
..... but I do need to get some w**k done this week!
I seem to have spent much of my w**king life w**king for businesses that would prefer not to spend money on a proper solution. But are willing to accept a cobbled together 'temporary' bodge that only ends up costing them more in the long run.
So relatable, I remember cobbling together a firewall and FTP server (FTP! ... so a very long time ago)
This was also an old desktop PC, spare NIC and good ole Linux
This was a "stop gap" solution, only for a few weeks. ha!
My story ended up happier though, after 6 months I was able to convince the then MD that this had become "mission critical" and in fairness they invested in a hardware based firewall and a rack mount server for the FTP services
This was a medium sized business turning over nothing like a bank, I guess the larger the organisation the harder it is to speak to someone with the hope of changing things
Reminds me of a bodged up server I built while working at a high school. We need a sever to run a single app (SIMS, the school database interface) via remote desktop for electronic registration due to the separate computer networks for admin and students. So we dug out a copy of server 2003 (I believe it was the enterprise flavor) and builtba server with SCSI disks in raid 5 configuration. All was well until I realised the order of the disk IDs didn't aligh with the order of the physical drives and my boss was a bit anal about things so we decided to scrub it and rebuild it in the correct order in case a drive failed in the future and a drive needed replacing at least we'd know which one it was.
Anyway when I went to rebuild the SCSI array, it never came up again so in the end we scratched the SCSI interface and stuck a single desktop IDE HDD in the single interface the machine had and built the server off that. The server had 32GB of ram (a hell of a lot in those days) to support all the instances of SIMS it would run (one for each teacher in the school during registration) and am IDE hard drive to load the software and OS off. It worked but it wasn't pretty. The drive was hanging loose in what would've been a cd bay lol.
I like SCSI. Mostly. As long as you don't accidentally mix LVD and HVD. Use of proper active terminators is a must. Internal termination (if offered) on most devices might work if if you only have couple of targets, but I'd avoid it and use decent terminator instead.
It can also get messy when it comes to the time to perform the obligatory sacrifices.
I once had a situation with a PC server using 2 mirrored SCSI adapters running Netware on a token-ring network that brought down the entire corporation including the mainframe because of a magic app running a btrieve database...
Shows how old I am... and what an a** saver the NSE was (pre-internet!).
SCSI=SCARY
I wondered about this. I've long had an impression that the higher the corporate pyramid the harder it is to get a decision on things that are actually in the remit of the middle managers. And also that this isn't their fault, but of a system that will penalise them for any cost it sees, but not reward them for any cost saving from bad stuff that's not allowed to happen.
Tell me about it...
In a previous life I had to fight insanely to get a back up system approved. Nothing too fancy, just a basic, working back up system for all of the office server content.
It was rejected multiple times because of the cost, which was no more than few thousand (it's amazing what one can do on a budget) and because they didn't see the value in something that they wouldn't immediately use.
The next proposal I included the approximate salary of all employees, applied a few metrics to the amount of "work" that they do that involved the shared server data and extrapolated this out into the value of the "work" stored on the office servers over a short of period of time and compared this to the cost of the backup system.
Rejected, again, of course but this time with the added threat of an investigation into my estimation of the staff salaries and how this was a privacy violation! A privacy violation based on guessing average salaries based on recruitment adverts...
About a year later two drives on the central file server crapped themselves within a day of each other taking down the entire array and preventing any work from happening in the company. We weren't allowed to keep spares of these and therefore while we immediately ordered a replacement for the first failure, before it could even be delivered the second drive failed.
The backup was approved the same day... a day too late of course.
I had this working for an educational service. Hundreds of kids' files that were potentially evidential if there were any kind of legal challenge against the LA. But no one was interested in providing for any kind of secure backup. For many years the only backups we had on or off site were ones I took myself (illegal at first because in those early days, not encrypted) with floppy discs and later CDs.
It was only long after there'd been incidents elsewhere that the idea slowly percolated through to them, with frequent reminders from me, to provide an automatic backup, in a (relatively) secure cabinet in another room. When I retired another decade later they were still umming and ahhing about some kind of off-site backup.
Of the time I built a quick and dirty Oracle DB server for the devs on a project (story I've told before, but bears repeating...) to get them out of a delivery delay bind. Making it extremely clear, in writing, that the Fujitsu desktop PC with a single PII-350 (it was quite a while ago), 4 GB RAM and non-resilient IDE hard drive (and no backups) was for development of their stuff only, I let them get on with it. As you would already have guessed time passed, I forgot all about the PC, and about a year later I was contacted by the programme manager wanting me to look into improving the performance of the "production database server as it was just so horribly slow".
This is an odd machine. Pentium II with 4GB of RAM?
I never have heard of such a thing. Is there even a chipset that supports that combination?
It would have been in any case, unbelievable, and I refuse to believe that any desktop PC could have had space for the number of RAM slots/sticks needed.
I once worked as the IT manager at a tourist attraction, and was given a miniscule budget to keep the back office and point of sale systems working. After several staff losses, I also took on the reponsibility of keeeping the Audio visua systems running for the punters. I did this on the cheap, scrapping obsolete VHS machines and replacing them with 20 quid DVD players from ASDA.
I used several old PCs from home for vital tasks, and an old Epson dot matrix of mine was used to print everyone's payslips. One day I was called in to see the boss to be told that I was being made redundant and should pack up my things and leave the building, serving out my notice on gardening leave.
At this point I requested the return of all of my equipment, with the exception of a 22inch Dell CRT monitor that I personally gifted to a lovely colleague with very poor eyesight.
Panic ensued and I cared not a jot - most of the kit I dropped off at the tip on my way home.
I mean while he was IT manager and setting this up, could he not have mitigated the risk to his employer?
If it was not himself but a 3rd party asking the company to run critical systems on loaner kit with no support and no exit plan would he, in his professional capacity, have thought it was a good idea?
You mean the employer thhat was using kit so old that spares had to be sourced from eBay? The ones using a Windows95 pc with file sharing set up as the host of the main POS database? The ones who bought a job lot of redundant kit from a closed down care home for £250 to find that it was 10 times better than the kit they had?
It was a case of do anything to keep the business running and keep 50 people in work!
Out of our team, only two of us remained.
We had tried a smooth transfer of knowledge & procedures to our field based colleagues, who were absorbing the back office, general support, asset tracking into their roles & dragged in to Corporate HQ.
They didn't want to know, made all sorts of excuses of "Too busy", everything me & the other guy had scripted & tweaked to simplify the post build process into a mostly autonomous task ignored.
At 5pm on the penultimate day, we nuked the lot, just in case of unexpected account deletions at 8am the last day,
The last day we terminated our accounts, wiped our machines & left at lunchtime.
As the only IT support with the necessary Lab experience to manage the IT effectively, I gave the department 2 years notice that I was thinking of retiring and a year to say I was going and that they should maybe give me back the one enthusiastic staff member who was at all interested in doing dual IT and Microbiology. (he had been taken away when the Lab Manager went off sick and the deputy (who hated "they computer thingys") grabbed my apprentice back.)
This was ignored by the Lab Manager.
A month before I went, the clinical director stopped me in the corridor and asked who was to replace me. I shrugged and said that she'd have to ask the Lab Manager about that, but that I knew that the others in my lab team (in a different discipline, but also tasked with IT) would help out where necessary.
Cue large explosion in the management team.
I did leave a textfile with hints...
So basically the conversation should go like this:
You: we need hardware and software for a mission critical reasons.
Manager: oh, we don't have a budget for that.
You: I can lash together something from my personal kit.
Manager: Great!
You: Since it is my kit I'll have to have you sign a contract where you indemnify me and hold me blameless and a monthly lease of BIGNUM money.
Manager: How much money do you need in that budget?
I don't need knowledge of his config. I know lilo. Changing the kernel[0] called for in /etc/lilo.conf and then running (surprise!) lilo at the command line and rebooting is hardly difficult.
It would appear some folks disagree.
[0] The kernel was the only thing upgraded in the OP's scenario, so the only thing needing to be changed would be the image called out by the config file (and possibly a kernel parameter or two). And probably just the one kernel, even in a multi-boot system. Unless you're foolhardy enough to change all the kernels in such a system at once, sans testing.
There’s always sellotape involved in any project. I don’t even see it as a bad thing anymore. First you get it to work, then you come back and fix…
…ideally you fix in a controlled fashion, but sometimes the system makes the decisions for you.
Let’s face it: this was just the hack that failed. I’m sure there are dozens of other hacks in that deployment that never caused an issue and were never fixed.
For all around use, ShurTape P- 628 will handle just about anything you throw at it. I generally buy it by the case when I see it on sale. I also keep various versions of ProGaff on hand for specialty work (wet conditions, automotive/boat upholstery backing, etc.). Both are kinda spendy, but you get what you pay for.
It's not just IT.
One of 'our' guys knocked up a temporary replacement for an interface module on a cardboard box maker that had been 'reconfigured' by a fork-lift driver. That was about 2009-2010. To the best of my knowledge it's still there.
It uses one of the original Arduinos
I think the big mistake here is offering his former colleagues help on the phone call, other than just a bit of fake sympathy. All this has served to do is reinforce to the company that employed him that you can extract useful work from people and then kick them out, knowing that they'll still help out for nothing if it all goes tits up.
Wrote a quick and dirty in 1991 specced to run on Intel 286 machines that did all the things its mission critical bigger brother did and a lot more. I was made redundant in 2003, and heard that it was only decommissioned in 2018 along with mission critical big brother.
When I took on the role of Network Manager at a large university hospital just beforeY2K. Amongst the other horrors i discovered was an old (even then) 386 desktop with no monitor or keyboard sitting on a desk in the main comms room. I was told that it was the DNS - I knew this to be wrong as there was another - rack mounted - box which was. No one knew the password for this box so there it sat, until my curiosity got the better of me.
I disconnected it and waited for the call! - Didn't wait too long before I heard that people were complaing about not being able to access from the Uni with whom we had links. I connected a monitor and keyboard and found it was running some ancient version of RedHat (nothing but the best here!) , Not having the password, I rebooted the box in maintenance mode and set a new root password to get in.
It was running Bind but it's sole purpoose was to act as a reverse lookup - i.e.provide the name from a given IP. I assume that this was someone's idea of security!
A new zone file on the "real" DNS provided some extra space on the desk:)
Some executive assistance, such as hourly phone calls about a critical problem is not welcome. You spend more time on the call or the pre-call call (to make sure you did not say the wrong things to the executive).
We had an executive planning to come to the UK and wanted to see a demo of our bright new shiny software. We were stuck trying to get two products to talk to each other, and could not fix it. In our call with the executive assistant who was setting up the visit, we mentioned the problem. Within 10 minutes of the end of the call we had a phone call from the top guru of the other product. Within another 10 minutes we were told which fix we needed, and what the bypass was.
The executive visit was a success!
I had a fault on a bit of software that we supplied that I needed to escalate to our Development Team. Support was refused until it could be fitted into their 'sprints'.
Once I managed to get them to look at it (by escalating internally), it took them <5mins to see the issue and about 1hr to fix it. To get to that point was months.
If you are reading this, you're probably using the hack that I put together in 4.1BSD (now called 4.1aBSD) for part of the TCP/IP stack to be included in 4.2BSD[0]. It was supposed to be one of those "just get us through the demo, dammit" hacks. I got 'er done over Christmas/NewYears break in 1981. Virtually every version of TCP/IP since has used it. Not too bad for a quick hack ...
[0] Just to cut the usual pack of idiots putting words into my mouth off at the socks, no, I didn't write the whole stack. That's why I said "part of". It was only about 120 lines of C in total.
Not just in software. Many years ago a local bigwig had expanded his office into the adjoining garage. As a sparks I was roped in to wire it up. However, his office shared the leccy with the plumbers next door, and they would only agree to the power being off for a prior arranged 30 minutes at some weird time of day.
So, wired up the extension, tested everything in situe, and prepared to patch into the supply. The 30 minute window opened, I patched in and tested, and everything passed. However, the linking cables passed over an area where a shelving unit was to be inserted, so I had to leave the cables and a temporary junction box which was to be replaced with a socket outlet, exposed there waiting for the shopfitter to finish off so I could then finish off.
So I waited for an opportunity for the power to be off. and trim that last cable and put in the last socket box. And waited. And waited. About five years or so later I popped in for something and the temporary connection point was *still* there. The next time I passed the office it had been demolished!
Got called to a site of a major player in the flight/cockpit simulator business. Back in the day this thing was huge, at least 30 cabinets full of 'stuff'. It had been down for several days and the simulator engineers were stumped until they 'found' a PDP11/05 in an a cabinet they had never looked in before, and its 'RUN' light was off. Desperate searches for tech information were made to little avail except a roll of paper tape was discovered with the 'Digital' logo on it, with the hand written 'start at 1000'. I checked out the 11/05 and found DCLO was asserted, fitted a new PSU and bingo, it ran a keyed in loop. A Teletype was found and plugged into the 11/05, loaded a loader then the mysterious rather large tape. It took a good 10 minutes to load and when running the whole simulator started to come alive. I never did find out what the 11/05 did.
same here. In previous life on arrival over the year I found decades old manual procedures that could be automated with some shell code mostly. Set up code to run over year at appropriate times or on the common admin server, documenting what code did what and reason for it. Needless to say on third year when I left, a common comment when something time expired ( remote passwords, etc) was, Oh you've been xxxxxed" . No suck-cessor (sic) read docu of course.
Many years ago, we did set up 3 layers internet platform. Serious stuff, no routing across 2 non adjacent layers.
The internet platform devs were utter crap, barely able to do their shoelaces. Couldn't even figure out what an IP address was, less so a TCP port.
One called me one day asking: "Can we circumvent the firewall to have $UTTER_CRAP_APP work ?".
Hmmmm, no mate, this is the whole point of a firewall :)
Sounds like the desktop PC we were asked to fix that had been sat under a desk in a certain tower of Canary Wharf. Said PC's owning department had contrived to automate emails from it to all and sundry on a daily basis, containing a batch of data thrown out by their spreadsheet following overnight updates. It's failure was traced to a new group policy that changed the power schedule and automatically shut down any PC without an active user after 7PM. Needless to say much mirth on our part as it was nothing we could do anything about and a great deal of questions to be answered by the owing team as to why this PC was sat under a desk and not registered as a proper service running on a server.
I got called in some time ago, to help a two-node Solaris cluster talk again to a two-node Windows cluster via Windows Services for UNIX. The Windows resource had left the company (not by his own will, as many others out there) and nobody could remember how we concocted that solution 10 years ago. Hell, even I was unable to remember what the heck we did. I had left that account in search of greener (and Solaris-less) pastures shortly after we got the solution working on dev/test/prod and properly documented it, and I made sure I had enough beverages to obliterate the Solaris knowledge from my brain.
The guy on the phone sounded pretty nervous, and was trying to engage me to get back into supporting that hybrid monster, and then I recalled the exhaustive documentation me and my Windows counterpart compiled, both from the official documentation and our own painful experience. So I politelly pointed the caller to the account's wiki, and told him all he needed to know was there. It was evidently enough to have them restore the hybrid Hydra to its working state, because nobody else bothered me again.
Very early in my career I learned no to suggest temporary solutions since they have a habit of becoming permanent solutions,
So now when somebody asks for a solution I design what's needed and price accordingly. The questions "can't you do it cheaper?" and "isn't there a quicker solution?" are met with blank incomprehension.
Do it nice or do it twice is my motto. The problem with far too many customers (mostly project managers) is that no matter how hard you explain that the cheap and nasty solution is only a short term solution you just know it will become a long term solution right up until the point where it shits itself. At which point you are suddenly expected to do it nice the second time round, but now somehow at zero cost because the project was completed and signed off months if not years ago and there's no budget for the solution that should have been installed in the first place.
Been there, seen it, done it.
Replaced a crappy Wintel solution (that failed more times than it worked, hence the emergency to do something) with a RedHat 6 server (not RHEL6 by the way, so shows how long ago this was).
It just worked. It worked so well that I never got a call about it in over 6 years (4 years after I left) until there was a hardware failure.
At a previous company we once had a 7-disk RAID5 chassis fail on the main server. Ended up removing all of the HDDs from their caddies, and connecting them up with a SCSI ribbon cable on the desk and positioning a desk fan to blow some air across them. It was just a temporary fix until the chassis was replaced. I think the server stayed like that until it was retired a few years later.
I didn't have to deal with it since I was consulting on another project, but my desk was near some other guys working on a different project that had recently gone live. The guy responsible for the temporary fix hoped to make sure it was only temporary during project testing and wouldn't survive to production, so he had the server that provided the temporary service scheduled to reboot every day at noon.
You'd think that would focus them, but no. The project went live and people depending on it just got used to it going down at noon which they felt was fine since that's lunchtime, right? They were actually using the sudden disconnect as a reminder it was time to go to lunch! He wanted to push things along and wished he hadn't picked noon, but one of the higher ups knew that daily reboot was deliberate so changing it would point the finger of blame squarely on him.
I said "hey daylight savings time is in a week, why don't you move it an hour earlier and you can claim the daylight savings shift was responsible?" He came in Monday morning after the shift and changed the reboot time to 11am, and quickly the howls were loud enough for him to get the powers that be to allocate money for the permanent solution he'd been promised!
My current employee is a service provider. Our customers want something done, we say it has to be done this way. If the customer does not want it that way to cut some corners we are not affected. It is the customers responsibility to leave such solution going on for too long or ignoring our recommendation.
But when I am an internal IT in a company I face exactly what makes up so many stories here: I know what is right, I get overruled, and I am still held responsible for its operation and failure. Usually by those who overruled me.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
