What do you do when all your source walks out the door? Who has got your back up? Forget comments in code, what do you do when all your source has been packed into the trunk of a family sedan? Welcome to Who, Me? Today's story, from a reader Regomised as "Al", concerns his time at a company in the 1980s. The company was working on a project to replace thousands of ageing "dumb" …
I have tried the 'sudo rm -rf /' on an old machine that was due to be reinstalled just to see how it would go.
The answer was very far indeed! Eventually the GUI display lost all text when the fonts were deleted, and when the disk LED finally went off I rebooted in to a live CD to view the results - pretty much everything had gone bar a few in-use directories like /proc and similar. rm had even rm'd itself!
I wanted to remove hidden files on a linux machine (you all know where this is heading, right?). Unfortunately some idiot (yeah, me) had done some work as root (yeah...), and some of the hidden files were actually owned by root. So some knucklehead (yeah, me again) did some "hard thinking" (not really) and regexp'ing, and the hidden files, beginning with a "." and then letters following, or maybe even numbers (not that said idiot cared), would surely be matched by ".*". Unfortunately the command rm -rf .* also matched .. and everything under that one as well.
Even spinning rust is really quick when it comes to just unlinking the references to the inodes to clear up space.
I did not lose too much stuff, but I learned a valuable lesson that day.
I wanted to ...
dd an 7Gb *.iso file on to a USB stick, a routine and always successful operation which, for some reason, was not going well this time: the image would not get written to the USB drive.
We've all been there at some time or another so I won't ask if you know where this is heading.
As it did not copy over properly on the first and second attempts, I unplugged it from my box, cleared and reformatted it on my netbook and reinserted it into my box's USB port, where it should have shown up as /dev/sdb.
For whatever reason it did not mount as /dev/sdb or any other device.
As always, it's just a microsecond's distraction that will do you in ...
I ended up dd'ing the 7Gb image to a 300Gb SAS drive with 200Gb of of stuff I could not remember much about but of certain importance. I had been looking for a way to back it up somewhere else but it was still pending.
TestDisk helped me recover what was on the part of the drive that was not overwritten.
But it was a back-breaking week long task to sift through incomplete, duplicate and mislabelled files.
No idea what was in those 7Gb. so I'll never know what I lost till I need to look for it again. =^|
O.
You'll be haunted for years. You'll remember something but not be able to find it. Was it on that disc? Did you actually get to doing it, or just think that you wanted to but don't remember...
How do I know? A tiny insignificant slip caused a screwdriver to slip out of my hand and hit, pointy end down, directly on the soft top of a 2GB IDE harddisc (in the days when 2GB was massive).
The death throes were immediate, painful, and completely unrecoverable. I opened the drive afterwards and there were no heads on the top, the bottom ones were mangled, and there were impressive gouges that made my wonder if I could unscrew the platter and pop it into a record player.
The worst part? I knew there was important stuff on that drive, but since I had a frankensystem with half a dozen drives (and three IDE buses), I cannot tell you what. Every so often, even now, twenty odd years later, I remember some bit of source or a program I can't locate and... "I wonder if it was on that drive...".
I almost did this just yesterday. I was creating a USB install stick, but didn't bother to look carefully at the device it was pointing at, which turned out to not by my thumbdrive, but instead my USB-attached backup drive.
Fortunately it failed with a permission error, so I didn't wipe over my backups with an installer.
rm -r ./* a few days ago with the dot not quite registering as a keyboard press caused a sharp intake of buttock.
Luckily I was on a Raspberry Pi with a rather slow SD card and it was crunching its way through erasing a huge backup directory which wasn't an essential when I hit ^C
Amazingly no actual harm done. Lesson learned.
Even doing that as non-root pretty much screws the system up to the point a rebuild/restore is necessary.
Had a user do that. Twice. With a crappy script (his script took the directory as a parameter and he forgot to supply one, so the directory ended up being "/" where he then tried to run an rm -rf).
When we recovered his machine the first time, he went to show me what he had typed and promptly hit the return key and ran it again!
I was NOT amused.
Just over a billion years ago ...
... as the Internet measures time (call it roughly 1984), I received a brand new Sun 2/160. It was a dual pedestal beast, with all of 8 Megs of RAM and a pair of 380 Meg CDC SMD drives. Roughly 65 grand worth.
I decanted it from the boxes-on-pallets, plugged all the cables in, and fired the thing up. Into a beautiful new GUI on the Sony Trinitron monitor, just as advertised. Logged in as root, on purpose as there were no other accounts as yet(!!), using the default password(!!!!) ... and poked around. All was well, near as I could tell.
The plan was to repartition the disks to better suit our needs and then reinstall the OS. So I made absolutely certain I had the correct tapes, and did the one thing I had never done as a sysadmin ... closed the GUI, and from the # prompt ran rm -rf / intentionally. I was curious to see how long it would take to lose it's tiny little mind. It trundled away to itself for a few minutes, but seemingly was still working fine, enough of vmunix and the shell were in RAM and the swap partition to keep doing simple stuff. I was quite surprised, but that wasn't really what I was there for ...
So I shut her down, went and got a cuppa coffee, reached for the first tape and went to fire up the machine ... only to discover it didn't ship with a tape drive, despite one being listed on the packing list. It had a lovely bezel that LOOKED like it might be a tape drive, but the space behind it was empty. Oops. So there I was, 8AM and stuck with 65K worth of dead Sun hardware that I was supposed to demo for the Brass at 4PM.
Fortunately the 1980s Sun had Clues about customer service. One call, and their field service rep had the SCSI tape drive, the requisite cables, and a couple of VMEbus cards, (E)EPROMS and spare OS tapes "just in case" on my desk in under forty minutes. She even hung out and made certain that the system worked properly after we took it apart to install the bits that needed installing, and then partitioned it and re-installed the OS.
I made the 4 o'clock deadline ... and bought the Rep the first of many well deserved dinners.
Silly Con Valley was a very small place back then ... Sometimes I miss it.
We had Sun hardware from around 1988 until the mid 2000s when Linux was good enough and PC hardware sufficiently better value for money to move away. Early on we were also impressed by how good Sun's support was, not just getting spares but also technical queries, etc.
Alas our last Sun purchase was a 7410 storage system just before they were bought by Oracle and both that system, and the quality of support, turned out to be dreadful!
Sun were lovely to work with. Proper engineers without too many egos.
Went in on a sales pitch; the buyer said he needed to take it upstairs but was happy with the pricing model, offer, etc
And then he invited me and my colleague back to his house with his family for a BBQ and a few beers.
Sun customer service was definitely brilliant, especially with Gold cover (hold the line, I’ll put you through). However a fellow Sun 4 sysadmin mentioned that deleting vmunix didn’t have an effect for about 10 minutes, whereupon the kernel had a bit of a fit, he’d discovered one day!
pretty much everything had gone bar a few in-use directories like /proc and similar. rm had even rm'd itself!
On Unix filesystems, unlike Windows, the directory entry is NOT the file - it's merely a directory entry. The actual file is stored under just a number - the "inode" you may have heard/read about.
So yes, rm could delete itself - or rather it could delete the directory entry you know as rm. Under the covers, the inode containing the binary will still exist but without any references to it. When rm exits, the inode will no longer be in use, and as it has no references to it the filesystem will clean up and remove it.
You can also have multiple directory entries pointing to one file - there are many commands that are actually one binary, but with multiple directory entries linking to it.
This is why Unix/Linux/etc. systems can be upgraded live. Binaries & files that are in use will simply remain as orphaned inodes until the process quits/closes the file. In the meantime, a new directory entry can be created that links to a new file under a different inode. Thus software that is running will keep running, using the old binary - then you restart the process, it loads in the new binary and the old binary finally disappears.
Windows filesystems are different - the directory entry IS the file so to upgrade things, all the relevant software needs to be stopped, and when it's system files that means the (often multiple) obligatory reboot cycles so the system can go into a minimal stuff running mode and be able to replace files.
> "On Unix filesystems, unlike Windows, the directory entry is NOT the file - it's merely a directory entry."
You are technically SO OFF you'd be better of deleting your comment, You embarrass yourself to a level way beyond your grasp. And yes, I may get downvoted for saying the truth here, well, that is life. No one like the truth to be smacked into the face, but that is just my German trait to be this direct. Just google for NTFS information, what the $MFT ist, and so on.
The reason why rm can delete itself is because it is a active process loaded into memory, and once it is there the original executable is not needed any more to keep on doing its job of killing everything. This can be done on Windows as well, but here is where Windows, at least "New Technology" versions, kick in with their locking mechanisms. Go back a few versions and you will see: Oh, that worked in the past as well to kill yourself!
> "You can also have multiple directory entries pointing to one file"
Windows can do it since NT 4.0, maybe before. Hardlinks, softlinks, junctions etc. They are on the filesystem level, not a .lnk file. And since Vista microsoft uses this to a HUUUUGE extend on your userprofile and in the windows directory. Download Hardlink shell extension, which gives an easy to use UI for that, and adds overlay icons in the explorer for Hardlinks and other stuff. And then start digging in c:\windows\system32.
This makes it possible to run Windows 10 or Windows 11 with just a 30 GB SSD with their default truckload of useless stuff, and still have 10 GB free to the usual "Windows Update Temp" stuff. Windows 8 needs 40 GB for that, and Windows 7 can fill 128 GB easily unless you manually use cleanmgr.exe with admin rights.
> "Windows filesystems are different - the directory entry IS the file to upgrade"
No the directory entry is just a pointer, the actual file entry is in the MFT. And to get a half truth out of it: If the file is small the NTFS filesystem can opt to store the complete file or subdirectory within the 1KB MFT record instead of allocating a cluster for it. So you are not entirely wrong about "directory is the file" if you stretch it, but you are not right either - the truth is more complex.
I recommend reading this official kernel documentation to see how much a directory is "not a file" for ext4...
I am not defending Windows over Linux or MaxOS and anything else, all those OS-es are shit, they are just a different kind of shit you have to put up with, and they have their place to be useful. However I have a problem with the religious type of fanatics spreading false information, especially since that orange guy in the US and that leek-root-hair-type guy in brit appeared on national government level...
After a particularly nasty argument with the "manager" who was supposed to run the department I worked for at a very large British Telecommunications business, I went back to my office and constructed a particularly vicious resignation letter - which I immediately mailed to make sure that I wouldn't change my mind. I then spent the hour and a half before I finally left the building carefully deleting the contents of both my personal machine and the department file server - everything that I had contributed to the project team was deleted irrevocably. All the rest of the Department's work was still there, but would be entirely useless without the work that I'd done. All the electronic design and firmware code I'd written was utterly deleted.
I felt much better after my thorough deletion session.
A week later or so, when applying to a subsidiary of a Japanese electronics giant, I was asked at interview if I was "the one who deleted three years' work from the system at ....."?
I admitted that yes - it was me, and explained why. The manager interviewing me asked if I'd tangled with a certain Mr Ful**n-S**the at my previous employer? I said that this was the reason I'd left there. He came quickly 'round the desk, and vigorously shook my hand and welcomed me to his team! Apparently he'd been my predecessor at the Big Telecommunications conglomerate!
Anon, because the Big Telecoms organisation are still smarting from the loss of the work....
Yes and no.
My old boss was a dick. The kind of person who would sit next to you and send you an email rather than turn his head to have a conversation. The kind of boss who would somewhat loudly criticize you and never apologize when proven wrong (like the time he had a go at me for requesting an order for WORM media because we were running low - because he wrote a bit of code that told him we weren't and didn't need any. Until I looked over his shoulder and pointed out an error in his code).
The week before I left I had a genuine failure on my workstation and had to rebuild it. I just didn't copy back my documentation. I got a panicked call a couple of weeks later because he realized he didn't have any docs at all.
So here's the 'no' bit. I made sure the guy I worked with who was left behind had all my docs. I'd spent time going through some of the stuff with him. I even took calls from him if he got stuck (he was a friend of mine, and it made him look good to be able to swoop in and save the day).
Went to a consumer electronics store with a mate who was looking to buy or build a decent PC in the mid 90's to be used for accounting and salaires on his father's farm. Sales guy was very pushy trying to upsell the most expensive machine he had in the store, pushing some big name box with an Intel MMX over the faster but way cheaper Cyrix, because it had a gaming graphics card and refusing to answer questions about upgradability - like getting a cyrix machine and adding RAM and an extra drive would be cheaper than the top of the line at the time gaming rig (that was nothing special other than the high price), and ignoring our question that storage space was more important than cpu and graphics power but the sales guy obviously knew more about IT than 2 young geeks who talked Intel technical manuals over breakfast at high school. After his agressive speech and ignoring our actual questions for like 15 minutes and finally worming out of his grasp and deciding to roll our box, we wandered around the computer section, logging out of windows on all 6 machines on display and quietly running a format c:
Petty revenge from a bunch of 16 year olds. We were young and dumb at the time. (Now I'm just old and dumb :) )
Petty? Well deserved, I'd say. At least you knew the obvious bullshit. How many punters don't.
I had something similar when looking for a CD-ROM for an Acorn A5000 in PC World. It needed to be a standard ATAPI. The clueless git there was trying to sell me some weirdo (expensive) thing telling me it would plug straight into my Amiga A500's SoundBlaster. A what now? <sigh>
I once caught a sales dork at Office Depot trying to upsell a customer a gold plated USB cable, stating that it would make the print jobs go faster. When he walked away, I politely informed the customer that it was nonsense and to just buy the cheap one.
Ah the infamous cable upsell. Buy a 50 quid printer and having to pay 8 quid for the usb cable.
When I was doing professional network setup one of my suppliers was a mid size cable wholesaler. Network cables by the kilometre sort of company, and did a sideline in consumer cables. They had one off-brand doing (new then) HDMI cables, going from 1.20 each going to 89 cents for 10 or more, before VAT (so add 20.6% at the time), for a 1 metre-ish cable. My local supermarket had that exact same brand, going for 25 euros. Markup much?
USB printer cables were like 35 cents each IIRC
I don't think I would have waited. Sales droid also needs to know that's nonsense. (He may have been misinformed by somebody else, just parroting the line.)
I'm still wondering why Wal-Mart sells *FLAT* Ethernet cables. They look too thin to be proper Cat5e.
Mine's the one with the RJ45 crimper in the pocket.
That's kind of like when, at around that age, I went to buy a calculator. The one I thought I wanted didn't have a price ticket or any specifications. The sales staff were bunched together in the middle of the display area, as far from the calculator section as they could be and refused to acknowledge my presence. So after a few minuted of vainly trying to get their attention and another few minutes entertaining myself (bear with me) while I waited I went round to as near as I could get to where they were all stood chatting Saying loudly, "You've wasted my time because you didn't want to come and help me buy from you. So I'm going somewhere else and I've wasted some of yours." And put down the stack of price tickets that I'd collected from the display models (those that did have them), while I'd been waiting.
Those were the days.... AMD K5/K6 cpus ran 50% faster than Intel for integer ops, so if you wanted to run non-GUI stuff (linux servers), that was what you purchased
Cyrix were ok but rather....toasty (not as toasty as 80286s - who else got serious fingertip burns off one?)
We were all getting made redundant and as our notice period got shorter, our workload lessened. As I am someone who can't sit around and do nothing in work (despite the circumstances!), I got hold of one of many spare laptops and over a couple of weeks tried to work out the deadliest and quietest way to trash a Windows 98 install using just basic OS commands. If I remember correctly, I ran a silently-run batch script on start-up which deleted particular directories in a specific order and the end result was that after about five minutes of silent deletes, the filesystem had so many things missing that it just stopped working.
As I said, it was purely for educational reasons and to while away my time - I had declared all my documentation "up-to-date and accurate" (ahem) the previous week - as even back then trashing systems on departure was a perilous thing to do.
I remember back then occasionally working from home with the code - mainly because the day to day issues at the office were a complete pain and there was never the time to review all the code because people were messing with everything. So when there were bugs that nobody could fix I'd take the code home for a day and fix it, bringing the fix back to the office on a floppy disk in my pocket (icon - LOL).
At the end of the life of the Met. Office's IBM Power 7 775 supercomputers, which I had supported the entire time they were there, the Met Office data security team asked me to carry out the complete logical destruction of the 4 systems that made up the environment (I had the appropriate security clearances and technical knowledge and their trust as I had been there for the entire life of the systems. This was so they did not have to monitor the process too closely, beyond checking the various processes would definitely clear any restricted data, and a quick interview when it was completed to sign off that the checklist had been followed).
Through a combination of primary disk initialization to destroy the RAID arrays, through scrub operations on some of the particularly sensitive disks, and format operations on the disks of the Hardware Management consoles, over the course of the first three days of the process, I systematically destroyed two supercomputers that were still in the top 200 of the top 500 list, and two smaller systems that were for test and collaboration purposes.
It made me very sad that the systems I had invested nearly 5 years of my working life to keep running had to be destroyed, and I had mixed feelings about the being person doing the logical work (and some of the physical work, as I was helping the IBM decom. team who had flown in from the US as I was 'escorting' them to make sure they did not touch the other systems). The entire decommissioning and removal process stared on the Monday morning and the systems were on the trucks removing them, and the floor tiles replaced by Thursday evening. The only thing that was left was the water cooling infrastructure and floor bracing, which the Met Office were removing themselves.
Sad that that gig had to end, but very happy I had the chance to be involved at the centre of the project. It was a chance very few people get to enjoy.
For the curious, is it possible to say vaguely what could be on a met office supercomputer that would be quite so sensitive?
Not Met Office myself, but it used to be part of the MoD, and they work(ed) on the assumption of everything having sensitive data on it just in case.
And presumably some very specific forecasts for some very specific locations. And maybe weather data shared from some very sensitive- in the intelligence sense- equipment. I know nothing of the Met Office beyond what anyone would. But that seems likely, somehow.
I had some conversations with engineers on the bleeding edge of machine design. The latest, greatest, most powerful machines with serial numbers 0001, 0002, and 0003 went to the following organisations:
0001 - the folks who design nuclear things that go BOOM!
0002 - the folks who listen to the folks on the other side who think they can't be listened to
0003 - the folks who tell us whether tomorrow is OK for a picnic or whether we should dress for snow
considering that all three organisations are part of The Government, and are looking for as much computing power as they can get, it wouldn't be inconceivable that "spare" time on one organisation's computer might be requested by one of the other organisations, thus the requirement for all three computers to be treated as having (or having had at one time) classified information on them
I've done the occasional job that involved MoD computers and yes, they tend to err on the side of caution and assume anything with storage may have been used for "secret stuff" at some stage in its life. They even pay for manufacturer warranty parts because the faulty parts have to be retained for destruction rather than exchanged. Most parts have updateable firmware so could be used to exfiltrate data if allowed off site.
Agreed. I have worked with a piece of software (runs under DOS, from a floppy, believe it or not) that deals with accurate tidal simulation around the coast of the UK. Because it was developed for the RN, it is classified, even though much of the data and even the simulation is essentially 'public domain'.
Anon because.
Actually, the weather models are available for other organizations to use. There is a licensing process, but it is often advantageous in the weather modelling world to cooperate with others in the same field.
The commercial side kicks in not from the weather model itself, but the presentation tools and the forecasts themselves. The model itself is not as useful, as not everyone can afford to run a large supercomputer, or have access to the extensive input data from around the world used to feed the forecast.
Sad that that gig had to end
I know the feeling - you've spent years making sure everything is as good as you can get it, the cabling is tidy and documented, backups are working, there's enough space for the systems to keep working on their own for the near future, etc.
Then you know that the week you've left, the ******** running the place has shut everything down ("shut down" == yank all the cables) on the basis that he's not allowing anything to stay running that he doesn't understand or that I put together. What hurt the most was knowing that when a load of customers' stuff went offline a week later (yanked cables on DNS primary, didn't know he needed to promote a secondary to primary) he'll have lied through his teeth to the customers affected.
I've worked at several places where the source code was really the engineer who wrote it, and despite having the code left behind, the intellectual process that created it was gone and the projects withered on the vine without it. Too many managers only consider their "digital assets" to be of value.
Too true.
I am literally currently dealing with one who told me he wants <the program> to do X and Y, and he wants the result to be Z.
The bit he's missing is that if X and Y are done (inside or outside of the program), then the result is A, not Z. So despite me doing what he wants, I'm still wrong.
I am literally currently dealing with one who told me he wants <the program> to do X and Y, and he wants the result to be Z.
The bit he's missing is that if X and Y are done (inside or outside of the program), then the result is A, not Z. So despite me doing what he wants, I'm still wrong.
Seems easy enough to me, let the program do X and Y and present result Z. Reminds me of the story about the programmer, who had to program some account ledgers. Add both sides up and the result must be the same. It took a couple of people quite some time to figure out he added one side and wrote the result under both.
Deciding what the program should do was never going to be the difficult bit.
Unfortunately, there are fewer development tools available for modifying the brain of a moron. I ended up writing the entire issue up in our workflow management system, spelling out what X was, what Y was, that adding them together equals Z, and if it needed to be A then X or Y should be changed accordingly. Tonally, it was one notch above "explaining addition to a child". I would be kinder, but this guy pulls this stunt two or three times a month.
When we drew up the terms of a startup, I included language that indicated that I would be sole owner of all the software that I generated for the new company should the company go under within three years. As is typical for most small startups, we went under before two years was up. Bad management. The Bank shut us down and took possession of all equipment (complete with Santa Clara County Sheriffs ensuring we didn't walk out with the kit).
I managed to start the SCC disk formatting and my computers wiped of all traces an hour before they got there ... There were no backups, because expensive. (Have I mentioned bad management?) However, I was quite secure in the knowledge that I had already transferred the entire code-base to my coloed server under Bryant Street in Palo Alto. The new-fangled Internet was handy for that kind of thing.
It took 'em a week to realize that the code no longer existed, and that without it all the hardware was so much scrap. As the only IT guy in the former company, they figured it must be all my fault, and the Bank sent me a very threatening letter. Which I immediately passed to my lawyer (brother). Who sent them a copy of the initial terms. I never heard from them again ... but I did find some of our kit at Weirdstuff Warehouse, selling at scrap value. I ignored it. The whole project had left a bad taste in my mouth.
I still have the source, but it's useless now. We were trying to fill a niche that was filled by a couple of multi-nationals a couple of years later. It's a shame, really, we didn't even have patents on our tech, because spendy. Perhaps I should mention bad management?
That's interesting Jake. I wonder whether the bank would have been as willing to fund the company if it had known that any developed code could never be treated as an asset (at least in the first three years) should the loans not be honored. If they had known, they may have been more prepared to try to get it over the three year threshold.
It sounds like it was not particularly valuable to them anyway, unless the product could have been sold and brought to market by another company. But investing in a proto-software company, where the only real value in the company is the software they're developing, without having the rights if the company fails, sounds like a financial hiding to nothing. I wonder whether the bank managers approving the loan kept their jobs?
Of course, you say that you were the only IT guy, but I'm not sure from that that you were the only person developing the software. If there were other developers, as the agreement only appears to cover your code, what happened to the rest? If there was anything else, your deleting it from the servers could be seen as willful destruction of property, and if you allowed yourself to be on the board, could be corporate malfeasance (although I'm not sure whether that would have covered a small start-up - I don't know that much about US corporate law).
Sounds like your brother had much of this covered, though.
Basically, me and four or five other guys were enjoying a beer and wondering how we could MAKE MONEY FAST ... It was right about the end of the early '90s mini-resession, at a time when banks were starting to sit up and take notice of the money making potential of Silly Con Valley.
Well, we came up with an idea, something the world needed. It could be built with modern tech, we thought, and with our range of talent all we would need was a finances guy. The guy we hired played hard and fast with a couple of financial institutions, playing one against the other. I honestly don't think they bothered to read our own internal already drawn up contract.
It wasn't a software company, it was a combination of specific hardware and the software to read that hardware and display what it was reading. In near real time.
It turned out that the hardware capability of the day didn't quite match what we were trying to do with it. Drew too much power to last for very long on battery power, and it was a hair slow for the more 3D features (OK, two or three hairs slow).
But the writing was on the wall ... We had a working(ish) prototype, and hoped we could keep moneybags off our backs long enough to reap the benefits of the next CPU upgrade. Didn't happen. The financial institution wanted their cut, and they wanted it now. So they closed us down. Last time I ever went with a bank for this kind of thing ... Sand Hill Road is the way to go!
ANYway, I moved on to other projects, and a couple years later Hitachi and Seimans came out with what was in essence our project ... but they targeted Humans in First World. We were only targeting Veterinarians here (medical certification is a bitch) ... although our device would work well on humans as well, and we had plans on selling it as such in the Third World ... and once we had proof of its safety, here in North America and possibly Europe etc., as well.
What was the product? An ultra low cost Ultrasound Machine built into a laptop, with just a subset of the "all the bells and whistles" that you'll find at your Doctor's office. We weren't targeting professional radiographers, we were targeting the little-guy physician, offering a tool for quick & easy diagnosis ... "Yes, Mr. Client, your pup DID swallow several little balls" ...
Can't blame us for trying ... but I do sometimes wonder "what if".
A company I worked with (hence AC) had a software engineer who died unexpectedly on holiday, she must have been only around 50 and no signs of ill health so a real shock to all of us.
However, said company decided to wipe and re-purpose her PC shortly after. About 6 months later I got a request if I had any information on some software, eh?! Turns out she had made changes and built/deployed the new version that fixed some things but the changes were only checked in to the local disk, not the central source management system.
Had they only put in a new HDD to re-use that PC!
We had a developer who got his marching orders midway through the day because he threw a spectacular tantrum, insulting everyone form his team leader to the company manager (the reason was n his side, the form definitely not). He didn't even lock his machine before collecting his personal stuff and storming out.
What did the genii in leadership decide to do? Sent his PC to be reimaged without making any kind of backup first. Problem was he had his hands in several projects, with many checkouts under his belt.
Redoing what was lost that day took several people working for several weeks...
Impatience on every side never got to anything positive
Impatience on every side never got to anything positive
Add stupidity and incompetence to that. One place I worked, had backups, but no test area so testing was done on live data. We ended doing restores from tape probably 5 times (with the added costs of the whole company shutting down until the restore process was done). At that point, someone finally listened and decided a test environment isolated from the live data one would be a good idea.
Of coure, having a backup process in place is only half the story.
I once worked in a government scientific organisation that used networked Windows PCs and had a process that backed up certain parts of the shared drives. One directory of the shared drives contained a large number of small files of geophysical data totalling about 600 MB which was input data for a tsunami simulation system. According to the powers that be, and the file system viewer, these files were backed up. But I had my suspicions, so I moved one of those files to multiple other places and then requested that the file be restored from backup because "I had accidentally deleted it". I was never told the details, but I was told that the file could not be restored. As my contract was ending, I just alerted my boss to the problem before departing.
Moral: backups are pointless unless you test the restore process.
My IT correspondant on a remote site was instructed to rotate the backup tapes one a day for a week, checking the report the following morning and call a ticket if there was an issue.
One day of course the accounts got corrupted and a restore was called in. All tapes were blank. And the backup software since god knows when was wiping the tapes then starting the backup, and stopping because "drive dirty. Please run the cleaning tape".
Up until 11 pm doing wizardry to get the data back, and using a language that was making the air turn blue and the 55 year old IT correspondant turn red. He did stay with me until the end, which was nice though.
It's nice that when people offer to stay. They may not be able to help, but the fact they offered makes you feel appricated.
Reminded me when I worked at a remote office (2 hours away). Sales weasel wanted to set up demo, talked to me about want he wanted - then buggered off home. Bugger this - did a half baked job and went home soon afterwards
I suspect that they were doing something like syncing the data from one directory to another, complete with removing deleted files from the copies during the sync.
This type of 'backup' will only save you from some types of data loss, something that I have to keep reminding some of our architects on a regular basis.
You need a versioned backup system that will keep copies of deleted files offline (or at least protected from being overwritten so no ransomware can encrypt it) for a defined period to call it a proper backup.
And, of course, you need to check that you can restore the files, plus the ability to reconstruct the systems important in the process of restoring the data in case of a complete disaster.
Backup *AND* Restore ?? Back in the early 80's, the Northern Telecom 445/585 system had a 'interesting' feature. Backup worked just fine. Coming to restore a few files from customer's system, it turned out the restore process first built a directory in memory of the entire tape before the restore process. Lots of files = run out of memory & restore program dies. The solution was to reconfigure the system to support just one screen & no printers making the OS smaller and more space for the 'directory' (no reference in the documentation of course). Just made it. Phew !!! (It also had the even more interesting feature of deleting all file indexes occasionally. I never did work out how to do this on purpose.)
"Ever been tempted to dash out a quick FORMAT C: or a sudo shred just for giggles? "
Seriously? No!!!
It took me 10 years of fricking hard graft to pay my mortgage off and buy the nice things I have, the last thing I want is to end up on the end of a seriously stupid lawsuit 'cos I couldn't contain my childish temper and wiped machines and data, pathetic. Never burn your bridges.
I never understood the "me and mine" mentality, closed minds trying to hoard info to make themselves invaluable, you know what companies do with that? That find ways to undermine it and work around you until you become obsolete without even realising it, I've seen that done on at least a dozen occassions by people who thought they'd make themselves invaluable. I'm the complete opposite, I write everything down, always post my source code to repos, document my projects and comment my code to the Nth degree. I don't want to be invaluable and i don't want to be tied to the same old boring crap work for evermore. I don't want anyone asking me for anything or bothering me with daft questions. , if you cant' be arsed to read documentation then i can't be arsed to help you!
RTFM forever!
At a previous job I was doing repairs of Apple laptops, AIOs, and the odd Mac Pro. I had set up a netinstall server to speed up the process of imaging systems. Anyway, after a time it became increasingly clear that I was being set up as the fall guy for some serious management failures, so I decided it was a good time to redo the images on the server. There were some legitimate reasons why I wanted to remake the images such as integrating recent OS updates, so I deleted the existing ones and then just sort of slow walked making the new ones. Sure enough, before I even got around to making any of them, the guillotine blade fell. No one else had any clue how to set up a netinstall server, let alone make images for it, so they were left with the only option being to use the restore CDs, which were slow and could only be used on one system at a time. No one else there even had the necessary certification to do repairs on Apple units and I made sure to remove my account from their GSX account same-day so they couldn't try misappropriating it.
Many years ago back in my days as EvilKontraktorSkum I took a contract to provide a extra manpower on an engineering project which was putting a large amount of money at risk due to the software component falling somewhat behind schedule.
When I got on site it turned out that development of the software had effectively been hijacked by a single engineer who regarded any attempt to involve anybody else in a project he’d laid claim to as a personal fiefdom as a personal attack and went to considerable lengths to actively sabotage any attempt at collaboration. I thought things might be going in the right direction when eventually, following a series of increasingly tense meetings involving engineering management, project management, HR, and other members of the engineering team it was announced that the individual concerned was going on some kind of extended sabbatical leave, taking the opportunity to address some stress related issues.
Imagine my surprise on going down into the development lab the following Monday I found a large empty space where the development kit had previously been and it turned out that he’d popped in over the weekend “to retrieve some personal effects”, loaded the whole lot into his car (with the aid of site security!), and subsequently announced that from this point on he’d be taking on the project from home.
I did manage to recover reasonably up to date source code, recreate the build environment, and get things moving again but it became clear that whatever else the client needed to sort this mess out and get the project back on track it wasn’t me (or at least not without the assistance of mental health professionals) and a few months down the line I declined a (lucrative!) contract extension in favour of a somewhat healthier working environment.
[1] Client was a provider of subsea control systems for offshore oil and gas installations…
"a single engineer who regarded any attempt to involve anybody else in a project he’d laid claim to as a personal fiefdom as a personal attack and went to considerable lengths to actively sabotage any attempt at collaboration"
It was Lennart Poettering, and I claim my 5 quid.
Client was a provider of subsea control systems for offshore oil and gas installations…
The more I learn about the North Sea gas infrastructure, the more I see the need to switch to renewables.
Back when the land side pipeline to Bacton was being installed the welders were very happy the fields had a copious supply of magic mushrooms growing in them.
Only ever knew one - at the time he had moved on to be Head of IT. And who famously said: "the mainframe stays as long as I am here."
More infamously, he was very reluctant to have any changes done to "his" mainframe programs and hence largely dictated what business could or could not do - without understanding much of it. For example, after years of refusing to implement a certain business request ("it cannot be done"), business decided to acquire a system to fulfil said request. And suddenly, our Mainframe God quoted a two-weeks effort to implement this change request. Two weeks passed, and a bit more, and the mainframe did something which was not exactly what business asked but enough to deter them from acquiring another system for the time being.
Luckily, unlike "real" gods, i.e. imaginary friends or foes, Mainframe Gods are indeed mortal. And for the record, no, I did not help with verifying his mortality. Actually, even before his departure, a project to replace those legacy mainframe programs was started (might have helped with that, though).
Having seen the former field staff, now relocated to Corp HQ & one of those field staff would be sent out to a different remote region each week for support, taking our jobs, the new centralised team was:
Disinclined to participate in handover knowledge & suddenly too busy for any web handover rmeetings.
Refusing to perform tasks that had previously ensured we got nigh on 100% audits from PWC.
The new team lead:
Deciding he was going to shirk his responsibility over Christmas & stay home with his head in the snow when a outage occurred, instead of taking charge.
Bring in a new machine naming convention 7 days into the new year, when we had already started using the existing one as per usual with the new year.
My sole remaining colleague purged everything we had crafted to automate\simplify the post image setup in the run up to departure, as the new team weren't going to use our image, but created a whole new one & on the last morning I deleted our accounts & we re-imaged our machines, instead of the usual backup first.
Worked as Tech Support for a company that had an SQL client written in an interpreted language. Way too often the call would start off as a simple question with the obvious answer being to modify the code to increase a limit, or some other minor change. Customer would then continue down many angles of enquiry that might possibly fix the problem but not involve changing any source code. Yes, they had lost it. A strings command on the p-code would give them some SQL statements and other messages, but that was all. We had some very insistent customers, and high dollar requests, to solve their problems. In the end we produced an un-compiler (it wasn't a true decompiler as it just dumped out descriptive names for the embedded opcodes) and sold it to hapless customers who had a rogue employee, or had mislabelled their back up tapes. or were just generally asleep at the wheel when it came to source code control.
What Dave was thinking when they did this:
> "During the afternoon of that day, Dave's manager looked out the window to see Dave loading boxes and boxes of floppy disks into his car," said Al.
I'm guessing he wasn't feeling particularly rational at the time, but unless there was some sort of special contract involved, both the floppy disks and the IP they contained belonged to the company. So even in the best-case scenario he'd have been absolutely hammered in court and forced to return them.
Equally, given how fragile floppies can be, I'd have hated to be the person trying to restore all the data from said disks...
when I left a company 10 years ago. it was 6 months after being told I would be getting a raise at annual appraisal and then getting shafted when the appraisal was done just 2 weeks later.
well in that 6 months I managed to offsite all my work. and make sure my machine drive had been used enough to make everything completely unrecoverable, ok to be completely sure a month before I left the day before I handed in my notice I had rebuilt my machine on to a new disk and redeployed my cleaned original disk in to some unknown accounts drone's machine.
my last act as I left the office on my last day to go to the Pub was to kick off a reinstall of the standard base build company image from the deployment server on to my machine, apparently they were shocked on Monday morning. :-)
It didn't help the department that another other long serving person in the department changed roles and departments on the same day I left. ok he was only a floor away but was suddenly uncooperative and "forgot" everything about the internal IT department setup and his new boss (great guy) kept him busy so as not to make him available to them. ;-)
But not being a complete Ba5t4rd I had copied all my personal documentation on to a friends Hard Drive, and gave him a quick primer of what was where in the server. despite him not being in the internal IT support team he is more Technical than us but was external customer facing. still in contact with him to this day.
About a week after I left when my replacement (who had been in the department five months before I left) was telling the Boss that I had Not left ANY documentation my mate was able to look over their shoulders and go "IT shared drive Mobile contracts folder, Blackerry_users.xls Oh look there it is." he did this on more than one occasion.
my replacement only managed a year and in that time needed more staff. the same ones I had been told I didn't need and couldn't have which is one of the reasons I had left.
Anonymous to protect the Guilty.
Why format the drive yourself when the supervisor you hate can be Jedi mind tricked into ordering it to be done herself?
I had given notice and it was my last day. I taped an envelope to the top of my laptop marked "DO NOT FORMAT - read first". Inside the envelope was a printout of an email I had just sent my supervisor providing the laptop password and informing her that on the laptop is a password manager with all passwords, the department's local hosted CRM with 7 GB of contact data, and the encryption keys to the department's offsite backup.
As I left the building for the last time I smiled. She hated me as much as I hated her so (1) she would delete my email unread, as she always did, and (2) order IT to immediately format the laptop, never mind what that $%& might have said. Which was immediately done, said a former colleague in a phone call a week later.
- made a .pst of all the project-related mail in my inbox, just in case others needed it, saved in the project Sharepoint directory
- ZIPped my local directory of project related files and saved it in the project Sharepoint directory
- spent my last two weeks answering questions for the guy who was taking over the project
Loved the work and the people, but the corporate silliness and inertia drove me to retirement.
(I may have a backup of some stuff on a personal HDD for reference only, you understand, but if asked, I would deny that)
You see this is why, when firing a system admin, you make sure you escort them off tbr premises and disable any accounts immediately.
It’s not foolproof, or proof against a truly organised but malicious employee, but it does reduce the problems a lot,
When firing, sure, but it doesn't help against a dead man's switch. And in the case of a system admin leaving of his own volition, you better check if the karmic scales balance.
Small business I worked for in the 90s, long since defunct, boss was increasing annoying, and the admin even worse. I suspect she was sleeping with him; he was certainly shagging anything else in a skirt that visited the office.
In a fit of pique one day, and thanks to a unlocked door, I dropped a file on bitch's payroll machine that, should it detect I was no longer employed on the next pay run, would change the company name on the payslips to something rude, and adjust the tax paid figure on all records by a fixed amount such that, I hoped, everybody paid should get allocated a nice tax refund, payable by the company.
Unfortunately I knew they wouldn't really suffer - all payments were by hand written cheque, so it'd be spotted immediately. And, unusually I know, they were obsessive about backups. (and even if they weren't, it was a reversible change.)
So time came I found a new job and was glad to get out of there, and the file triggered. They bitched, withheld the last lot of wages they owed me, and vaguely threatened legal action. I think it was the rude name I called them on every payslip that did it. Still worth it, but I'd not do it again, or not so obviously anyway.
That happened to a chap I used to work with,,,,, in a Government Department in Whitehall! The ex-employee was escorted from the premises straight from the HR Boss's office where he was told that he was going on "Gardening Leave" for two months before his Contract was terminated....
He'd seen that this was coming - his Department was being blamed for all sorts of IT sillinesses that the Blair Government had perpetrated - so he made sure that there was a "Fail Nasty" built into the Departmental Network - if it wasn't stopped by 09:30 on each workday, every machine on the entire network would be nuked....
Of course he wasn't at work on the next work day, and every one of the Department's PC had irretrievably Blue-Screened by about 10:15. It took his replacement a couple of months to restore the entire network - including all the outstations - and quite a lot of the Financial, Personnel and other Sensitive data was never recovered....
Anon because it might be treason.....
About six and a half years ago I left a university job I'd been working in for a long time. For many years myself and a colleague had been running what we referred to as the "AV server". I can't remember the hostname it was given now, but it was not related to its function, that being rather clandestine.
The server was the brainchild of my colleague who retired a few years before I resigned. We'd cobbled this server together with various bits of old hardware. Eventually it consisted of a rack mounted server and three rack mounted RAID arrays, with the disks for those sourced from previous hardware upgrades to other arrays. The server ran Linux with access mainly via Samba, although SSH and FTP access was also possible. I seem to remember it had around 12TB of total external "AV" storage which we'd aggregated into one large volume via LVM. Not the most resilient set-up, but with the underlying RAID it was good enough.
The server was accessible to a select group of geeks that worked across different areas of the university, some in central IT, some in faculties. Originally the server was just intended to share CDs. People ripped their CD collections and uploaded them, as I did. It was my colleague who then decided to start adding video also, downloading stuff via torrent sites and uploading it to the server for the benefit of all. I uploaded some stuff I was watching to the server and also helped my colleague out a little bit by obtaining stuff via torrent and uploading it to the server for content that I didn't watch, but others were.
When my colleague retired I was left with this server. I kept it ticking over, but had no interest in adding more content to it or curating it, which was something my retired colleague had been undertaking. When I left, a few of the other users of the "AV server" had already left the university previously. Nobody was uploading anything to it or seemed to have much use for it, so I decided its time was up.
I was leaving on a Friday, but was only in for the morning and planned to do nothing more than sign my keys back in to security, relinquish my ID card and say goodbye to colleagues in my department. On the Thursday evening I'd completely cleared the office. The last thing I did before leaving that night was to walk to the machine room, log in to the console of the server as root, unmount the AV volume and issue a 'dd if=/dev/zero of=/dev/whatever' on it, leaving it logged in and doing its business.
I didn't destroy the entire server (not that it would have mattered if I had to be honest), just the incriminating evidence!
This post has been deleted by its author
In the early 1980s, I was on contract for a Fortune 500 company. My manager was notoriously, almost pathologically miserly. So while other teams had IBM ATs (80286 machines with 20MB hard disks) or at worst IBM XTs (8088 PCs with 10MB hard disks), our team had IBM PCs (8088 machines with no hard disk, only floppy drives), in order to save money.
Floppy disks cost about $3 each, or $25 for a box of 10. That was if you got the good ones (Dysan or Maxell), and of course, we didn't. Our manager got something like 10 boxes of BASF for $89. As you can imagine, these disks were cheaper for a reason, and data loss was very high.
Being the paranoid sort, I kept redundant backups. My source and tools grew to about 8 diskettes, so I just allocated one box to the project. One at home, and one at work, plus the original source meant I was using 3 boxes.
When my contract ended, in my exit interview, I turned over all 3 boxes. My manager very angrily said "so you're the one who's been hogging all the disks". He summoned another team member, and handed him the two boxes of backups, and ordered him to reformat them and put them in the general diskette pool, "where they should have been in the first place".
I left the company, and life went on.
A few months later, I got a call from them. The manager had gone to another office to install the component that I'd been working on. Rather than "waste" two more floppies and make copies of the executables to install, he took the entire box of disks, including the source and tools, with him. So when he slipped on the ice getting out of the streetcar, dropped the box of disks, and they were promptly run over by a car, that meant everything I'd worked on for them for over a year was lost. Source, executable, tools, documentation, everything.
Did I... by chance... happen to have any copies, even partial ones, at home?
I pointed out that would have been a violation of NDA (it would have), so no, I didn't.
Fortunately, for me, if not the company, I still had the copy of my exit interview, where the manager had put in writing the fact that I'd been "wasting" valuable floppies by "hoarding" floppy disks from the rest of the team. So if they wanted to claim that I'd been the one at fault for the loss, I had proof that I'd provided complete source when I left, and they'd accepted it.
The person one the one sounded like I wasn't the first person to tell him that, and this wasn't the first time this had happened.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
