Let's play immutable infrastructure! A game where 'crash and burn' works both ways If you’ve ever had the misfortune to work as a systems administrator (and it doesn’t matter if it’s a Windows or Linux shop) you’ll know the feeling of logging on on Monday morning, checking a few log files and noticing something’s not quite right. It might be file systems filling up, a spam attack has filled the log …
Another depressing article about the commoditization of our career field. Now we can follow along with an article regarding vertical market cloud shops vaporizing all of the small company IT jobs and our rush to the unemployment line will be complete.
Wait, maybe I can make iPhone apps, I've always been told that I coded like I was using crayons, how much harder can coding with Lego's be?
This post has been deleted by its author
And that's it. Making them immutable is not relevant, the relevant keyword here is "repeatable"
"there’s a OS fix to counter the attack, in the old days your admin team would deploy the fix across the server range, embarrassment avoided. Now you need to go to the test environment, test the fix, and then redeploy. Time to fix has stretched"
That would equate "old days" with "throw in some patch and see if it fixes things". Old and new days should be the same: you make your change in some environment that is not live , test and then deploy everywhere. The only difference is that when everything is scripted you no longer depend on someone clicking a few hundred times in the right place or typing the right commands many times.
It’s worth bearing in mind that the immutable dream isn’t all there yet, companies have been working with Docker, AWS, Azure to make it happen, but there is no simple and cheap off-the-shelf solution just yet
Really? Try Solaris "Immutable Zones" (zones are like Docker containers, but were invented 10+ years ago)
"Immutable Zones are solaris zones with read-only roots. Both global and non-global zones can be Immutable Zones.". Been there since at least Solaris 11.1, out a good few years now.
There is a world beyond Linux and Windows, and it's been doing stuff for a long time.
"The last thing your script does is turn off the SSH port (or whatever method your OS uses to let you logon) so you can’t log-on to the machine anymore."
There goes any chance of performing diagnostics on that machine then, or presumably copying files off the machine to read logs or extract core dumps etc. Or if you can copy files back and forth the machine is no longer immutable.
Perhaps sysadmins should be employed who can follow procedures and can be trusted to only make manual changes on machines when they have to?
Though I'm assuming that if you shut off SSH you will still have console access, so the idea of shutting off SSH is to make it inconvenient for dumb sysadmins to login and change things willy nilly. Obviously you must have some form of access to do diagnostics, because monitoring doesn't always provide enough info to know what bit of hardware is going bad. I wouldn't go so far as shutting off SSH myself, but the idea of cloning servers instead of installing them all 'properly' is something I've long believed in.
Back in the late 90s I built a standard install for my 100+ HP-UX workstations. HP-UX allowed building a single kernel that supported all the different hardware types (this was before the 64 bit CPUs came along) by including the necessary drivers in the kernel's system file, and resource allocations could be done by percentages to account for different amounts of RAM.
I put that common kernel and the rest of the OS into a giant tar file that HP's installation software would let me deploy in lieu of their standard install process, so I could re-image a workstation in about 10 minutes. I had written a few custom scripts that ran on the first boot and 'personalized' the workstation with its own name, IP address, etc. based on its MAC address. If there were any problems with one of the workstations the first step (after checking dmesg and other diag logs) was to re-image the workstation and see if the problem went away. If it didn't, we could rule out software as the cause for something that was only happening on one workstation since we knew the software was identical on all.
The workstations were never patched or upgraded, instead the gold image was altered and redeployed from scratch on every workstation, which I could do in an evening from home doing a dozen or so at a time to keep from overloading the server/network.
This strikes me as a really bad way to do things. Every little problem the server encounters now, instead of being a basic troubleshooting problem, is an OS rebuild. "MariaDB went down? No, we don't check logs anymore. Just rebuild the server from the script." I could see it making sense for the big boys who have tens of thousands of servers to deal with, but I can't see it ever making sense for the rest of us no matter how good the infrastructure for doing it gets.
"This strikes me as a really bad way to do things. Every little problem the server encounters now, instead of being a basic troubleshooting problem, is an OS rebuild. "MariaDB went down? No, we don't check logs anymore. Just rebuild the server from the script.""
It is a really bad way to do things. It presupposes that the machine's state is fully defined by your configuration [code]. In the real world that simply isn't possible, typically servers have an assortment of firmware, some of which can be updated from the host OS, some which can't, and of course malware has already been caught hiding in those dark corners.
Rebuilding machines with broken firmware is going to leave them just as broken.
As for "immutable", not even an old school drive with a physical switch that isolates the write head offers immutable config... Bits flip by themselves sometimes. :)
That said having a repeatable build is great.
Use Salt/Puppet/etc to build and manage your OS image.
This includes patching, logging should be to central loghost anyway
If the host become badly borked due to a cockup or intrusion; burn it down & get the config management to rebuilt it for you.
I would advocate for burning down & rebuilding dev/test/qa systems on a semi-regular basis
Helps keep the cruft levels down
simples
> It’s worth bearing in mind that the immutable dream isn’t all there yet, companies have been working with Docker, AWS, Azure to make it happen, but there is no simple and cheap off-the-shelf solution just yet - look at how much code Netflix has built (and open sourced) to make their platform work.
Boxfuse (https://boxfuse.com) offers an immutable infrastructure solution for JVM apps on AWS. With literally one command you can create new AMIs for your app and perform zero-downtime blue/green deployments behind an elastic IP or an ELB. You can then combine this with auto-scaling to have a truly self-healing system. And you can get started for free with zero engineering effort required.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
