Posts by l8gravely 188 publicly visible posts • joined 11 Jun 2012
I just shutdown a VM with 1800+ days of uptime. And yes, it was up way too long and should have been patched, etc. But the end user customer doesn't like paying for support, doesn't like reboots because they tend to fire/let go/reassign the contractors/outside vendors who built the damn app, etc.
Of course the same idiots don't mind testing UPS to generator failover during the day without any notice....
Since Chase had a consistent monthly spend of $1-2k, he should have just bought a couple of beefy servers and put them into a colo. Then suddenly you have saved a bunch of money over time. Now if this $40k is a large part of the sponsors monthly cost, then sure, maybe they want to make a statement to AWS about their shitty controls. And moving that spend to somehwere else might be a good idea.
But honestly, runnign your own infrastructure, when even a basic 2 cpu and 8gb of RAM VM can run you $120/month, it quickly makes sense for consistent loads to move onto your own hardware in alot of cases. But hey, the PHBs all belive that purple money (opex) is free, and that capex (red money) costs way too much, so they go with the more expensive option. Perverse incentives.
Funnily enough I just had a Solaris 5.9 box lose a disk, but the onsite tech pulled the working disk by mistake. Oops! I had to get him to move it to another system entirely so I could fsck the damn thing and removed the broken mddev DB entries before it would reboot again. Once it did, all was well. Just took several days since he's only on site sometimes and I'm not a priority all the time. Oh well...
Wow, there's a name I haven't heard in a while, Acopia. We tried out their NFS appliance and promptly broke it with the sheer volume of files and directories and data we had in our environment. It was a really need idea to Virtualize NFS volumes and let you migrate data transparently. But it just didn't scale well. Got to visit them at their office in the old Wang buildings, on the left side of the Pond.
Man, now I'm tempted to try and get a copy of 'MechForce' running, which was THE game for me on my Amiga 1000 or 2500 back when I had them. I loved that game, which was freeware and let you design your own Mechs and then fight them on a hex grid battlefield. If I had free time and ability to focus, I'd try to recreate it natively. There looks to be an alpha implementation in Java, which means I won't be able to help... https://sourceforge.net/projects/mechfight/
Ah... memories!
I actually view the including of RAID into the filesystem to be a mis-feature. Just look at all the problems btrfs has had with their RAID5 implementation. And ZFS has had major problems with expanding or growing (or shrinking!) zpools much less filesystems due to design issues. Use MD for mirroring/RAID, LVM for logical volumes and filesystems on toip . Three layers, it's not that complicated and it gives you more flexiblity and control.
I've met Ted and even had dinner with Linus many many many moons ago as part of a group. Neither will remember me. But both were well spoken, able to take criticism, and still be graceful. Ted moreso than Linus as times. Kent is another talented programmer, he really is. But he's a total tool when you make a comment he views as against him or his code. Once he's decided the right way, it's his way come hell or high water.
Which isn't the way to collaborate with other people well.
And honestly, I've been watching the various mailing lists and Kent is getting better... but I think he burned one too many bridges and Linus is either going to make him sit out a cycle and then merge it for 6.18 and hopefully Kent will learn the lesson. Or his will just 'git rm' the entire sub-tree and purge it all for the next release. I hope it's the former, and I hope that Kent realizes this and gets his ducks in a row and all his patches and testing done so that when the NEXT merge window opens up, he can big there with a well explained patch set that's been in linux-next tree and doesn't do stupid crap like dropping major patches into release 4 or 5 of the window.
Kent's reasoning was that "I needed to post these patches because people had data corruption and this was the only way to get them working again!" which is BS. Anyone running 100tb of data on an experimental filesystem deserve to lose data and should expect to lose data. And they can god well compile their own kernel with out of tree patches to get their data back. At that point to send in a patch to disable the bad feature. Thne you can send in a second proposed patch that could either go in now to fix it, or be pushed out to the next release as a real fix.
It's about patience and an understanding of the process.
I've had my own runins with Kent. I absolutely an amazed at his coding ability and focus. But he once asked for some comments on the useabily of bcachefs tools and such and I gave him my opinion, carefully noting that I was not a programmer and this is from my IT hat as a Unix SysAdmin. Boy did I get flamed for providing useless feedback that wasn't all praise and hallegleuyahs. (not going to bother spell checking that!) and so I'm a bit off on bcachefs at this time.
But I do like the ideas it has, and the scalability and no-fsck really needed, etc. It's not as fast as hoped, but honestly people have some many special cases in filesystems for directio, threading, and other hacky ways to get more performance for their special sauce which can't be changed for some reason, that I know writing a filesystem is a chore. And with all the new features people want/need, it's hard to be performant.
So yes, Kent is a prickly pear who can't/won't/hasn't yet learned how to deal with people and pushback in a graceful manner. This is his problem. It might be our loss... but it's his problem to solve.
And don't you love how Win10 let you put the status bar vertically on the side of the screen, not just at the top or bottom? Now notice how Win11 has removed that wonderfully useful feature for some insane reason.... it's always a step backwards.
Once you re-configure screen to use 'Ctrl-t' instead of Ctrl-a as the screen attention prefix, it works a treat when running mulitple sessions.
I even have my screen setup to echo the screen number and hostname into the title bar of the Xterm. So I can flip between remote connections over a single putty session and manage multiple remote systems. I'm on the east coast and most of my work is on the west coast, so having ssh/putty/screen is a god-send for me.
I've tried tmux before, and while it's nice and works well, my fingers are programmed with screen shortcuts.
And this brings up what I think would be an awesome project. Basically a keybindings translator which lets your fingers do things how they like (for me, emacs movement bindings are etched in my brain, and since my terminal CLI history also supports those exact same key strokes... that's what I use everywhere).
So basically, I setup a screen+emacs+whatever bindings, and then no matter what editor, or screen management tool I'm using, they keybindings are the same. I'm also a strange duck who still reads email using 'viewmail' inside emacs. But I use mutt for more heavily trafficed mailing lists, but replying only happens in viewmail because my brain is wired for those key bindings. I need to seriously investigate changing all of mutt's bindings to match what I do for 95% of my email reading and composing.
And this is why people get so upset when vendors re-tool their GUIs and interfaces. They've invested so much muscle/visual/expectation memory into common tasks, that when things are changed (for no good reason they can see) they go ape!
And don't get me started on all these 20 year olds with transparent terminal windows with busy graphics bleeding through! How they can read that, I'll never know. But they will regret it down the line when they have to start wearing cheaters just to read the screen. Just you wait.... just you wait....
Sigh... I've just run into this, again, for some old Brocade FibreChannel switches we still use. And for a 20 year old application written in house using Oracle Forms 6 running on Solaris 5.9 on a V120 that's 20 years old. Sigh... luckily I can just quietly archive that system and application and it's data. It will never ride again.
It's also because people and companies want "the next new thing!" and "you have to upgrade because of our new X, Y or Z", when the old version works just fine. If it ain't broke, don't fix it gets more and more useful asI get older.
This past weekend I tried getting Photospring inside Docker working better (it's already pretty nice for what it does!) so I could upload photos more easily from my iPhone so I could finally sort and organize and do some facial recognition, etc. Nope, not a fucking hope to get it working. But the system I host it all on is a Debian system upgraded since version 5 (I think) piece meal over the years all the way to Debian 12. It works. I runs some reasonably bleeding edge linux kernels I self compile. It works! Until it doesn't...
I'm thinking I need to go back to basics, where I have a core systems that just provides KVM, NFS, Docker, etc. But no real user logins except for management. Then I just have various VMs which mount home directories from the core server part. I can upgrade, change, update, play with VMs to my hearts content. But the core services are just there and working and stable.
Ah... it's like a good workbench. It has to be durable, adaptable, flexible but above all solid and unmoving when you're bashing away on top of it on a project. So much so that you just forget about it mostly. Until you need to do an upgrade.
So I want a really good workbench (Debian? FreeBSD? ProxMox? ) whcih can provide good services, but not much more. The rest is all for the VMs ontop of the abstraction to play with.
I stil have, but don't use, a Northgate OmniKey Ultra T keywhich which I used to love. See https://en.wikipedia.org/wiki/Northgate_Computer_Systems for a picture. But having a number pad which I don't use at all very much was just bad for my right wrist carpal tunnel, so I've moved to a tenkeyless CODE from www.wasdkeyboards.com, but it looks like they've died and gone the way of the dodo. Which is sad.
I need to start looking for another backlit board with really really really solid keycaps, doubleshot that let the light through. Suggestions?
So I don't like ergonomic keyboards because they're strange and my muscle memory isn't helpful. It's even worse when I plunk down at some user's desktop and they have capslock still on the home row. Sigh... the one true emacs way is to have Ctrl be way out left! But if you do have carpal tunnel syndrome, then maybe you're sitting wrong? Or you've got the mouse way too far out to the side? I've gone to 88 key ten-keyless keycord (CODE from wasdkeyboards.com) as my goto driver at both work and home. It's backlit, narrow so my mouse isn't way out to the side and it clicks and feels nice. It's got the Cherry MX Clear switches, so it's loud, but not nearly as bad as the blues.
What I wish I could find is a keyboard with the number pad and/or function keys on the LEFT side, but I'm a lefty who can't use the mouse with his left hand, but can't draw with my right hand to save myself at all.
I turned down a job offer in '92 because they expected me to wear sport jacket and tie, though I was assured you could take off the jacket and leave it on your chair back when you got into the office, and then put it back on when you left for the day.
The real reason I didn't accept it was that I was going to install the OS on new HP-UX systems. Not unbox, not install the application. Not rack them. Just. Install. The. OS.
Which sounded like hell to me because it came on tape to be installed once you booted the base installer from CDROM or BIOS? Can't bother to remember how the described it. But it sounded like hell on earth. I got into IT because I could get my hands into all kinds of interesting nooks and crannies.
The jacket/tie was just the icing on the cake of 'nope!'
Way back in the early 80s (late 20th century as my kids call it... sigh) my school got a new computer lab with 40 IBM PCs in 10 Severance style pods sharing a dot matrix printer. Our teacher was nice and a good history teacher, but out of his depth teaching BASIC. Three friends and I would amuse ourselves by writing completing the day's in-class assignment in as few lines as possible of BASIC, more kudos if we got it done in one line. Fun times. Usually we'd be done before he finished explaining things.
I'm very glad I dodged that bullet and have had very little IBM experience my career, mostly AIX which is NOT Unix in any way, shape or form. Just from the application interface, not the management side.
Though I do have an AS/400 sitting in a corner which needs to be run every once in a while since no one can be arsed to get the data off this system and migrated to something simpler to use and maintain. This is what happens when you let go all your application support team... you end up with horrors like this. Or the 20+ year old Java Tomcat app they're trying to replace now. Sigh...
Back in the day at my dad's office we have Zenith H-89 and Zenith Z-100 systems running CP/M with probably 4mb of RAM. 8" or 5.25" floppies. I would do a bunch of data entry into MagicWand or PeachTree DB software (or just a flat file formatted properly, I forget which but suspect the flat file now...) and then write up the marketing letter, format it and let it all rip on a Diablo 630 daisy wheel printer. Many, was that thing loud. Made the whole office grimace when I kicked it off, so they soon got one of those stands with the hatch you could lift up to get it, but otherwise left down to muffle the noise. Came with the noise absorbing foam traingles as well to hopefully cut down the noise. Those were the days...
I'm in this place too, but it's an old old app I wrote for a local library to allow searching of their collection of newspapers. Luckily it's a readonly DB updated from the master internal server and lives in the DMZ. Simple mysql, php, apache. Trivial. But it's on an old version of Debian, with an older version of PHP. And the date time library I use for dates (all before 1900...) is semi-broken and abandoned for newer versions of PHP. Sigh... And of course some of the PHP language stuff has changed as well, so I had to re-write some stuff to get around that.
So I'm not in a rush to replace it on my spare time. But I know it needs to move up to something newer. It's not sexy, but it works, I can spin up new versions and new interfaces to new a new DB of collections and the internal people know how to push updates to my server. Works a treat. Until it doesn't. But it's read only and in the DMZ, so it's not critical.
Too many systems don't have this mindset of keeping the production seperate from the reporting or query systems. Yes, I understand you want to be able to enter new orders and such into your production system and you need to do that. But for alot of things, have a query only system you can not care about because you can rebuild it quickly is "good enough" in alot of cases.
and now empty fuel and oxidizer tanks are up high. They went with two stacked tanks, instead of Blue Ghosts four tanks arranged around the center and loiwer down.
This doesn't bode well for Starship landing on the moon either.
For example, most big (and little) cranes are spec'd to be level to within 1% or their load charts drop off precipitously. Just a little off plumb and things start looking really really bad.
Way back in the midst of times when sh was all you got, along with maybe csh as possible shells, there was much rejoicing when tcsh was installed and you could edit the prompt to show various things, like the current directory you were in, along with the hostname you were currently logged into. This has saved lots and lots of time and gnashing of teeth over the years...
Now days bash is all you really need anymore, it's got all the features of tcsh and none of the downsides of actually scripting in (t)csh as well. Whew.
I ran into this recently, but in a more prosaic way. I was cooking some meet for a beef stew and the grease ended up on my glasses. Not enough to stop me from seeing stuff, but enough to stop my iPhone from being able to recognize my face. It was really wierd... and sent me down all kinds of wrong paths trying to figure out what was wrong with my phone, as opposed to me. Or more accurately, my glasses. Once I cleaned off the schmutz, all was well again.
Except when you need to do dates before 1970, say for a database of old Newspapers in a library collection that people might want to search on... then it's a pain. And when the library you use is abandonware and not exactly happy with newer versions of PHP without some hacking... then it's not fun.
Ask me how I know...
sendmail was designed to be easy for computers to parse, so of course it's painful to work in. Eric has admitted he made a poor decision there. Then he compounded it by using m4 to try and make it readable.
As for perl, you think the syntax (not semantics!) is worse than any other language? You think python with forced indentation is great? Or rust, or go or C? They all suck to some degree or other.
Heck, I wish all languages would get rid of the ternary operator!
Still using a PCengines board for my firewall, sad to see that he's shutdown the business since I really liked his hardware and the lack of a fan. But I run OPNsense instead, which I think would run on here just as well. The only gotcha I see here is the lack of a serial console, but maybe I missed that in the details.
And we're a small shop with only 5 or so ESX servers. We just want a cluster of systems so we can VMotion stuff around at will and keep the lights on. We don't; need funky storage, VxRAIL, VxWhatever. Just a management pane, VMs, and hosts with NFS datastores we can shuffle things around on live. ProxMox looks pretty good for this, if only because the price is better.
But the C-suite always wants someone to blame or get support from, and they don't trust ProxMox quite yet. But with Veeam support, it's starting to look much better.
And now that Veeam is oppering support for ProxMox natively for backups... even though people say ProxMox backup tools work great... it means one less thing to change in your infrastructure.
We've deployed some test instances of ProxMox on some ancient Cisco UCS hardware. Which works. It's different and I have to re-learn how to do somethings, and it's not quite as polished, but all I really want is NFS datastores, VMotion between nodes without having to tell my VMs they're moving (I'm looking at you Cisco for those crap ISE VMs which freak the hell out when you VMotion them!) and just work.
Don't really need much more than that. We're not that dynamic a shop, we don't need extran networking spun up, vritualized storage, or other things like that. Keep it simple, keep it reliable and let me get work done elsewhere.
I'm both in agreement with you that it should be easy these days, but that it's also terribly hard because you need to get your techs used to something new. And sometimes us techies stick with the old because it's comfortable and we know it. And we're terrified of being made obsolete by new things. I've worked for years with people who insisted (resisted?) on using the GUI to do long tasks because they didn't want to learn the CLI or scripting to help automate their lives. Because it was new or different or strange or just _harder_ than mindlessly clicking on GUI buttons.
And when you're clicking away madly, you're getting work done! And you can show people! And you can look busy!
Me, not so much when I automate crap and it just happens and I get to surf the web looking for new techniques to automate my life.
I did some headless server VMs last week without any problems. but they're just VMs, I had four identical ones to upgrade and if I screwed up I could always clone one of the others, etc.
I suspect it's more the sheer number of packages in a normal desktop rollout that caused the problems, but for targeted servers, maybe it was ok?
We had a row of VT100s hooked up to a DEC TOPS-20 when I arrived at Uni in the mid to late 80s. One of the more enterprising chaps had come up with a way to animate a choo choo train to scroll down all the terminals and then back again. Much fun to watch.
I still use it every single day, it's simply an amazing tool for what it does. Doesn't get in the way, you can manage it all from the keyboard and it makes my work be so less stressful. I've been using it for close on 30+ years now. I've looked at tmux, but the key bindings are hardcoded in my brain so I switched back. Tmux didn't give me much that I really needed.
Now to find a replacement for 'putty' which will auto-reconnect my SSH session(s) when they get disconnected. That would be really nice.
Tried mobaXterm, but it's just not my cup of tea. Expecially it's annoying re-opening the file share sidebar every time. I know I can disable it, I did for a while, but having the abiltiy to put two putty windows (100x48) side by side on my desktop is key, and mobaXterm just doesn't cut it because it takes too much screen realestate.
On the left side of the Pond, where my late departed Brit mother only drank coffee, I got introduced to Tea at a young age and have loved it ever since. So far the best of the best for day to day is "Tetley's British Blend" closely followed by "Yorkshire Gold". Guess where Mum came from! :-)
I'm a total AI skeptic, but I like this article and others like it because it gives you a foundation to explore on your own what's happening in the hype(r) world of AI these days. It's interesting, and if I was more of a graphic artist maybe I'd be more interested in making silly pictures, but I'm not. I'm also very hesitant due to the hallucination problems in my co-workers, much less AI tools doing wierd things.
But as a curious person, this is some fun stuff to play with and try to figure out how it would help me make my life better. Or at least let me learn something new and keep my brain exercised. I don't see AI taking my job away any time soon, so I'm good with this stuff.
I used to manage a pair of Netapp clusters at opposite sides of the country, and trying to keep them in sync so that they were DR sites for each other was... tough. We had a T3 at the time, which was huge for our small company. But 85+ ms of latency just killed Netapp SnapMirror copies, you could see the spikes in the classic sawtooth shape as the TCP Bandwidth Delay Product kept hitting us.
We tried using WAN accelerators like Riverbed, SilverPeak and others. We tried using 'bbcp' (great tool btw!) for single large files to fill the pipe. It all just sucked big time. It's just hard to move that much data when you had (at the time) 1tb of change in the data per-day. It just didn't work. It must be fun having google's resources to play with and to test things and just to learn how to do stuff better.
I really like how the data moving is under 10% of the code, it's always exception handling that bites you.
I'm a recovering CommVault user (not really, still have to use it) but having done multiple migrations of backup systems over the years, you *never* get rid of the old one or it's name until it's long past the "need to restore" date. In this case, going from DAT to DLT, along with new backup software would have required *zero* need to keep the backup server name the same.
In fact, I would have set them up in parallel and run them in parallel for a bit, so I can make sure things are working. So there's some serious incompetence here in terms of management, along with the techie.
Especially since CommVault is it's own special brand of special where you probably need a windows server, plus unix media agents to actually pull the data off things.
So I call BS on this story...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
