Upgrading what might be the world's oldest running Linux install

There are some complexities involved in upgrading what the Reg FOSS desk suspects may be the world's oldest running Linux installation: an OS install dating back to 1993. The machine called chiark.greenend.org.uk appears to be a relatively ordinary webserver, hosting a bunch of home pages, a few mailing lists, usenet groups …

[steelpillow]

Well well well

"He resigned from the Debian Technical Committee in 2014, after his proposal that Debian packages should remain init-system agnostic was defeated."

So the next upgrade ought really to move across to Devuan. That would be an interesting exercise indeed, at many levels.

[Doctor Syntax]

Re: Well well well

The upgrade to Jessie must have been interesting, given that Jessie was the first with systemd

[7teven 4ect]

Re: Well well well

Devuan is anti-systemd to an almost militant degree, perhaps not?

[Wellyboot]

Chiark is well on its way to being a significant cultural artifact

Very good, carry on.

[tip pc]

Triggers Broom

What's remarkable about Chiark is that it was originally installed with Debian Linux 0.93R5 in 1993, and the same installation of the OS is now running Debian 11 "Bullseye", the x86-64 version, freshly upgraded from an x86-32 installation of Debian 8 "Jessie".

So it started life on 0.93 then got upgraded and virtualised (I assume over many iterations) to version 8 from where it has been upgraded to version 11.

I had initially thought it was a direct impressive upgrade from 0.93 to 11 on the original tin!!

Until a few months back I had preference and other system files on my Mac from my original lcii from 1992, I clearly no longer have need for Macdraw, Claris works, HyperCard or Chuck Yeager's Air Combat files!

[Doctor Syntax]

Re: Triggers Broom

The linked account (well worth the read, BTW) doesn't suggest that it's virtualised but it does say "Obviously it’s had several new hardware platforms too." so, yes, definitely Trigger's broom.

[Liam Proven]

Re: Triggers Broom

[Author here]

Yes, this is, AFAICT, on the "original tin" and *not* virtualised.

> I had initially thought it was a direct impressive upgrade from 0.93 to 11 on the original tin!!

Your original thought was correct. It is not _direct_ from 0.93 to 11; it's been through several versions since, but not all of them. This, as I said, was from v8 to v11.

[Doctor Syntax]

Re: Triggers Broom

On tin but clearly not original tin.

One thing that surprised me - and himself - in his account was that he spent 5 hours remotely rebuilding his rescue system, an ancient 2Gb USB device. I think I'd have just built a new one, posted it off to the colo and asked them to plug it in. (OTOH would I be happy with a colo that just plugged in a USB drive they were sent?)

[Lennart Sorensen]

Re: Triggers Broom

Yeah no way is it using the same motherboard/cpu/ram that it was originally installed with. That would not be 64 bit for sure. Nor would it have had USB ports at all.

[Dave559]

Re: Triggers Broom

The whole exercise sounds way too much like trying to rebuild an aeroplane while in flight, or trying to solve one of those interlocking segment wooden cube puzzles, but whatever floats your boat!

But the question is, can it really count as "the same install" if (on at least one occasion) the data/filesystems have had to be copied somewhere else (and then back again) while the actual hardware was upgraded? I suppose the 'consciousness' is the same, even if the 'body' has been replaced at times, i guess?

[jake]

Re: Triggers Broom

"But the question is, can it really count as "the same install""

Not in my mind ... Even with completely redundant fall-over boxen,"The System" as a whole might remain up according to onlookers, but if the original hardware is going up and down like a yo-yo for kernel upgrades and the like, the install has clearly changed.

[Lennart Sorensen]

Re: Triggers Broom

Well the Debian installer has only been used once for the installation. The root user has only been created once in the passwd file, although clearly with the password updated over time (I would hope). So yeah I think it can be considered the same install. It's not the same physical machine anymore though. I have a 486 around with Debian installed on it and upgraded for years. It was installed in around 1999 or 2000, as far as I remember using potato that was under developement because slink just didn't want to install. Haven't used the machine for about 7 or 8 years now because the poor HD sounds terrible (ball bearings don't last forever) and I don't believe Debian supports running on a 486 anymore since 2015. I think I should put a compact flash IDE disk in it and use it for DOS gaming instead. It used to be good at that before it became a dedicated linux server for many years.

[oiseau]

Re: Triggers Broom

... Debian installer has only been used once for the installation.

... root user has only been created once ...

... it can be considered the same install.

Indeed ...

I was about to post more or less the same point.

He just moved the installation to a different container so to speak.

+1

O.

[Dave559]

Re: Triggers Broom

Yehmmm, OK, fair point, that's a reasonable argument: only ever installed once, everything since was an upgrade, albeit with greater or lesser amounts of advanced wizardry involved… :-)

[Anonymous Coward]

Re: Triggers Broom

Quite. I've migrated (Linux and FreeBSD in my case) sysdisks from old 32bit hardware to newer, either copying the installed OS or physically moving the media, sometimes needing to change a network interface name or something due to different NICs, and carried on upgrading the same OS install as desired until the next iteration.

Usually when the time came for 64bit hardware I did a fresh install, but it wouldn't have been strictly necessary in all cases.

But I came nowhere near 30 years continuous upgrades in any of my efforts, so well done there.

[jake]

Re: Triggers Broom

Digging through my piling system, I've discovered that I still have a 386SX16 from the Linux 0.96 days (middle of '92) that was updated piecemeal into the end of Linux 2.6.32.71 ... it still happily boots and will do work, for rather small values of "work" in its whopping 8 megs of RAM.

While I probably can't prove it (I might have backups from back then), the notes I made upgrading the box over the years strongly suggest that the password file is the original, and I have only created a root account once. Of course that root account has been transferred to a new boot disk umpteen times ... In your mind this is the same installation?

In my mind it is not. Not by any stretch of the imagination.

Now, the VAX that I keep running for a small community of hard-core MUDders, on the other hand ...

[herman]

Re: Triggers Broom

My great grandfather’s axe: The handle has been replaced three times and the blade once.

[jake]

Re: Triggers Broom

My family has my great-grandfather's kindling hatchet sitting next to the wood stove. It's on its sixth head and 13th handle. My dad started calling it Theseus's Axe before I was born.

[Mr Flibble]

Re: Triggers Broom

Copy for backup or to replacement storage devices, yes.

Copy when you can attach the existing storage to the new board? Pull the other one…

[jake]

Re: Triggers Broom

"On tin but clearly not original tin."

Indeed. Unless I'm misreading "Obviously it’s had several new hardware platforms too.".

Basically a Ship of Theseus[0] ... Only the owner and registration are the same, the original hardware hasn't existed in this context in (probably) decades.

[0] Or Arkwright's Brush, if you're of an age ... Trigger's Broom is a johnny-come-lately.

[Doctor Syntax]

Re: Triggers Broom

Arkwright? Trigger? Sounds more like Jason than Theseus.

[Nick Ryan]

Re: Triggers Broom

Never heard of Arkwright's Brush, and a quick google search isn't returning much of obvious relevance.

[ChrisC]

Re: Triggers Broom

From https://tvtropes.org/pmwiki/pmwiki.php/Series/OpenAllHours

Granville: We need a new brush.

Arkwright: Nonsense! That's a marvelous old brush, that! I've had that for fourteen years. It's only had two new heads and three new handles.

[jake]

Re: Triggers Broom

"Never heard of Arkwright's Brush"

https://www.youtube.com/watch?t=29&v=8XOKopHtoJQ

[Nick Ryan]

Re: Triggers Broom

Aha, that Arkwright! I was thinking you were referring to something much earlier than Open All Hours! :)

[Ian Johnston]

Re: Triggers Broom

Where did he get a 64 bit x86 processor in 1993?

[Ozan]

Re: Triggers Broom

I did something similar with my old taptop a Dell Insparion: I continuously updated Slackware on it for round 10 years till it break away. Not the OS but the machine. Plastic eventually gave away.

[Zolko]

Re: Triggers Broom

I clearly no longer have need for [...] Chuck Yeager's Air Combat files

WHAT !?!? Everybody needs ALWAYS to have Chuck Yeager's Air Combat around. As a side-note, I didn't know there was a Mac version, I've been running it on DOS. And now I managed to run it on emulator

[Missing Semicolon]

LVM over RAID over LVM

I'd love to know the reasoning behind such a structure. Is it really more reliable? More fixable? It just sounds.... precarious.

[Lennart Sorensen]

Re: LVM over RAID over LVM

I suspect LVM at the bottom to allow combining different disk types and sizes into pools, then raid on top of those pools to redundancy, and then LVM on top for flexibility in allocation of the disk space.

I have never done that, by just throwing money at the raid layer to keep things identical and hence simpler, but I can see how lvm would allow doing it cheaper and with less initial planning required for handling future upgrades.

[TJ1]

Re: LVM over RAID over LVM

If this OS install has grown over time with in-place incremental upgrades it makes a lot of sense. Logical Volume Management (LVM) has gained features over the years that probably were not available when LVM was first adopted.

Physical > LVM > RAID >LVM is probably due to LVM not supporting RAID modes originally so likely it is Multiple Device (MD) RAID - probably RAID-1 mirror.

My guess would be originally the install was on a single HDD. As more storage is required it is far easier to manage it flexibly via OS (e.g. LVM) services rather than hardware RAID. So, add in more physical HDD/SSD, "pvcreate ; vgextend" and then "lvextend" for those volumes needing more space.

So over time, without any major OS re-installation, using several physical HDD/SSDs, the host has a RAID mirror with OS and data volumes on top.

Nowadays LVM supports RAID modes natively (using the kernel Device Mapper (DM) MD RAID functionality under the hood) so the additional layer could be removed whilst the OS is operating without too much trouble (I've done this on multiple systems over the years). This is one of the delights of using LVM - being able to re-shape storage architecture quite fundamentally whilst the system is live (including more exotic options like adding iSCSI block devices as LVM PVs to create remote mirrors).

I've also done a similar live migration from 32-bit to 64-bit in-place (original 2007 install, host still in operation). Once the kernel is switched to 64-bit it supports both 32-bit and 64-bit user-space. At that point you can create a 64-bit chroot install with all the required packages followed by copying over configuration files package by package and switching the running service from the 32-bit to 64-bit in the chroot.

Eventually you've a 64-bit kernel with a base 32-bit core running all 64-bit services. At that point the boot configuration can be pointed at the 64-bit root file-system (a Logical Volume) and the system rebooted.

When doing this it helps to actually upgrade the 32-bit packages to the target OS version first so that the package upgrade scripts handle most of the per-package configuration file changes for you. If skipping several OS releases it's unlikely we could rely on that to correctly handle all changes and would have to manually check and review each package configuration. Once that's done the switch from 32-bit to 64-bit should be straight-forward.

[Doctor Syntax]

This must surely be one of the paramount examples of a "server as a pet", as contrasted with the devops style of systems management: "servers as cattle".

Them's not pets, them's work hoses.

[Doctor Syntax]

Dammit - horses!

[Anonymous Coward]

Trusty and devious mule

"One of the most loyal of companions, hardy and shure of foot, and a bit stubborn and set it it's ways."

Sounds about right for the home server of a long running SSH client.

[Anonymous Coward]

Re: Trusty and devious mule

And as such one of the most important systems on the Internet.

Lets face it, without PuTTY there really wouldn't be any point to Windows at all

[Anonymous Coward]

Re: Trusty and devious mule

Yack! I'd prefer to install a base Cygwin and use its ssh than use PuTTY.

[Vincent Ballard]

Re: Trusty and devious mule

They both have their uses. In particular, the ability to use PuTTY without installing was great when I was travelling for several months and had to ssh back to my home machine from random web cafés to check my mail.

(Obviously there was some risk involved, but I assessed that the risk of web café operators being savvy enough to not only log the keyboard but also sniff the keyfile on my USB stick or launch a hacked PuTTY when I tried to run the one on my USB stick was low enough to be acceptable).

[Anonymous Coward]

Re: Trusty and devious mule

> log the keyboard but also sniff the keyfile on my USB stick or launch a hacked PuTTY

Now you made me wonder whether you can use TOTP with SSH.

Off to google and that's my productivity done for the rest of the afternoon.

[druck]

Notes from the master

I've upgraded the OS which ran on the original 256MB Raspberry Pi all the way up to Bullseye now running on a Pi 4B, but I now want to take it to 64 bit without reinstalling. There is some stuff out there about how to migrate x86 Debian from 32 to 64 bit, but some notes from the master would be much appreciated.

[Version 1.0]

Re: Notes from the master

The "world's oldest running Linux installation" was an upgrade too, I remember seeing the original request for people to start writing the first code and that was several years earlier.

[Gene Cash]

Re: Notes from the master

This is how I did it in July 2013 to run 64-bit KSP

http://www.ewan.cc/?q=node/90

I made backups first, of course, but it Just Worked. I was shocked.

Edit: I'm also a little shocked to see a page from 9 years ago is still there under the same URL...

[oiseau]

... it is complex, difficult edge cases such as this that expose some of the most difficult issues in software design and implementation.

Exactly ...

No two ways about it.

See if you can pull off 1% of all that on a MS crapsule.

O.

[oiseau]

Genius and principled

Hello:

... resigned from the Debian Technical Committee in 2014, after his proposal that Debian packages should remain init-system agnostic was defeated.

Defeated indeed ...

In what I have read was seen by a great many as a bogus and irregular voting process.

The man's not only a genius, he's also principled.

Not like the rest, who had no issues with going behind Microsoft's newest hire.

Kudos + a case of fine ale for Ian Jackson. ----->

O.

[Nate Amsden]

mailman upgrade

There doesn't appear to be a way to upgrade mailman, as far as I could tell there was no real migration path from mailman 2(python 2) to mailman 3(python 3). Mailman 3 looked to be an insanely complex beast(compared to 2). I posted a few times to the mailman mailing list looking for advice and saw others in far more serious situations than my personal mail server that sends maybe 50-100 mailman messages a year. One person said they had been down for weeks trying to get the new mailman working with big mailing lists that were offline, and lots of big ugly errors trying to get mailman 3 working.

Mailman 3 looked entirely too complex and not ready for prime time in my opinion(exception being perhaps super large lists with dedicated staff to manage it). So my solution for the meantime was to build a dedicated VM with an older Devuan release with mailman 2 and just have that do the mailman processing. My main system can remain whatever version I want and then I just have postfix route the mailing lists to that dedicated system for processing. Works fine.

I feel I have a good knack(?) in being able to quickly determine if something is more complex than it needs to be and so I usually try to avoid such solutions where possible.

My oldest personal system (upgraded from Debian to Devuan since) looks to date back to what looks to be about 2010, about when I switched to ESXi for my personal public servers. I think all of my servers have work have been rebuilt at least once over the years(first systems fired up early 2012). Running Ubuntu at work and when 20.04 came out I radically restructured how the VMs were configured and rebuilt everything that could run on 20.04(some things still stuck on 16.04).

Was running Debian since 1998 with the release of 2.0. Switched to Devuan in the past couple of years.

I estimate I spent over 100 hours of work dealing with systemd related issues at work going from Ubuntu 12.04 to 16.04 several years ago but for the most part got it all figured out there.

[A Non e-mouse]

Re: mailman upgrade

I think a lot of Mailman 2 users are moving to Sympa for similar reasons.

[Anonymous Coward]

"We found it notable that it was all done remotely"

But did he use PuTTY?

[Ian 55]

Re: "We found it notable that it was all done remotely"

I doubt Ian runs Windows on anything that needs to work - he just hosts PUTTY for Simon, along with Simon's games programs and assorted other stuff for assorted other people.

[pip25]

I don't believe in the Windows reinstall

I had to do it exactly once, when a lovely uninstall program wiped out my Windows directory (or at least the parts it could). Luckily, I was able to preserve most installation and registry data even then.

That was in the Windows 98 years. Since then, my desktop PC has always been upgraded to the next OS version, including a switch to the 64 bit architecture thanks to some brilliant third-party software.

Needless to say, this article truly warms my heart. :)

[phuzz]

Re: I don't believe in the Windows reinstall

My Windows install has been through every version from Vista onwards, up until a motherboard failure prompted a clean install.

[Lorribot]

No one else concerned that one of the most widely used SSH tools was hosted on such an elderly beast of a system? One would worry about the security of such an installation.

It is also a good example of just because you can doesn't mean you should.

I appreciate some people get attached to things like vintage cars, but they generally don't use them as daily drivers even if they have been patched and modified, migrating to some new hardware or virtual environment and fresh install of an up to date OS would seem the most prudent way forward.

[Throatwarbler Mangrove]

Well . . .

"Jackson is highly qualified to perform such a complex upgrade: as well as being a former Debian project lead, he also wrote Debian's underlying package-management tool dpkg. Jackson stewarded the release of Debian 2.0, the first multi-architecture version of the distribution, and the first to use libc6."

Perhaps you should weigh in with your qualifications and accomplishments before criticizing.

[-bat.]

The hardware does appear to have been migrated, but a fresh OS install is unnecessary when you can in-place upgrade the existing one. that way you dont have to reconfigure everything. I have a similar box, albeit nowhere near as as old, installed with FreeBSD back in about 2000-odd, and has been upgraded repeatedly ever since. Including a move to 64 bit in-place (as 64 bit kernels run 32 bit binaries. It seemed like the easiest option, and still does. What does a fresh install give me apart from a lot of work to do ?

[Cederic]

No, not at all. Should I be?

It was secure when first put live, it's been continually patched and upgraded since, so it's no less secure now (and probably more so).

[matjaggard]

Sounds basic

I'm so confused. It runs a web server, some mail groups and usenet? Surely that's perfect for running in a few Docker or equivalent containers on some unnamed cattle server or two?

[Doctor Syntax]

Re: Sounds basic

It might be perfect for Docker but as it runs perfectly well without, there's no indication that Docker would be perfect for it.

[Anonymous Coward]

Re: Sounds basic

> Surely that's perfect for running in a few Docker or equivalent containers on some unnamed cattle server or two?

I wouldn't do it for a variety of reasons (especially if my university has its own data centre) but yes, a priori it's a perfectly viable and legitimate approach.

[heyrick]

Whoa!

I was there just yesterday picking up some docs from Theo's bit.

Small world.

[John Brown (no body)]

My Amiga...

...is still running the original 3.0 Kickstart/Workbench. Well, it might if I switched it on :-) I'm not sure I should do that, there info out there about leaking capacitors on old kit and needing to replace them all before attempting to power up old systems)

A few years back I pulled the HDD out and dumped it to a file on my PC and now the same image file is running under FS-UAE. Now, having said that, I'm not sure how old the disk image is. I remember buying my first HDD for it. a 3.5" device which meant removing the HDD cradle designed for a far to expensive at the time 2.5" HDD and needing a different cable and cardboard packing to hold it in place. But when I dug it out from the loft and opened it up I found a 2.5" drive in there. Still jammed in with cardboard since the proper cradle was long gone. But I have no memories of ever getting that 2.5" HDD. Maybe it shrunk over time?

[joehonton]

Voyager 1 and 2 were launched in 1977 and are still in operation 45 years later.

They each have two redundant sets of three computers for operating the measurement devices, communications controllers, and positioning systems.

Their software was upgraded, when they were beyond Saturn, to perform a type of image compression that hadn't existed at launch time. After passing Saturn, Voyager II was upgraded to send it on a previously unscheduled mission to Uranus and Neptune. In 2019 their power management software was upgraded to forestall the inevitable.

Today, they continue to monitor conditions in the outer Solar System, although it is predicted that all scientific instruments will eventually go offline around 2025, and all communication will be lost around 2032, when the juice required to run things finally dries up.

Needless to say, all 12 computers are running headless, and asking a COLO operator to stick a thumb drive in is simply not going to happen.

[jake]

Re: Voyager 1 and 2 were launched in 1977 and are still in operation 45 years later.

If you want to go that route, my IBM 1401 still boots and runs exactly as it did in 1963. Yes, it is still used ... I use it to pull old format media[0] into more modern storage.

Mind you, there's no TCP/IP stack for it ... but then there is no real need for one.

[0] A couple different kinds of mag tape, punch cards and tape, and occasionally a very old HDD.

[werdsmith]

I once encountered a Netware 4 server that had been running since the mid 1990s on the same Compaq server. It had virtually no downtime either, apart from a PSU replacement when the fan bearing started to rattle.

That was about 4 years ago. Nobody dared switch it off, nobody knew how to support it, it just kept going doing its little job. I will enquire to see if it’s still running,

[Hubert Thrunge Jr.]

So not running on a 386 then

And there was me hoping that this whole thing was still running on a 386/25 based box with a 20Mb HDD and 5-1/4" drives.

Silly me.

As the bloke at the Council IT department said "Look after your server"...

[Christoph]

" If you know of anything older, we'd love to hear about it."

Voyager 1 and 2

[phuzz]

Minesweeper

Chiark also hosts a good Minesweeper clone, which only presents you with guaranteed solvable maps.

[Anonymous Coward]

Re: Minesweeper

I've had Simon Tatham's Puzzles running on every phone/tablet I've owned (reqd OS: Android 4.4 up)

[Lee D]

Re: Minesweeper

Spoken to the guy, he's a nice bloke, works for ARM.

[nautica]

same ol' axe...

"...relatively modern hardware, running a nearly 30-year-old OS install that's now pretty much current...."

See this axe? I've had it for 30 years. Only changed the handle three times, and the head once, but it's still the same ol' axe...

[Lee D]

'Tain't broke, don't fix it.

More surprising is that chiark as a naming still goes to just one server, rather than some kind of DNS round-robin even if it incorporates the earlier names (referring directly to a server is generally a bad idea if that server then dies...)

But, again, I suppose that it's working so why play with it?

I always have a dedicated server to do my stuff, only ever cheap junk from OVH or Kimsufi - lets you have an outside server to test things from, lets you provide services acts as a proxy in many regards (my websites are mostly reverse-proxied from that well-known, static IP, always-on server back to an (isolated) internal network over VPN so that you don't need hole punching and port-forwarding and TLS/SSL and other things, for instance Let'sEncrypt on my internal servers would be a nightmare to configure, but if I have the dedicated server LetsEncrypt and reverse proxy, then there's always a server online to handle all the renewals for everything, but the back-end is also encrypted and secure without having to pass ACME tests directly), and just keeps going. They usually get to about 5 or 6 years old before there's some cheaper deal involved and I migrate and upgrade across to the new server, but otherwise I just leave them running. Things tend to just keep running nowadays, I have to say.

I'm not sure I'd be doing that level of constant upgrade because problems do creep in and I've done it several times (e.g. Ubuntu 10.04 -> 20.04 in stages) and run into problems half-way through that take an age to resolve, and at that point I'd rather clean-install and prove to myself that I have everything that I need properly recorded to recreate the configuration from scratch anyway, rather than just rely on constantly bringing forward some archaic config that I don't remember how it worked or what insecure old defaults it may still be using.

[Anonymous Coward]

Meanwhile in the real world

I've just been to my doctor's surgery and been told they can't make any appointments for the next *three weeks* because they are "transitioning to a new IT system".

Kudos to Ian Jackson for demomstrating what "IT professional" should mean.

AC for obvious reasons.

[Ian Johnston]

Been there done that

I be had the same computer running continuously since 1991. Admittedly it has had seven hardware changes running W3.1, three versions of OS/2 and about ten Linuces from Ubuntu 6.06 to the present Xubuntu 20.04, but like chiark it's totes the same system.

And actually, it has had the same Thunderbird profile since 1995. True.

[Ian 55]

Re: Been there done that

If you're talking about the mail client, that's quite impressive given that the first version wasn't released until 2003.