Overload: A one-way ticket to a madman's situation Take your mind off Monday with a look back nearly 30 years to the era of OS/2 and chips ending in "DX". It's The Register's weekly delve into the Who, Me? mailbag. This week's confession comes from "Jon" and takes us back to around 1992, just as the Windows juggernaut of Microsoft was readying itself to crush IBM's OS/2 under …
In my previous job as an email administrator we had a request to add a server to our email relay for a new project that was under development that would generate and send email notifications to senior staff within our customers company. As part of the request process it was asked how many specific messages were likely to be generated per day by the new system, and the answer given to us was 'around 200'. The request was approved and on the agreed Friday I night added the server ip address to the message relay on our Ironport cluster, ready for the project manager and a couple of developers to do some testing on the Saturday.
As the on-call person that weekend I get a call around 9.00pm on Saturday from my manager saying a large number of our customers senior staff had received multiple copies of the same email throughout the day on their smartphones and can I 'find out what the hell is going on?'
I duly login and check the Ironport logs and find that the new server added to the relay the night before had sent out around 100,000 messages during the day so I immediately removed it from the relay, updated my incident ticket with the details and added that until the project manager could fully explain why this had happened we would not be relaying messages for the project any time soon.
On the Monday the project manager sheepishly admitted that they had done some testing on the Saturday that had completed correctly but after he had gone home the second developer stayed on run some more test scripts unauthorised and had left them running but hadn't checked them thoroughly first, and due to a bug they were churning out blocks of messages at regular intervals.
The developer got a bollocking for using unauthorised and untested scripts, the server was added back into the mail relay and shortly thereafter the project went live without a hitch.
This post has been deleted by its author
I'm known in my office for being a "bit" of a cowboy but I also tend to be the "panic manager" that gets called in when things go to hell. But thats another story.
Many years ago we had just gotten new development and production machines that had "GASP" 2!!! CPU's so there was no way the machine could be killed by a runaway process... I was asked to see if I could do it (on the development machine obviously). Easy peasy. Code a 10 million entry string array and write a bubble sort algorithm and put all that in an infinite loop. Within seconds it had 100% utilization of the one CPU and was chowing RAM and swap space like the cookie monster on speed. The DBA managed to kill the process and I was asked NEVER to throw that process in twice as it would totally disable the machine. Oddly enough I've never been asked to stress test anything again. Can't imagine why....
I used to run simulations like that during my PhD. I had to diagonalize a 43x43x43x43 density matrix on a machine with not enough RAM. Mind you, if there had been more RAM, I'd just have upped the sizing :-)
I couldn't do this all the time, however - it tended to annoy the other users. Gave the hard drive a good workout, though. CPU usage percentages were down below 10% (I think I recall even 3% at some point).
I wanted to see if a Raspberry Pi would break under stress so I coded a quick script to clone itself and churn a bit.
Less than a minute later the whole thing was shirtcanned gracefully by the system for being a dick.
$35 computer does job properly. Newsateleven.
I read this with 2 hats on (I have multiple hats).
Administrator:
*That's BRILLIANT!*
Programmer:
That's catastrophically bad.
I need to know what's _actually_ going on in order to adjust my approach to the extant limitations.
The key point is obvious: different Contexts/Environments are appropriate/required for different tasks. The Admin's task is to have a PRODuction environment running smoothly, for a large number of people; the Dev's task is to identify and create Changes which can then be introduced to PROD. As such, Dev needs the ability to safely break things, so needs its own DEVelopment environment. Isolation; not just for the role, but per-programmer, AND per-task.
Problems only arise when you mix the 2 environments.
I'd say perhaps half of the Admin bitching on ElReg re Devs boils down to Devs being forced into a Prod env. "Interestingly", about half the time it's clear the Admin's own actions have forced the Dev into a non-dev environment.
.
Worrying that the Raspberry Pi, nominally a development tool, has adopted an Admin approach. Hopefully the kill was triggered by a separate and disable-able process.
And related here before. A fellow programmer was the proud recipient of the first VT1000 in the company. This was an X-Terminal (i.e. X-Windows terminal) at a time when the rest of us were on VT100 or VT220 terminals or DEC Rainbow PCs. It had a "huge" 17" display and, most importantly, it ran X over thin coax Ethernet.
Our programmer did some demonstrations for those of us not so lucky to have such an object of desire. He showed off xEyes, a pair of eyeballs in a window that followed the mouse pointer around the screen. Gales of laughter. Then I asked if it could instance more than one copy... He dutifully filled up the monitor with over 50 copies of xEyes, very carefully placing each one. The VT1000 was stuttering a bit by then. Then he quickly moved the mouse around the screen in random directions. The first couple of eyes kept up for a brief few frames...
Then the VT1000 stuttered to a halt.
Then the VAX on wich the xEyes were running stuttered to a halt.
Then the network collapsed.
Over a hundred eyes and very quick mouse movements were too much for the VAX and for coax Ethernet running at 10mbps.
Ditto!
Add in xtank wars on Friday nights. And xNetTrek. And writing to each other's screens if they'd been lazy and used the default xserver syntax (open/no security -- accepted all calls from any machine). (The swearing when someone closed a window and their screen exploded with cockroaches :) And ... And ... And ...
So much simple novelty. Pioneering days.
A few days ago I was randomly browing information about 1980s era terminals VT100s, VT220s, and the likes, and my brain clattered to a halt when reading "....typically cost $1500..." "several thousand dollars..." etc.
WTabsoluteF???? Several *THOUSAND* dollars just for a ****ing monitor and keyboard! When you could get an actual *computer* for just a few hundred quid that could do everything a terminal could do *plus* it was an actual computer! How the actual F was this ever a viable business?
I remember back at uni in the 1980s wondering if I asked the computing department nicely about the bashed-about terminals if they'd skip one in my direction as they'd make a nice cheap monitor better than the TV I was using.... never realised the contents of the terminal rooms probably cost close to a million quid.
Except, a good quality keyboard, like those used on the terminal, would cost a couple of hundred dollars, heck, an equivalent keyboard mass-produced these days still costs between $100 and $300.
Then you have the high quality display, that could display 132 columns compressed or 80 normal. Most PCs of that era either used a telly (32 columns) or you had to provide your own monitor.
Don't forget, an IBM PC at that time would cost several thousand dollars ($4K?), didn't have a graphics card, monitor, floppy disk drive or keyboard - they were all extras - and had 64KB RAM. All "optional" extras.
Terminals weren't cheap, but they also weren't that expensive, compared to "professional" computers.
They were also on the COCOM list, you couldn't sell a VT100 to the East Block, it was advanced technology! We had a lightning strike and around 400 VT100s died. We had to get them all "professionally destroyed" and a government official came down to witness the deed and sign a certificate to say they had been destroyed (we used the car-compactor and shredder of the scrapyard next to the factory).
Ah yes. I was at Unisys in 97, we used dozens of xterm pizza boxes for our xwindows GIS app. Then we scrapped them for NT4 with X.
Optimistically I thought I could sell them so, with approval, rescued them from skip. I never could as no one had invented Ebay yet so I stuffed all 18 of them with 21” monitors in my rented house loft and forgot about them.
Cue irate phone call from landlord 18 months after I left as his new tenants couldn’t fit anything up there.
Umn! I saw some of these 'illegal' imports in Soviet Russia when sorting out stuff for their Petroleum refineries. They were in locked offices with a full time 'female guard' to permit authorised access. I was told these and other prohibited hardware and software came via 'returned hardware' in USA. The scrap merchants were mostly foreign owned. (Like sending ships to scrapyards in Indian sub-continent nowadays)
Even in the 70's Commodore and Apple went with default 40 column displays, Tandy went with 64 columns (but only 16 lines instead of 24 because still only 1KB of video RAM). 80 columns on a TV was pushing it a bit though. I remember BBC computers on cheap monitors where 80 columns was a bit of a strain on the eyes.
Interestingly, at about the same time as its release, BeOS machines with a very different low-level architecture could run 3 or 4 videos simultaneously. AND (the absolute jaw-dropper) you could grab one's title bar and shake it all over the screen without ANY of the videos so much as flickering.
It's only in the last few years that I've seen "modern" retail kit capable of doing the same.
of my first PC. Quite a beast it was in its day, sporting an 80386DX with Cyrix 387 floating point coprocessor, a whole 4 MB of RAM (later upgraded to 8), a graphics card with a whole further MB of RAM) and an Adaptek SCSI Controller with 88 MB disk! Cost quite a fortune at the time. Sped up my development work no end, and Windows 3.1 and MS-Office worked quite happily in 4 MB. I don't think that would do for the latest incarnations, would it now.
Wow, 4MB on a 386 was quite a bit back then. And the name "Cyrix" takes one back to those olden days...
I did manage to upgrade my 486 to 20MB at one point, helped a lot with gaming. Quake (Team Fortress), Rage, ... (yeah, and the dynamic object oriented machine earlier on as well, DOOM for short...). Memory fades a bit as to when I upgraded to which machine later on.
I also came from a family with a 386 DX33 with 4MB RAM (later upgraded to 8MB), although we only had a 512k graphics card. I did remember my dad shelling out a small fortune for a 440MB hard drive to go in the thing alongside the 85MB original - it was huge for the time when all software came on floppy disks!
From the article, there is actually one small inaccuracy (if I put my pedant hat on). Intel's desktop 386s topped out at 33MHz, only AMD made a 40MHz 386 CPU...
Testament to the solidness of OS/2...
Heard that some folks preferred to use OS/2 when they have to collect massive amounts of data over serial links as Windows could not keep up with the demand.
...still trying to find the article where OS/2 on a single CPU outperformed NT on a quadprocessor setup. Ah, those were the days.
Hmm, someone needs a history lesson! https://en.wikipedia.org/wiki/Email
Yup - we were using it in 1980. And by no means the earliest regular users. My partner (now) was working in the UK at the time, and email between ourselves in Norway/US and Aberdeen, not to mention a very crude system we had set up (it was awful) save us a bucket load of scarce money.
Pish. I used to do so casually to pull people's legs.
I can remember ~98 a major quant scoffing that computer systems were secure and fraud impossible; laughed at me when I said anyone could send any email from anyone. It was just before Christmas and he came boiling back in wide-eyed and laughing 5mins later.
"I JUST GOT AN EMAIL FROM SANTA CLAUS @NORTHPOLE SAYING I WAS ON THE NAUGHTY LIST AND I SHOULD BUY YOU A CHRISTMAS DRINK!! DID #YOU# DO THAT?!"
telnet -25... :D
Hear hear.
The same day Floyd was killed, Chicago saw 10 black men killed by guns, all fired by black men. Last figures I saw were approx 6% USA population being black men. The chances of that not being black vs black choice AKA "racism" are therefore 0.06^10. Put that into a calculator for a brief boggle -- makes winning the lottery look like a doddle. Oddly, this didn't get the same airtime let alone outrage.
The average, incidentally, for USA black male violent deaths is that 90% of them are committed by black males. Racism!
Back in the heyday of mainframes, the university had installed its new Amdahl box (IBM 370-like). They opened it up for stress testing. One bright soul noted that the "MVCL" instruction would take registers containing source address/length and destination address/length. So, that one instruction could access 32 Megabytes of virtual memory. And one could arrange that the memory was not yet created in the OS. Unless they did something illegal, instructions would run to completion before anything like a task switch could happen. Run a few copies of that program and the mainframe was on its knees. The offer of wide-open testing was soon withdrawn, but I think I heard that the systems folks were good sports.
Later, in a job with a computer company, several folks were working from a site in California (head office in Alberta, Canada). They would often have a phone connection to the systems at home to allow access to sources, etc. One late night that connection was forgotten. When the phone bill showed up it had just asterisks for the cost (Fortran and other languages tended to print asterisks for "number too large to fit in output field"). I believe in this case, the phone company was eventually convinced that this wasn't possible and that it must be some kind of error in their systems. All was well.
In all honesty, I have to admit that neither of these miscreants was me. Sigh.
I wondered that myself a while after my post. My recollection is that it wasn't just making the system page itself to death - there was something special about using MVCL (can't recall if CLCL would have worked as well). Hmm. I wonder if the pages were all ready to go, so no page faults, but the instruction then somehow locks them all during execution?
MVCL was always interruptible.On an interrupt, the hardware would update the registers; on return, the machine would just re-execute the MVCL.
A much more interesting way of swamping a 360 or 370 with virtual memory was to process a huge matrix along rows, not knowing that Fortran stored arrays by columns. This could bring our university's 360/67 running a time-sharing OS to its knees, as the offending process ended up having to swap in a new page for each element access.
Bit baffled myself. I'm reading it as: run the script's run-time commandline parameters in background, then copy them again to the commandline minibuffer and then.... type a meaningless dollar sign (AKA syntax error).
At best, requires you to enter a valid command after the script name when invoking it.
Unless there's some new Linux bash oddity?
Vs: while true; do; fork; done.
BTW, re $*, nearly always better practice to use $@.
$* merely copies the parameters as-read.
$@ delimits each with "s.
Becomes suddenly very important when whitespace (or sometimes even non a-Z1-0) appears intra-parameter.
Essentially, for the sake of switching a character, you occasionally eliminate major ball ache or even catastrophe.
In my case, as a pre-sales engineer, I once brought an iSCSI array out to a prospect - s well known web music broadcasting platform that you may well have installed on your phone.
The customer network/storage guy was eager to see what to performance looked like, so he provided a couple of shiny new Windows 2012 servers with lots of ports on it, and I dutifully set them up correctly with all the Microsoft multipathing - with 4 ports on the stroage device, and 4 per server, that is 32xGbE paths between the storage and the servers...I also had a load generator built from IOmeter that basically would pound the snot out of any storage that you pointed it at. It got used regularly to make sure that storage was running at
The Customers storage guy was unconcerned when I asked several times that the load generator was absolutely segregated from the production network. His answer was simply "You're on a completely different switch" so, there you go.
After risking RSI with the 14 mouse clicks x 32 paths for the test, I had it all ready to go, and issued the command to start the load gen. The prospects eyes grew wide as the performance numbers ran quickly up to insane levels from what he was accustomed to, But his surprise was quickly interrupted by three people running at us from three separate directions yelling "SHUT IT OFF! SHUT IT OFF!"
It turns out that rather being on a separate network switch as I had understood, we were actually on a separate blade on the same chassis switch (an older Cisco Catalyst 6500) and the switch had fallen over from the load...some quirky thing called "Flooding the Backplane"...and yes, that switch was the production switch and we had just taken them completely "Off the Air" until the switch could reboot.
Post mortem, it turned out that the storage/network guy had not been dilligent about keeping the switch updated, and consequently, the network flood would could been avoided. The punch line was, the company eventually bought my storage device, but the demo delayed the purchase for several months as funds had to be reallocated to PS to get the switch properly updated and configured...and the person who finally bought the storage was not the (no longer with the company) former storage/network guy.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
