Bad UI killed the radio star

When designing a procedure for the operation of any system, the first and most important rule should always be: follow the rules. Unfortunately, this particular rule is often forgotten. Take, for instance, our reader "Lee" – the protagonist of this week's installment of Who, me? Back in the 1990s, Lee was managing a radio …

[Dan 55]

Always follow the protocol

If it goes wrong then it's the protocol's fault, not yours.

Although if you followed the protocol and it went wrong and you wrote it in the first place then perhaps you don't have a get-out clause. But even then following the protocol looks better than just mashing buttons.

[yetanotheraoc]

Re: Always follow the protocol

Advice to self: The protocol "you wrote it in the first place" while viewing the process end-to-end is almost guaranteed to be better than the steps you contemplate when "new content be uploaded to the on-air system as quickly as possible".

[NoneSuch]

Re: Always follow the protocol

"It's worse when you do so because you didn't follow a clearly stated protocol. It's sooooo much worse when you designed the protocol in the first place."

I feel for her. Been there myself. "My rules don't apply to me..." then egg-on-face lesson learned.

[Anonymous Coward]

Re: Always follow the protocol

I used to work in radio and we used scheduling software from a large company. Well that had always been DOS based and mostly worked okay but it was a little clunky and had faults, when the rest of the world had moved years ago to Windows. They were known for being inflexible and if you used the newest DOS version you were supposed to expect faults. Then they announced a new version which ‘worked’ in Windows to everyone’s amazement. Another radio station in our group were the first to use it and I was given a look at the shiny shiny.

My friend who was in charge of the testing showed me the system and said that it was basically a port from the DOS version with the same issues. He also said it was riddled with new and different bugs/faults and backups were still to floppy and not a server. To demonstrate this he ran the scheduling process in the music scheduling software for the next day and then offered me a coffee. He said we had time to go to roast the coffee beans ourselves as it was so much slower than the DOS version. He said it also normally crashes on the first run, so when we did go back it had died to no one’s apparent surprise. He started it again and it was slow, very slow, painfully slow. He said the company still had a different program for scheduling the adverts and jingles. He had tried not using that and putting everything into the music scheduling one instead. None of those would schedule if in the music scheduling system only in the jingles/adverts one.

He said that was as slow to schedule as the music one and just as painful. He said because it was a separate program you couldn’t automatically schedule a promo (advert) for say an Elton John concert with an Elton John song if they were in two separate programs. You also couldn’t do the inverse and automatically separate out adverts using particular songs from those songs. So he had asked the software company why his experiment hadn’t worked using just the one program. They either didn’t know or just flat out wouldn’t explain but since the Advert/jingles prog is free you might as well just use it.

He said this is typical of their attitude and that they’d sent someone to look at the slowness of the software. The bloke had turned up and after looking at it for 10 minutes said they needed a faster PC. This computer had a very high spec (for the time) so my friend thought that probably wasn’t the problem. Strangely this wasn’t rolled out to the rest of the group, I can’t think why. We all got a competitor’s program which was built for Windows not ported from DOS and much much better. It also used one program instead of two.

[Anonymous Coward]

Re: Always follow the protocol

Working in industry, we had a tech perform a PM (first time he did it) that tested the fail-over of two redundant controllers. Pull out first controller, verify second one took over - check. Reinsert first one, wait a minute, pull out second, verify first one took over - check. Apply PM label and walk away.

A few minutes later, the very sharp operator heard a distinct sound he recognized and yelled for everyone to get clear of the machine. Just in time to prevent at least one worker from getting severe burns from a huge blast of steam, caused by the controllers defaulting to a setpoint of 0 when reinserted.

Tech was (correctly) not disciplined, as "restore the setpoint" wasn't in the procedure at the time. It was VERY rapidly added.

(Obfuscated a bit to protect the guilty company, but it really did happen.)

[MachDiamond]

Re: Always follow the protocol

I've run into this time and again. Plans are made, meetings are held (too many of them) and a protocol is put in place to avoid doing bad things. One day there is a big "emergency" and the higher ups want something done right now and damn the protocol. Bad things happen.

[Pascal Monett]

Only one thing to say

Ouch. That must have hurt.

Backups ?

Nah. Wasn't in the protocol, I'd wager.

[b0llchit]

And the lesson

Never ever play with a live and running system. Resist the pressure to do a quick fix. Don't touch the live and running system. Period.

[Juan Inamillion]

Re: And the lesson

Hallelujah to that and double ditto. Errr never happened to me, oh no...not ever...

[Anonymous Coward]

Re: And the lesson

You cannot claim to be 'experienced' if you haven't made that misstep once.

That said, that status changes to 'idiot' if you don't learn from it and do it twice :).

[the spectacularly refined chap]

Re: And the lesson

Yep, it's always the case, typically when you have already drafted a script to do the job properly and robustly but decide to wing it because of some minor quirk of the immediate task. Swapping the source and destination to DD or rsync comes to mind.

[oiseau]

Re: And the lesson

Hallelujah to that and ...

Indeed ...

Long ago, another life in a ministry with Murphy doing the rounds.

Live NT4.0 file server and the infamous EDS in lieu of a decent RAID setup.

Budget issues and a dumb (accountant) director.

What could possibly go wrong with that?

Back up?

Yes, of course.

But the particular tape was worn, same budget issues/dumb director.

Worked Thursday from 18:00 to 03:00 and went in the three days of the long week-end and managed to fix it with a minimum data loss.

O.

[darklord]

Re: And the lesson

Yep remember those days well, spent many a happy weekend restoring Servers at the same customer due to failed backups and dodgy reboot procedures due to system slowing on a weekly basis. only thing to do was a weekly reboot of all the servers after the backups had run from Friday. and 1 server on the Red side always failed on reboot. we had tried to replace and rebuild it numerous times but it refused every time to come back up cleanly.

Even replacing the system drives motherboards and daughter boards and network cards, nope never worked so eventually the server was scrapped after 3 years of this.

My overtime alone would have replaced that server I don't know how many times!!!!!!!

[Anonymous Coward]

Re: And the lesson

"Resist the pressure to do a quick fix."

My manager has frequently said that it doesn't matter that production is down and we're losing $$$$$$/minute - STOP and think through the problem, the proposed solution, and all the possible side effects. Half an hour of double-checking before implementing a solution can very quickly save a few days' downtime.

[spuck]

Re: And the lesson

"Hours of effort can prevent minutes of planning."

[swm]

Re: And the lesson

In the early days of time sharing (1965) we would sometimes live patch the DN-30 (a real-time computer handling all of the teletypes and the master operating system). We would octal patch somewhere in unused memory. Then, if everything looked good, we would insert a branch instruction in the main line to the patch.

After the carriage return one of two things would happen:

1. The teletype would issue a line feed and the patch was running.

2. No line feed was issued and the DN-30 would reboot from paper tape.

Immediate feedback. We were just students back then. But we had fun.

[Barry Rueger]

Kids these days.....

The process involved recording content to seven-inch tape

I'm going to guess that they actually dubbed the audio (probably from tape cartridges) to seven-inch reels of 1/4 inch tape. Even 1/2" tape was pretty rare in radio.

[DougMac]

Re: Kids these days.....

I read that, and was trying to imagine 7" tall tape running through their editing bay.

The largest I've seen is 2" tape.

[Barry Rueger]

Re: Kids these days.....

I imagined head alignment....

[Anonymous Coward]

Re: Kids these days.....

the 7 inch refers to the tape diameter

[chivo243]

putting your best foot forward

then blasting it into the floor! Do as I say! not as I DO!

[BOFH in Training]

So what happened in the end?

You left the story at a cliffhanger!!!

[DJV]

Re: So what happened in the end?

Yeah, I came here to post something similar. So, el Reg, WHAT HAPPENED TO LEE?

[Filippo]

Overconfidence

It will get you every time.

[Will Godfrey]

My claim to fame...

rsync --delete {old data} {new data}

[Gene Cash]

Re: My claim to fame...

--dry-run is a godsend. Why rm/mv/cp don't have that option is an unexplainable travesty.

[Anonymous Coward]

Re: My claim to fame...

And it should be the default, with an --actually-do-it parameter instead.

[JulieM]

Re: My claim to fame...

I've tried that myself, and have the receipts to prove it. Even requiring a command-line switch to enable "dangerous mode" does not work, if you have a command somewhere in your history with --actually-do-it in it.

[C R Mudgeon]

Re: My claim to fame...

"And [dry run] should be the default, with an --actually-do-it parameter instead."

That would be hugely annoying -- and for zero gain.

I worked at one point with a GUI application that when you asked to quit, *always* responded with an "are you sure?" dialog. Not just if there was unsaved work; the dialog appeared unconditionally.

Select "quit" then click "yes" quickly became lodged in muscle memory ... with the result that quit/yes/argh! was a not infrequent occurrence.

Moral: Unconditional confirmation roadblocks don't protect the user but just piss them off for little gain.

(To be fair, that confirmation dialog wasn't quite useless. If you'd been aiming at the menu option next to "quit", it could save your ass. But it helped very little with the arguably more common case.)

As I recall, MS-DOS "format" got it right -- it asked you to confirm formatting a hard drive partition, but not a floppy. So at the cost of allowing you to shoot off a toe, it gave you serious pause before taking off your whole foot.

[Anonymous Coward]

Re: My claim to fame...

Who at Cisco decided to add a long delay and clear the command buffer between the user entering 'reload' and the 'proceed with reload? [confirm]' prompt?!?!

It used to be nice and simple... type 'reload', hammer the return key half a dozen times to get past the 'confirm' prompt and have a quick break while the thing reloads. Now you come back to find the thing grinning at you as it taunts you to hit 'return' confirm the command you entered 20 mins ago!

[Anonymous South African Coward]

My claim to fame...

...was not by formatting... but the following things I did :

1. Deleting COMMAND.COM on a MS-DOS 3.30 system (easy to fix)

2. Cloning an NT4 hard drive to another, overwriting an OS/2 hard drive (which I've used for testing). Upon rebooting, OS/2 booted. Oops, cloned the wrong way around. (Anybody remember GHSTWALK.EXE from that time?)

3. Fudging up a virus removal, and messing up the HDD. Should've done a backup first. Meh.

Otherwise you could avoid trouble by just thinking what you're about to do before punching the Enter key...

[heyrick]

Re: My claim to fame...

Mine, a quick fix to a system, just needed some wires wiggled. Didn't bother to shut down, it would take longer to reboot than the task took.

Now, about that screwdriver that got accidently dropped into the harddisc, pointy end down? Hmmm...

[much later on, opened up the harddisc, reckon I could have rigged up a stylus to try listening to whatever mystical magic was contained in the gouge made by the heads]

[NATTtrash]

Re: My claim to fame...

What about...

"Can you help us out and install xxx please, because we try to do..?"

"Sure, no worries, piece of cake."

sudo apt update

"Look at that, a new kernel! Let's install that too while we're at it."

Hmmm... Not enough space on the root partition. Jeez, what a mess. Been too busy I suppose...

OK, let's clean it up and start with removing those old kernel images...

Ooops...

I didn't remove the current one, now did I?

[aregross]

Foot...

..meet bullet!

[Richard 12]

Always cycle your disks

Many years ago, I worked with a system that used floppy disks.

It had the running data on board in non-volatile RAM, and could save one copy of said data to a floppy disk.

Save and Load were next to each other looked almost identical in the UI, right down to the "Are you sure?" confirmation prompt.

You know what's coming, of course.

At the end of a very long, multi-hour programming session, I accidentally hit Load instead of Save.

Fortunately, I followed protocol and saved every 20 minutes to a different disk, so only had to re-write the last 20 minutes of work. That only took five minutes as I'd just done it.

The modern versions of this system have an onboard SSD that can store many thousands of said files, and automatically keeps the previous hundred or so revisions.

And by default, it automatically saves a new revision before Load, so you literally can't make that mistake. I made sure of that.

[cd]

iThis

The spelling of the company name back when the Bernoulli drive came out, see Wikipedia photo...iOmega.

Yet another thing Apple invented.

[EVP]

Well done!

A bunch of annoying commercials didn’t air because of her actions? Lee, have a couple of those on me —->

[Anonymous Coward]

Re: Well done!

On the day of the Queen's funeral many of the broadcasters decided not to show adverts...

It's only then that you got to understand just how much time you spend watching the likes of that bloody Go Compare ad!

[Robert Carnegie]

Re: Well done!

Showing an advert screen for 5 minutes (since she died actually) which says that adverts aren't being shown... yes, a bit unnecessary. And no programme trails too, and I use those to switch my brain back to "pay attention" mode.

[trevorde]

dd = Dead Disk

Had a Linux noob at work run dd with the arguments the wrong way around. It didn't end well.

I know my way around Linux but still use Etcher or some graphical front end to avoid this exact problem.

[Jou (Mxyzptlk)]

Re: dd = Dead Disk

This is why I prefer to use gparted with linux whenever possible instead of using the command line. A visual representation helps avoiding mistakes by having just ONE wrong letter on the command line.

[Stevie]

Bah!

Pikers! Not one “flames shot out” in the whole comment section so far!

I’ll fix that.

I worked in an enterprise where an electrician was asked to run a new circuit. He decided to save time by not pulling the main breaker, then drilled into the breaker box with one of those long electrician’s drills.

A sad mistake.

His corpse set fire to the building.

I wasn’t there at the time, but I think “Flames shot out of the electrician” can be assumed.

[diver_dave]

Re: Bah!

Safe isolation is now pretty much the second thing you get taught after where to find the toilets.

Yes I occasionally work live but never if drilling or saws are involved.

[Hazmoid]

Re: Bah!

To be honest if you don't do safe isolation, it's likely you are going to need the toilet if you survive.