Don't touch that dial – the new guy just closed the application that no one is meant to close A story from the world of television in this week's edition of Who, Me? as a weather presenter sweats while panic reigns supreme in the backroom. Our tale, from "Mike" (not his name), dates back to the early part of this century, when Windows NT 4 and 2000 ruled the roost and processes were a little less slick than today. …
I managed to mistake two commands on a newsroom scripting system (early eighties - probably Basys running on mini-Vaxes). This provided access to stories for editing and printing, and also delivered them to the autocue systems and provided printouts of running orders, times, and the like.
One command restarted an individual users' session in case they locked out their dumb terminals, the other restarted the whole system.
Guess which one I chose at three minutes to six? Guess what took a good two minutes to start to come on line again? Guess who had an interesting conversation with the station manager after the show?
All this "who me" does is show that the organization had shiity practices including shitty training for new employees. Yes, the PA should have asked for clarification of where the logs were, but in my experience organizations with such poor practices generally have a culture that makes it impossible for newbies to ask for help.
Not just that, but if it's such an important application, why was it possible to shut it down with just a click? It should have popped up dire warnings, asked for a password, or simply refused to shut down until the computer itself was being turned off or rebooted...
Oh, yeah, that's another thing I've learned - applications should never, ever, become foreground unless the user has explicitly requested it in the last half second or so.
If something happened and you want the user's attention, blink your task bar icon like a good boy.
And do not, under any circumstances, overwrite the system clipboard without the user's express direction to do so. You never know what valuable data you might be erasing the one remaining copy of. (Bad practice from user, but still.)
I'm looking at YOU, badly-written Excel macros!
(If you think this isn't a problem, imagine spending half an hour writing up something to put in an Excel form, copying it, naively closing the window (without saving) where it was composed, and opening the form... which then uses the system clipboard to copy-paste stuff to set itself up, destroying your work.)
Oh, yeah, that's another thing I've learned - applications should never, ever, become foreground unless the user has explicitly requested it in the last half second or so
'
I know, right? And yet multiple MS program do it - several times, just for good measure - every time they feel the need. Never mind the opportunities to shut down important apps, if you're not the world's best typist you might not be looking at the screen while entering something important - only to look up and find you've entered an Office activation key into the wrong screen. This can be ever so slightly frustrating.
What the hell is it with the Office 365 web version of Outlook where some random part of its UI suddenly gets focus while you're typing an email and not only swallows several keystrokes but acts on them as commands?! If you're lucky, they just get redirected to the browser UI and you find yourself doing a "search within this page" but sometimes you end up deleting the message you're replying to, archiving another one, snoozing one, emptying the folder...
At first I thought it was either me palming the touchpad on my laptop (although I've never had a problem with that in the past, but maybe the anti-palm thingy had stopped working...) or using an "unsupported configuration" (the laptop runs Linux). Then I had exactly the same problem while using it on a Windows 10 desktop so it can't be either of those things.
I'm seriously tempted to defy the company standard practices and use a real email client.
I have to drag emails from Outlook into Explorer to save them (stupid Outlook Janitor...). I rename them to add the date of the email at the beginning. Since I'm putting them into a OneDrive-synched directory, it uploads them to OneDrive a second or so after I put them into the folder. I've had to make a habit of copying them across, waiting a couple seconds, THEN renaming them. Otherwise I'll start the renaming, begin typing, then the upload happens which changes the status, so it "helpfully" selects the ENTIRE filename while I'm editing, and my next keystroke wipes out the filename instead of adding the date at the start of it.
> why was it possible to shut it down with just a click?
I know nothing of course, but I can imagine that 1. this software wasn't supposed to run on a publicly accessible computer, and 2. mission-critical stuff operated by supposedly trained crews shouldn't discuss any orders received, they should execute them immediately. There is no place or time for useless legalese/nanny crap, a mission-critical-level user is supposed to know what he's doing, else he shouldn't be there.
The problem here was that people allowed to access that computer had not the required (or apparently any) training.
"There is no place or time for useless legalese/nanny crap, a mission-critical-level user is supposed to know what he's doing, else he shouldn't be there."
This. Because in many broadcast/transmission facilities, there is a lot of stuff one can do to knock the station off the air. We can't put access controls and padlocks on all the hardware. Sadly, many broadcasters (at least on this side of the pond) aren't up to the task of developing procedures and training employees.
Anecdote: We have a number of Christian broadcasting stations in my area (Jesus channels I call them). On several occasions, they have broadcast nothing but color bars or test patterns for weeks or even months instead of their program material. I guess they can't be bothered to check the quality of their outgoing signal. Or nobody (even their employees) wants to watch some hellfire and damnation minister yelling at the studio camera. At least the crazy guy on the street corner soap box has the advantage of personally seeing if he has an audience.
"On several occasions, they have broadcast nothing but color bars or test patterns for weeks or even months instead of their program material."
None of the regular viewers phoned in? Perhaps no regular viewers, in which case advertisers getting milked / or its a scam?
Monitoring the output sounds like an ideal application for one of these AI programs. IF the image does not change much if at all AND IF the audio is silent/single tone/ repeated cycle of test tones THEN email admin?
Monitoring the output sounds like an ideal application for one of these AI programs. IF the image does not change much if at all AND IF the audio is silent/single tone/ repeated cycle of test tones THEN email admin?
Yup. That's state-of-the-art AI, then....
I work on an alarm system for broadcasters and all the kit nowadays has black/freeze/silence detection, and when one of those happens for long enough it will trigger an alarm which will make a noise in the control room.
(it doesn't need AI)
Small broadcasters often don't have an alarm system, even now.
Some broadcasters (public service) will get fined for broadcasting black.
Commercial broadcasters won't get paid for advertising if the advert doesn't go out as agreed.
Knowing Murphy, there are at least two ways this setup can still fail.
One, the alarm could stop working.
Two, you could end up with a snowy screen instead of a black screen: something your system won't detect that is still a failure mode.
Agreed. A lot of what we take for granted as part and parcel of operations had to be learned from mistakes. Institutional knowledge is a thing which is why I think companies/governments/etc. are struggling right now with all the outsourcing. No matter how much documentation you provide me when I come on board, I can't digest it all and be the perfect employee day one. In the mean time you have a gap that the position I was hired for was intended to fill. Throw in the tight budgets reducing or eliminating redundancy of staff capabilities and you are treading a fine line of failure and success.
Risk management was not taken as seriously say 20 years ago as it is now. and I am sure this was an incident that caused change (as was related at the bottom of the article). Was the change perfect? Not likely but you have to start somewhere and re-architecting a tried and true process is NOT something done over night. And may be deemed too risky versus improvements to the existing process.
These days, whenever I design an application that's supposed to be a single executable, but also does critical service-like things in the background, I do it like this:
- if you push the familiar X button, the GUI closes, but the application keeps running, showing an icon in the notification area.
- if you double-click the icon in the notification area, the GUI reappears.
- if you attempt to start a second instance of the application, what actually happens is that the first instance's GUI is restored and put in the foreground.
- in order to actually shut down the application, you have to do something reasonably easy to figure out, but non-standard, such as press a non-system button or use a specific menu item, and then give confirmation.
- I usually don't require a password to shut down, because if someone has got to that point, they'll just use the task manager anyway.
This appears to work fine for both power users and the clueless majority.
I've actually seen Applications that require the user to perform a specific, but randomly chosen sequence of actions to exit.
OK, nothing stopping someone really determined using task manager to forcefully exit them, but it actually helps prevent accidental shutdowns if someone needs to read, and then follow a sequence.
Generally, I find it best not to put restrictions like that in, unless they are necessary. People bypass them, sometimes causing more problems that the restrictions solve. For instance, at work, we have a restriction on all the PCs that the shutdown option is disabled if you are logged in. The idea being to force you to log out then shutdown (although as Windows logs you out anyway when shutting down, not sure what difference it makes). Most users do log out, then shut down, but there are a group who don't. They just hit the power button, and wait the four seconds or so for the power to turn off. Thus, potentially, causing themselves to lose any unsaved work.
Not just controlling shutdowns; I've been asked to take away a "restart everything on the network" button even though it did have "are you sure?" "are you really sure?" buttons (*).
A tired broadcast engineer in the middle of the night can very easily hit "restart all" "yes" "yes" without thinking and when he really just meant "restart this" "yes" "yes".
(*)
It took about 20 minutes to restart all during which time you had no control of anything on the network (**).
(**)
Unless you were a properly trained broadcast engineer and knew that actually you could connect individually to each of the separate bit of kit, using the manufacturers own diagnostics (a web page (all different) or a particular manufacturer's own control application) and take complete manual control of everything one at a time. Networked control systems exist to avoid needing to go in at that level.
I have always found "problems" like this to be very helpful because they help you learn what the users have not been taught to do. This means that you have the opportunity to "fix" the problem by updating the training documentation.
All too often users get blamed for doing something in an attempt to resolve an issue that nobody has ever documented at all.
It would have been funnier to "accidentally" type into the teleprompter something silly like "I am a chicken!", "I have a pickle in my ear!", or "I wanted to be a lumberjack!" for the on air talent to read to the televised audience. Imagine the fun to be had! (And try not to imagine the right bollocking you'd get from the boss afterwards.) =-)p
When you have a president declaring war on Covfefe having an intern in charge of the teleprompter is probably an improvement.
And a "sonic wave" caused by the fire suppression system removing oxygen from the room that caused servers to shut down, apparently. Is that a thing? https://www.broadbandtvnews.com/2021/09/27/uk-broadcaster-outage-blamed-on-sonic-wave/?mc_cid=48094095be&mc_eid=UNIQID
When it happened in our server room, from an electrical surge caused by a direct lightning strike on the roof, the overpressure moved partition walls and brought down ceiling tiles. Several of the disk drives registered the pressure change in their telemetry, and some ceased to function.
Our data centers are setup to kill the power in the event of a halon release. No graceful shutdown, just a hard stop when the circuits trip and the UPSes stay offline. Seems as if it is by design to avoid the impact of a pressure wave or any other issues that accompany a halon release.
It depends on the fire supression system, but in some systems the Halon ( it's not halon anymore, but for lack of better term... ) is kept under very high pressure ( 600bars+ ) so when they get triggered, you basically have flying raised floor tiles and well explosive sonic waves. [ note : I only ever saw one datacenter equiped that way, and the security briefing was done that way :
If the Fire Alarm is triggered you have 30 seconds to leave the room, do not pick up anything, do now try anything, run, run, run, run, 30 seconds is barely long enough for you to reach the door from the back of the room before you get plastered between the hanging ceiling and the raised floor, as one will be flying down and the other will be flying up.
( and yes it had rows of those 600+ bars red bottles tucked all around with 20cm+ sized tubes going up ( in the hanging ceiling ) or down ( into the raised floor ). )
The whole point of this kind of system is to kill the ( notional ) fire ASAP, and to hell with eventual collateral damages, those can be dealt with after the event and once the room has been vented.
Though the most common I see uses 200 bar bottles, which gives you a little bit more wiggling room.
( you still do not want to be in a room where the bottles are triggered )
I'm old enough to remember the time when the football results came in to "Grandstand" on a Saturday afternoon on teleprinter from some central location (accompanied by a nice teleprinter clacking noise presumably dubbed over) to be read by the announcer, and one day the results came in as "Arsenal 4 Chelsea 2... Glasgow Rangers 2 Hibernian 3... Fire Alert..." - yes there had been a fire alert at the sending station and some quick thinking chap typed "Fire Alert" into the teleprinter before racing after his colleagues to the fire exit. Back in the seventies I think. No doubt someone with a better memory than me can give a better recounting.
Now, yes.
Back then, not so much. All users with access were expected to know what they were doing, so the password (yes, it would've been one) would've been shared to all anyway.
If it needed nursing back to life (as the article states), a watchdog to trigger a restart could well be worse than having to do it manually.
Yes, it most likely is done differently these days.
"Yeah, like who uses KVMs in these days of virtiualization?"
The O/P mentioned this happened in the days of NT4 and WIn 2000, so it was quite "normal" to employ KVM's back then...
Today, one could use AnyDesk or similar Remote Access software, so virtualisation is not even needed.
Whatever guards were in place wouldn't work. Apart of course from human guards with orders to shoot to kill anyone going near anything critical.
If a dialogue box popped up saying "You've chosen the destroy the universe option. Are you really, really sure?", fuckwits like the PA in this story would always click on yes. As would the fuckwit's boss. Or their boss. Or...
It isn't just your wife. There are playing-at professionals that do this. You can be walking them through a procedure where, for the first time, they're in control of the keyboard and mouse, and even though the rules are 1) you tell me what you're going to do, 2) i confirm that its the right thing to do at this time, and 3) you do it. Decide, in their infinite wisdom, that after the 4th This?, Yes!, click/type/click, that they can continue with the rest of the steps that they don't understand, aren't reading, or are just making up; and if you tell them to stop, THEY DON'T until they finish the step that they shouldn't have been doing (or doing in this manner) in the first place.
Did once make a colleague burst into tears, when, on watching them slide inexorably toward disaster despite increasingly frantic pleas, they were about to jollily tap Enter to destroy all that month's accounting data, I had to intervene with a swift fist to the Escape key. Keyboard broke, as did colleague. Only one of the directors appreciated this intervention.
Our tale, from "Mike" (not his name), dates back to the early part of this century, when Windows NT 4 and 2000 ruled the roost and processes were a little less slick than today.
<Looks at the gaffer tape and shellscript holding it all together>
<Thinks back>
Well, the tape is a bit shinier I suppose. Oh, and there's less Perl....
If you make something idiot-proof, all that happens is that someone creates a better idiot.
The mission critical stuff at work has paper notices and pop-up-at-login messages that are quite blunt (Think "Unless you KNOW what you are doing here (Hint:You do not!), walk away now and call $KnowledgeablePerson immediately!"), But I still have to clear up the mess left by some over-eager noob or some stressed-out go-getter with poor judgement and over-confidence.
The latest one is that someone was told changing some DHCP parameter would help with their VPN connection...someone in authority gave them access to 'DIY' because us-that-know were elsewhere. This authorative person is now metaphorically bleeding from the mouth and nose....
"Certainly "bigger" and "idiot-proof" don't fit together very well."
Oh, they do some of the time. A small set of utilities likely don't include a lot of automatic recovery, warnings, etc. Very convenient for the knowledgeable, terrible for someone with more power than they should have. A bigger system created with the goal of idiot-proofing will have more ability to recover from a user doing something stupid, and that can be one of the most important things. You're probably thinking of systems made bigger not for idiot-proofing, but merely because of feature creep, which indeed doesn't help at all.
I don't think so. Idiot proofing is also making things easier for every user, including the knowledgeable ones. Things meant to simplify the common tasks for someone who would mess something up if asked are also useful for people who just don't want to take the risks or who want to get things done faster. You may be one of those people who take pride in using the smallest number of tools to get the job done, posting your comments via telnet and manually entering the HTTP headers, but if you're not, you're benefiting from years of others' idiot-proofing of the software you use.
Mid-80s, the BBC were playing with general distribution of 625-line colour TV around 70 Mb/s. 34Mb/s was used as the final stage of distribution to the transmitter, when no more coding was anticipated. And then along came HDTV, in its studio format requiring around 1 Gbps.
10 MB/sec for raw? Not likely, a good DVD encode will push 6-7 for an MPEG-2 steam.
Raw is going to be a fair bit more. I get 216 Mbps for PAL.
Source: https://www.erg.abdn.ac.uk/future-net/digital-video/mpeg2.html
Good find! Ta. Filed.
So looks like everyone in broadcast had access to and could rely on MPEG 2 availability.
I wonder what Raw *contribution* PAL is? I couldn't find a definition anywhere. (And that table only occurs twice in the search corpuses! How on earth did you find it?)
Raw contribution PAL is a phrase I never recall hearing in over thirty years as a broadcast engineer.
If it was PAL it was analogue, 6MHz bandwidth with a 4.43361875Hz colour subcarrier; audio was always separate until it left for the transmitter.
If it was digital it was CCIR601 - ten bits sampled at ~six times subcarrier frequency. Digital audio could be inserted in the sync pulse time. That is uncompressed.
But with a broadcasting chain as complex as the BBC with multiple channels, local opt-outs, provision from multiple sites to the network... naming gets a bit complicated.
I've mentioned this little tale before - though I must stress that it's one which was handed down to me.
Back in the mid-90s, there was a bit of a skunkworks project at a major telco, to see what could be done with this newfangled World Wide Web malarky.
And so a demo was put together for higher management - but as the technology was literally still being written, there was a lot of smoke and mirrors involved in said demo, and the (distinctly non-technical) person hosting the demo was given a very precise script to follow.
Come the day, and the guy doing the demo became nervous and started to wander off-script. Cue lots of curses from the techies in the background, as they frantically tried to wire together stuff in realtime to try and make his improvised actions work...
After the demo, said demonstrator was nearly throttled when he wandered off the stage, came round to the techies and cheerfully said "Well, I think that went pretty well..."
It's a well known fact that the original iPhone demo had to follow a carefully arranged sequence otherwise the phone would crash. They also hard coded the signal strength bars to show good signal.
This also reminds me of the time I was the person in charge of the dumb waiter for the play "The Dumb Waiter". During one performance the actors managed to skip 3 pages of dialogue, go through another couple, skip back to the pages they'd missed, did those and then skipped forward past the pages already done. Meanwhile I'm going frantic behind the set trying to work out if I'd missed a cue. Luckily I hadn't.
That's the best way to failure.
back in 1993. The teacher had told me to pick any free computer and quit any program that had been left running. So I found a nice computer at the end, that was running a program I didn't recognise called "File Server" so I quit it. I was later (well, about 10 seconds later) informed that there had in fact been a notice displayed above this computer saying not to use it, but I hadn't been looking up there, I'd just been following the teacher's instructions. I still think he should have said pick any computer except the end one. The school later put their server in a cupboard.
> I was later (well, about 10 seconds later) informed that there had in fact been a notice displayed above this computer saying not to use it,
Under such circumstances the appropriate location for such a sign is over the top of the screen with another over the keyboard
Taped to the screen yes. Keyboard disconnected, no. Although that does depend on how long ago it was and what is meant to happen after a power failure. XT and AT keyboards, you could generally get away with hot plugging them. PS/2 keyboards had a habit of not being recognised if plugged into a running system. Either of those XT/AT or PS/2 types could blow the keyboard fuse or, if not fitted, kill a motherboard. Likewise, again depending on how long ago and the BIOS in use, it may or may not have an option to ignore the missing keyboard error on boot, even if it did have an option to power back in after a power outage. (Prehistoric PCs didn't have "soft" power at all. The power switch was directly wired to the mains, and were very likely to report "Keyboard Error - Press F1 to continue" with no BIOS override option to be set in advance)
The school in question had Acorn Archimedes computers running RISC OS (I think it was at version 2 or 3). I have no idea what happens if you unplug the keyboard on that. I do know that if you press F12 it pauses all tasks and drops you to a command prompt, which is another thing you don't want to happen to a server.
An enterprise software company wanted to stress test it before making it available to customers. 20 of us new grads were rounded up, and given a list of all the possible commands, and we were told to type in any command. It the system stayed up for an hour the test passed and we would get a cup of tea and a biscuit.
Most people started from the top of the card, so I started at the bottom.
I typed in a cryptic string of characters like ZYQC P ST I, and there were cries as the system shutdown.
"Who shut the system down" the manager asked?
Deadly silence
"Did anyone type in ZYOC Perform STOP"
me:"I typed in p ST I"
"OK no biscuit for you - for the rest of you ... use any command except for the P STOP command"
Two years later they had software scripts executing the commands and programs, and avoided the problem of dumb users. These scripts could generate 100 times the load rate, and didn't get tired.
As a PG student I discovered, entirely by accident, that the command "shutdown" would, erm, shutdown my research group's microVAX, as they had failed to restrict it to admin types. We reported this an were assured that it had been fixed across the department. A couple of weeks later somebody asked me what I had done and, as I was sitting at a terminal, I thought I'd show them, so once again I entered "shutdown", confidently expecting the command to be rejected.
Unfortunately, it wasn't. Even more unfortunately, I wasn't logged onto the microVAX, but to the 11/780 which supported all the departmental teaching, most of the research and all of the admin. It all went very quiet and I got a written warning.
Now, what's the IP address of the computer history museum VAX ... ?
I think that particular one is no longer a thing. With ACPI and equivalents, it's more likely that holding the button will force a shutdown whereas pressing it quickly will do an orderly one (or, we can hope, do nothing). That cuts off the last second of abort opportunity, but at least it doesn't leave that painful moment or the despair of having missed that too.
Many years ago, there was a story about a computer running the late-night TV schedule on one particular channel. It was a Saturday evening in late March, before the era of 24-hour broadcasting.
As Saturday night rolled over into Sunday morning, the channel was showing a late-night film. As it reached 0200 hours, the system clock moved over into BST and jumped ahead an hour. At this point the logic realised that it was past closedown time and abruptly shut the channel down for the night, in mid-film.
There were no anguished calls to the TV channel. Apparently the late film's audience consisted entirely of video recorders.
I've seen this sort of problem where managers in an engineering department that aren't engineers themselves decide that a task is so simple that they can just have an intern do it so one of the other regular staff aren't pulled away from something else. The task looks so simple when staff does it that the intern should have no problem.
We lost several expensive rocket engines due to that sort of thinking.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
