back to article Weeks after Red Bee Media's broadcast centre fell over, Channel 4 is still struggling with subtitles

Confusion continues to reign in the world of television, including UK national broadcaster Channel 4, weeks after a broadcast centre cockup wrought havoc upon servers. Things went horribly wrong at Red Bee Media's broadcast centre back on 25 September. Yes, that was the weekend before we ran an accidentally appropriate episode …

  1. Dan 55 Silver badge

    Oops

    Red Bee: Doesn't really matter, the broadcasters have got a backup anyway.

    Ch 4: Doesn't really matter, Red Bee have got a backup anyway.

    Insert the adage about the backup hasn't been tested until you restore it here.

    1. Phones Sheridan Silver badge
      Mushroom

      Re: Oops

      All the HD masters for Babylon 5 were lost in the same way. The production company thought Warner Brothers would store them, Warner Brothers through the production company would store them.

      The production company had the foresight back in 1993 to see that HD and widescreen would be the norm in future, so they recorded the show in both, but in such a way to broadcast it in 4:3 such as having the main characters stand within a 4:3 square of a 16:9 shot. They didn't have the foresight however to make sure they had backups. We'll never get to see it as originally intended and shot. The DVD was a mix of what footage they had left, mixed in with recovered-from-video footage.

  2. simpfeld

    Missing a few key points here

    1/ Slightly glossed over is the fact the servers are fine but the hard drives in them died. It was generally known that the noise from gas fire suppression systems can damage hard drives, but not sure it was well known. I certainly didn't know how extensively, percentage of HDD's that would be destroyed, this can cause.

    A little bit more from C4 themselves.

    https://www.channel4.com/press/news/whats-happened-access-services-channel-4

    2/ "Red Bee Media got the pictures and audio up and running again quickly," , yes but the key thing is here not from the primary site but from a DR site.

    They are all still on DR, see Channel 5 with the white and black square in the right hand corner, indicating they are *still* in DR playout. The issue is Channel 4's DR subtitle playout isn't working. I heard some concern that fixing this is dangerous whilst in DR (as they have no further DR if they break something). So changes to the Red Bee's DR being made are very limited.

    1. H in The Hague

      Re: Missing a few key points here

      "It was generally known that the noise from gas fire suppression systems can damage hard drives, but not sure it was well known."

      Fairly well known to El Reg readers me thinks.

      My knowledge of fire suppression systems is v outdated. Does anybody know if such a system would be equally effective if it discharged the gas slightly more slowly? Or does it need to discharge so rapidly to get high turbulence = effective mixing?

      1. Cynical Pie

        Re: Missing a few key points here

        AFAIK it needs rapid release to effectively drive the oxygen from the environment and suppress the flames.

        Thats certainly how I was told it works in an archive such as the TNA and their storerooms

        1. Doctor Syntax Silver badge

          Re: Missing a few key points here

          In the case of TNA it makes sense. Properly boxed and shelved paper and parchment is probably going to survive the vibration. Spinning drives aren't, at least not with the same probability.

    2. skwdenyer

      Re: Missing a few key points here

      That post from C4 seems to suggest that there were no functioning backups, never mind functioning backup hardware. If so, that's pretty extraordinary.

    3. Flywheel

      Re: Missing a few key points here

      the white and black square in the right hand corner, indicating they are *still* in DR playout

      Ah, thanks! I noticed this and assumed it was a signal for "more bleedin' adverts coming up", but when it stayed on all the time I guessed it wasn't. Nice to know though!

    4. Dermot Bradley

      Re: Missing a few key points here

      > It was generally known that the noise from gas fire suppression systems can damage hard drives, but not sure it was well known.

      Yupe, well known indeed, e.g.:

      Zurish Insurance: http://www.zurichservices.com/zsc/reel.nsf/b777a8062cedf191c12571fe00467717/6361b933b360dbac862584790079666a/$FILE/rt_fixed_fire_protection_data_centers_and_HDD_noise_sensitivity.pdf

      3M (competitors to Halon systems): https://multimedia.3m.com/mws/media/1180481O/clean-agent-system-noise-hard-disk-drive-hdd-failure-faqs.pdf

      I seem to remember reading an article somewhere (maybe even on TheReg) about someone (re)designing fire supression output baffles and also having design software to optimise their placement within DCs to eliminate the risk to HDDs. After a quick search all I can find (not the article I'm thinking of) is this:

      https://solarfiresystems.com/news/silence-is-golden

    5. Doctor Syntax Silver badge

      Re: Missing a few key points here

      "They are all still on DR, see Channel 5 with the white and black square in the right hand corner"

      Ah, that explains it. I thought it was just another bit of channel branding.

  3. phuzz Silver badge
    Pint

    Oof, this probably hasn't been a fun few weeks for the sys-admins, engineers, and various sundry dogsbodies at C4 and Red Bee.

    We've all screwed up in prod before, but having your problems as publicly visible as a TV channel must be terrifying.

    I'll save one of these for them when they finally get it fixed>>>>>

    1. Anonymous Coward
      Anonymous Coward

      I had a trip up the BT tower a few years ago and met a fairly senior bloke from BT whilst there. He was very chatty and told me loads although some questions were left unanswered because of security etc. I didn't get to see the War Room in the basement but I did ask if I could. He told me that there's basically very little going out of the top of the tower these days. Useful as a place to impress people though and the view is indeed very good. There is the BT International Media Centre there and he said they carry a large amount of TV through there. He said if it had a problem you'd doubtless notice*, also said that they had redundancy, backups etc. Most interestingly to me he said if they were directed to do so by the government in a major emergency they could and would switch all of those over to a government feed. Whether that feed was coming from the "Defence Crisis Management Centre" or somewhere else he wouldn't say.

      Then some people** did notice when the X Factor went off air. I wondered if they really did have backups and redundancy etc.

      **Not me though.

      1. Test Man

        "He told me that there's basically very little going out of the top of the tower these days. "

        Yeah I heard that a lot of those satellite dishes don't work any more (maybe even all of them) but they can't be removed because the whole building is Grade-something listed.

        1. Somerset

          They are not satellite dishes! Point-to-point microwaves, rarely used now.

    2. Mr Neutron

      We've been here before

      Gawd knows how long Red Bee would take to respond & recover if given a four minute warning of nuclear attack.

      As I understand it, twas the VIBRATION from their ancient “disaster recovery" system (an oxymoron) that caused it!

      1. JimboSmith Silver badge

        Re: We've been here before

        Gawd knows how long Red Bee would take to respond & recover if given a four minute warning of nuclear attack.

        There was an actual plan sort of involving commercial broadcasters for the Attack Warning Red message. Well the commercial broadcasting transmitters anyhow. This was when the transmitters were all owned by the Independent Broadcast Authority. I was told that........All the feeds to transmitters would be switched to a feed from the BBC and the warning message would be broadcast simultaneously. Commercial broadcasters had no part in the plans once it was confirmed that the missiles were on the way.

        The codeword that would have been used to confirm and authorise the "Attack Warning Red" message was "Falsetto" It would have been sent by either the Director of the UK Warning and Monitoring Organisation (UKWMO) at RAF High Wycombe bunker or their Deputy at the UKWMO Bunker near Preston.

  4. Tom Chiverton 1

    What's up with Channel 5 though ? Still has the "backup" black and white square marker nearly all the time.

  5. Fred Dibnah
    Unhappy

    I've no doubt the Tories will use this as another reason to privatise Channel 4.

  6. Ol'Peculier

    Meanwhile, here in the darkest North, thousands still don't have *any* TV or *not much* radio because of the Bilsdale mast fire...

    1. Tom Chiverton 1

      Their site says you should have got a voucher to pay for some sort of OTT hardware as a replacement.

      Is this not the case ?

      1. John Brown (no body) Silver badge

        Yes, it's a £50 voucher to spend at Currys. But since there's no over the ait transmission for those affected, the best they can do is by a Chromecast or whatever dongle and hope their broadband is up to it. Bear in mind that the people most likely to still be affected, ie not within sight of the temporary mast or the smaller repeaters, are likely to be in very rural locations so less likely to have decent broadband. And the there's the folk, especially older folks relying on TV and Radio who may not have BB at all. You can;t get a Freesat kit for £50, let alone pay for installation.

        1. Anonymous Coward
          Anonymous Coward

          I think I'd cut over to Freesat if I was in their position and of course if I had the possibility to place a sat dish in an appropriate place.

          1. John Brown (no body) Silver badge

            After I posted, I was remembering my many trips through the North Yorkshire Moors and remembered many of the small villages and farms are in valleys and may not even have the option for a satellite dish at all for TV.

  7. Anonymous Coward
    Anonymous Coward

    Testing failover.

    At great expense, the last system I designed had a proper, offsite, in a different datacentre, DR platform. Whilst it needed a little work to point to a different DB, it was a complete replica of Prod. And it was tested. Yearly.

    Getting it approved was a nightmare. "Oh, can't you just rebuild it?" "Of course, me and, ahem, which army?"

    But it got built when I decided to turn my phone off one weekend when we had a small outage

    So, I ask again, was DR ever tested? I'm guessing that's a big fat NO

    1. Vulch

      Re: Testing failover.

      When I were a mere lad working in broadcast TV, "Pebble Mill at One" was essentially a regular and frequent DR test. Pebble Mill in Birmingham was the alternative network control for the BBC if TV Centre went down in a sufficiently spectacular manner, and PM@1 was used to check things could be fed directly to the transmitter network without involving London.

      1. Neil Barnes Silver badge

        Re: Testing failover.

        Twelve or fifteen years ago, when the BBC rebuilt most of its radio playout systems, A studio had a local server which cached the audio from the apps room servers. The apps room had three servers using two different OSes; all these ran at once to deliver audio. In most cases there were two identical apps rooms, at opposite sides of a broadcast centre, on separate power and network systems and fed from external circuits that didn't share ducts...

        I recall driving up the M1/M6 to Birmingham with *all* of Radio 2's music on a server in the back of a rented van; significantly faster than transferring it electronically (though a couple of years later we re-engineered the delivery backbones with 64Gb/s circuits).

        1. Doctor Syntax Silver badge

          Re: Testing failover.

          "significantly faster than transferring it electronically"

          That's what the old adage says. Nice to hear of it implemented.

          1. Neil Barnes Silver badge

            Re: Testing failover.

            Probably still quicker than network, even if we could have had the complete network just for this task. But other things have to happen, and the bandwidth budget was not great... and the BBC kept (keeps?) audio uncompressed, for obvious reasons.

      2. Anonymous Coward
        Anonymous Coward

        Re: Testing failover.

        I remember 'DR rehearsals' when we would set off from TVC in a load of cars for BM to try and create BBC1 and 2 from there. It was quite fun meeting up with colleagues we had not seen for months but I don't remember it ever going that smoothly trying to use a studio setup to playout TV channels.

        And then BBC DR moved to Elstree.....and now it is either Salford or W12 depending on stuff.

      3. chriskno

        Re: Testing failover.

        Remember the Pebble Mill at 1 presenter, Bob Langley? I bought his house in the Lake District from him 17 years ago. Still here, lovely spot.

        1. Doctor Syntax Silver badge

          Re: Testing failover.

          Wasn't it he who interviewed Ginger Rodgers and said it was always his ambition to dance with her? The look on her face suggested it wasn't one she appreciated.

    2. /\/\j17

      Re: Testing failover.

      Testing failover, or testing failover scenarios?

      I worked for C4 back in the 00's when failover from the main broadcasting control in Horseferry Road to the DR at I think Old Ford Lock was tested regularly and always worked. The failover process was for someone at HFR to switch control to OFL (I picture a huge blade switch but probably just a little, red button). As soon as that was done you had 30min or so to get someone to the desk in OFL and queue up the next set of programmes.

      What wasn't tested though were different failover scenarios, which resulted in the channel going off air for a good few hours despite a perfectly good, fully functional failover solution. From memory builders on the site next door cracked a gas main so HFR and the other surrounding buildings were told to evacuate, which HFR did by pulling the fire alarm. Assuming it was either another drill or false alarm everyone trooped out and stood on the street. And stood. And stood. And started looking nervously at watches and sent someone scurrying to OFL. And started asking the fire brigade if they could go back in and press the failover button (and being politely told to feck off). So stood while the programme queue ran out and someone sat at the desk in OFL watched the channel go off air unable to do anything about it because OFL couldn't TAKE control, it could only be GIVEN control.

      Doh!

  8. jollyboyspecial Silver badge

    People rabbiting on about whether or not a fire suppression system would (or indeed should) damage hardware are missing the point. This is a classic case of a lack of risk management leading to an error in egg location. That is to say that there are a number of catastrophic failures that can occur to a given basket (fire, flooding and overzealous fire suppression being only three). As such it's always best to have some hot or even cold standby eggs in another basket located as far from the first basket as is practical.

    1. Man inna barrel

      A classic example of the eggs in basket problem was a fire in a data centre in London causing Plusnet to drop off the planet. Not only did the internet service disappear, but the helpline went dead as well, because all services were routed through the same data centre. They could not even put out a recorded message when you phoned up.

      1. This post has been deleted by its author

      2. Lazlo Woodbine

        Remember Demon Internet

        They were clever, they had two separate lines from two separate telcos coming into their data centre

        All good, except they entered the building at the same spot, the exact spot where a van crashed into the building and took them out for about a week...

  9. Quando

    Audio level change?

    Anyone else noticed that the audio level of C4 HD seems to have changed too? Now a few levels below the BBC 1/2 HD level - find myself having to change as I switch channels when I didn't need to a few weeks ago.

    1. Coastal cutie

      Re: Audio level change?

      I thought it was just me - I was watching Bake Off last night with the volume turned up to a number that would usually ensure the whole road could hear, just to get the sound to reach the sofa.

  10. DS999 Silver badge

    Not suprising

    Even if they have tested their DR they can't possibly test everything. If you don't have any hearing impaired people involved with the testing, no one is going to think about testing the subtitling.

    You can be sure their ability to deliver ads during programming breaks was thoroughly tested though!

  11. Brewster's Angle Grinder Silver badge
    Boffin

    "Keep up the pressure."

    Yup, BECAUSE SHOUTING AT STRESSED ENGINEERS DESPERATELY TRYING TO FIX THINGS OR WAITING FOR DELIVERY OF PARTS ALWAYS SPEEDS UP REPAIRS.

    1. Zack Mollusc

      Keep up the pressure

      The stressed engineers will get no thanks, extra pay or resources no matter what, so keeping up the pressure by slagging off their management as clueless overpaid useless fat bastards on all possible public platforms is the way to go.

    2. Anonymous Coward
      Anonymous Coward

      “We need an update to the incident management team every 5minutes”

      Yes, because that will ensure I’m able to concentrate on the problem you useless fuckwit!

      1. Somerset

        Not 'we need to know when it will be fixed'?

  12. Anonymous Coward
    Anonymous Coward

    C4 DR

    When I used to work for Ree Bee as a techy engineering type, C4's 'old' DR in Camden was quite basic and the most important thing was to ensure it had all the programmes and ads delivered to it (file transfers). We tested it each week but it could only provide a cut down version of full programming and it was all SD only (IIRC) - I don't remember if it ever got put to air. They built a new one a few years ago but I don't remember it being staffed as such and again I think it was pretty much just a 'backup'

    The BBC (now) have a much better system with 2 playout centres which are both permanently in use - each doing a couple of channels but able to take the full load of all the channels if the other one keels over.

    I remember being told a story that many years ago some of my IT colleagues visited a bank and were shown how they had 2 identical systems which they frequently switched from one to the other to ensure both were working. One thing I always hated was identical systems being labelled 'main' and 'reserve' - people seemed terrified of using reserves thinking they were not as good and so they would not get tested in anger. X and Y or similar are far better.

    1. Anonymous Coward
      Anonymous Coward

      Re: C4 DR

      I don’t understand why this doesn’t happen more. Two systems and failing between them on a regular basis.

      1. Doctor Syntax Silver badge

        Re: C4 DR

        "failing between them"

        That form of words could be why not. It sounds too scary to manglement.

  13. Licenced_Radio_Nerd
    FAIL

    Not just C4/C5

    Viewers of FTA/Sky Satellite may have noticed that several of the usual music video channels are now only showing 'The Box'. Another casualty of the media centre failure - although sadly it has not taken out the Xmas video channel that started in September!

  14. Doctor Syntax Silver badge

    "The system works by discharging large volumes of nitrogen gas to displace oxygen and extinguish any potential fire"

    The reason for extinguishing the fire is to prevent the damage the fire causes.

    The operation was a success but the patient died.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like