back to article Techies try to bypass damaged UPS, send 380V into air traffic system

While weather and workforce strikes have affected air travel in Europe and the US this holiday season, Southeast Asia experienced disruption of its own due to a New Year's Day power outage at Manila's Ninoy Aquino International Airport (NAIA) that shut down both flights and airspace. "At around 9:49 am local time, the Air …

  1. elsergiovolador Silver badge

    Critical

    That power surge occurred when technicians tried to bypass the damaged UPS and sent 380 volts into the system, instead of the intended 220 volts, thus frying the terminals that receive satellite data from airplanes and air traffic management systems.

    One could think that such a critical infrastructure would have additional circuits (sometimes being just a fuse) preventing any kind of frying in case a human decides to play games?

    edit:

    (sometimes being just a fuse)

    That actually reminds me that I sometimes buy broken things on eBay to see if I can fix them. Very common thing is that the device has a fault and blow a fuse, so people tired of exchanging fuses just put a wire across and then device eventually gets fried. So for such a critical infrastructure, the fuse needs to be built in such a way, so that a human feeling clever wouldn't be able to easily install a wire to bypass it.

    1. Giles C Silver badge

      Re: Critical

      If they sent 380v into a 220v device that must mean they connected two of the three phases one to live and the other to neutral.

      Was this a qualified electrician and if so in what?

      To be honest I thought it was a who me a day late not a new story when I read it.

      1. Gene Cash Silver badge

        Re: Critical

        I'm sure he was. I've seen quite a few electricians (and other "professionals") that had tons of qualifications.

        Heck, they even had the NEMA 1-day "don't stick your fingers in the socket" course!

        1. Anonymous Coward
          Anonymous Coward

          Re: Critical

          That wasn't my finger!

          1. Claptrap314 Silver badge
            Coat

            Re: Critical

            But was it your dog? Asking for a friend...

      2. Winkypop Silver badge
        Alert

        Re: Critical

        “Was this a qualified electrician and if so in what?”

        Worked with such an electrician.

        His idea of fault finding was to put a large nail in place of the fuse and then go looking for the magic smoke.

        1. Anonymous South African Coward Bronze badge

          Re: Critical

          Instead of a nail, he can use an audiovisual alert fuse as an added safety precaution.

          Here's a handy guide on fuse replacements and their capacities :

          https://i.imgur.com/WrrEw1P.jpg

          1. blackcat Silver badge

            Re: Critical

            Long ago my grandad worked at a power station in Manchester. (back when they had coal fired power stations IN cities...)

            The power station fed some buildings with DC for the likes of lift motors. They had 2 generators for the DC. The normal one and an old one that was used for fault finding. Every so often the wire under the roads would break or just go bad so they would get the customer to disconnect at the far end, hook up the old generator, send some guys out into the street and wind it up until something flashed over. Smoke coming out from between the cobbles would indicate where to dig.

      3. Stork Silver badge

        Re: Critical

        That was done to our cooker by a qualified electrician in Denmark. It is common to have 3 phase sockets, and he swapped common and a phase.

        Cooker heated extremely well, but the fan in the oven gave up and the light was blinding.

        1. DoctorPaul

          Re: Critical

          Similar happened to me some decades ago, I was already doing WFH in the 80s so luckily was at home when it happened.

          It being water ingress to the wiring in the street which I think put another live phase on to my neutral. All I know is that the whole house started to hum and every standby light lit up like a Christmas tree. Cue a sprint to the fuse box (no fuses blew) to throw the main switch, if the house had been unoccupied I shudder to think what would have happened to the wiring.

          1. Stoneshop
            Boffin

            Re: Critical

            It being water ingress to the wiring in the street which I think put another live phase on to my neutral.

            Neutral wire upstream of your house or a couple of houses goes fzzzzrk, and the neutral voltage downstream of that break is now at the mercy of the load between each of the phases and neutral.

            Which is rarely sufficiently balanced that neutral is still more or less neutral.

      4. Anonymous Coward
        Anonymous Coward

        Re: Critical

        I suspect someone must have accidentally disconnected the neutral, and that gets VERY ugly with 3 phase power circuits. Well, QED, must have been quite an 'oops' moment for whoever did that. It's understandable, though, I can imagine the panic because none of the emergency power facilities worked - it's easy to laugh at this from the sidelines..

      5. 105kayem

        Re: Critical

        Many years ago a company that I worked for had a contract to install 3 big industrial sewage pumps in a new pumping station. The sparky made an impeccable job of it it, all the cable runs to the control boards were symmetrical and neat. Come switch on none of the pumps would run, due to him running the red phase to the first pump, the blue phase to the second…

    2. DJO Silver badge

      Re: Critical

      Not sure if a fuse provides protection for over-voltage. By the time the current increases to the kill-fuse level, the damage may already be done.

      I'm not arguing against fuses, they are a vital part of any protection regime just not sure if they would have helped much in this circumstance.

      1. Kevin McMurtrie Silver badge

        Re: Critical

        There's a big gap between the working voltage and the point where surge suppressors are fully active. 380V into a 220V system is in that gap. Everything should tolerate it just fine for several seconds. Then capacitors pop, MOVs protectors smoke, and the fuses finally trip from something permanently shorting.

        600V to 4kV would have been simple blown fuses.

        Here in 'Merica, elevated power lines are ordered from lowest voltage on the bottom to highest voltage on top. It makes tree trimming and comms repairs safe. If that top line falls into the lower lines, everything is toast. There will be a plasma fire in a large void where your protection components once lived.

        1. NeilPost Silver badge

          Re: Critical

          Respect to Hannah Gadsby….. it’s “Umerica”??

    3. 080

      Re: Critical

      Fuses won't stop frying, especially of the over-voltage variety

      1. elsergiovolador Silver badge

        Re: Critical

        It really depends on many circumstances and it was just an example.

        For instance, I managed to plug in US made device into UK mains (it only had IEC connector, labels worn off and I was under impression this was a domestic product) and only thing that fried was the internal fuse. After replacing the fuse and getting a step down converter, the device worked fine.

        1. Orv Silver badge

          Re: Critical

          Power supplies for some devices have a circuit called a "crowbar" that's intended to blow the fuse in situations like that. The idea is if the voltage is too high, the crowbar triggers and creates a direct short across the output, both preventing any voltage reaching the load and blowing the fuse.

        2. andygrace

          Re: Critical

          Sure and plugging a device which is looking for approx 110V AC into 230/240V AC won't usually have too much of an impact.

          The inverse is definitely not case although if the product is designed correctly, the only damage should be the fuse and maybe the Class Y capacitor across the mains and possibly some other passives or a bridge rectifier/switching transistor if things are really grim.

          Passing three phase ie 380V into a 220V single phase input will cause "bad stuff" every single time.

          1. Tim Cockburn

            Re: Critical

            Voltage. Back in the 70s Istanbul had different voltages in different parts of the city. Moving from the centre to the suburbs I found a 110 volt lightbulb made a photoflood for a brief time.

        3. Anonymous Coward
          Anonymous Coward

          Re: Critical

          After replacing the fuse and getting a step down converter, the device worked fine.

          You're lucky. Traditionally, the device protects the fuse by blowing first :)

    4. Arthur the cat Silver badge

      Re: Critical

      sent 380 volts into the system, instead of the intended 220 volts

      There's a German sysadmin on the FreeBSD mailing list who has a horror story of a backup generator system being wired up exceedingly wrongly so when deployed it put kilovolts through the data racks. I hate to think what that smelled like(*).

      (*) Probably not roast pork as no organic casualties were mentioned.

      1. Dimmer Bronze badge

        Re: Critical

        Even when wired correctly, a generator with a failed regulator will output too much voltage. I say this by experience.

        Running 3ph 480v 60hz system into an online UPS, My generator outputted 560v due to the regulator adjustment going bad.

        Breaker tripped due to 480V 3ph surge suppressors wired in the generator panel causing the generator feed to be disconnected. I have them on the generator feeds, the cooling tower feeds (for lightning strikes) and both of the power co. feeds and downstream in every cabinet. I have a low trust factor of power components and it has served me well. I also sleep better because I don't have a bean counter telling me what redundancy or safety equipment I can't buy.

        1. Anonymous Coward
          Anonymous Coward

          Re: Critical

          [I must have mentioned this before at some point]

          I was with a colleague fitting a mooring line tension management system on an oil rig going through a refit. I got flack from my boss for spending money on optical insulators for the sensing wires (the usual 3..20mA current loop idea) when building it as it increased our cost. We had another company providing a system for weather data from which we also took a feed (mainly wind direction and speed), and they finished installing first. We needed another day as our system had a couple of huge screens which took some time cabling up.

          This was in the days of Intel's 80386/80486 so when they switched on it took a minute for harddisks to spin up, the BIOS to pick up bootloader, start DOS and kickstart whatever they had in AUTOEXEC.BAT, and then the relays on the acquisition card kicked in - resulting in a Hollywood worthy massive flash, a very loud BANG, an impressively bulging system case and a lot of smoke. This was in the days of cases that were made of proper steel so that had taken quite a bit of force. It had not been built in yet as it was a test run, but I'm glad it at least held together because I don't know where the lid would have ended up otherwise.

          After the excitement (and smoke) had died down it was time to work out what had happened. Turns out that the sensor data they got from the generator (which theirs was also tracking) wasn't the mA loop they were expecting, due to a wiring mistake they got a direct generator feed of some 500V which upset their directly connected (read: not optically insulated) PC in a truly spectaculair fashion - I suspect every electrolytic capacitor in that machine must have turned into an explosive in about a millisecond.

          There were no further complaints about those optical insulators :).

    5. Phil O'Sophical Silver badge
      Mushroom

      Re: Critical

      Modern electronics will fail faster than a fuse, especially with "only" 50% excess voltage.

      It used to be said that a transistor was "the fastest fuse on three legs". Still true when there are millions of them in a chip.

    6. MachDiamond Silver badge

      Re: Critical

      " Very common thing is that the device has a fault and blow a fuse, so people tired of exchanging fuses just put a wire across and then device eventually gets fried."

      I buy things on eBay to fix too but I can say that if somebody connects something set up for 120v across 240v, a fuse might not act fast enough. Especially if said fuse is a blown fuse with foil wrapped around it or the largest capacity fuse that would fit. I had an amplifier come in like that when I worked at a service depot. The main transformer was toast and at that point, not worth looking any further.

      Full disclosure, I've substituted wire for a fuse before. I can remember one case where I had repaired the device but didn't have a fuse on hand at all and nothing I could find to rob. The show started in 20 minutes with no Radio Shack around (yeah, that long ago). I did remedy the situation as soon as I could get parts. Like a ship's XO, I kept notes on all of the things that needed looking into and always had a shopping list going.

      1. DJO Silver badge

        Re: Critical

        ...people tired of exchanging fuses just put a wire across...

        That's for amateurs, professionals know that the shaft from a dismantled 1/4" jack plug is exactly the same size as a fuse and will never blow.

        Don't ask.

        1. Potty Professor
          Boffin

          Re: Critical

          A 1/4" brass bolt will serve as well, with the additional plus that the thread will act as cooling fins.

        2. MachDiamond Silver badge

          Re: Critical

          "professionals know that the shaft from a dismantled 1/4" jack plug is exactly the same size as a fuse"

          Common knowledge for anybody that's been a member of the Federation of Allied Road Technicians or the Combined Union of Non-Theatrical Stagehands.

    7. GloriousVictoryForThePeople

      Re: Critical

      So for such a critical infrastructure, the fuse needs to be built in such a way, so that a human feeling clever wouldn't be able to easily install a wire to bypass it.

      You aren't familiar with Mr Darwin's theories are you?

  2. Anonymous Coward
    Anonymous Coward

    Future who me?

    1. Mark 85

      Or maybe this just didn't get labeled as such. Holidays and all that.

      1. John Brown (no body) Silver badge

        Nah, Who Me? and On Call are usually historical, not "breaking news" in recent days :-)

    2. Anonymous Coward
      Anonymous Coward

      Had to admit, I did click on the story thinking it was this week's Who me? story

  3. Anonymous Coward
    Anonymous Coward

    major fuck-up I reckon

    Major UPS screw-up, by all means, but the one below freaked me out !

    " ... ordered emergency response teams to implement irregular operations procedures, ..."

    Like, REALLY ? I'd like to know exactly what those were, or should I just make sure I never approach PH in a plane at all ?

    1. Yet Another Anonymous coward Silver badge

      Re: major fuck-up I reckon

      The procedures for when operations aren't operating as they regularly would

    2. dl1jph

      Re: major fuck-up I reckon

      As a matter of fact, that sentence should give you some confidence that they know what they're doing. They fully understand that any system can fail and they've come up with a plan to keep everyone safe regardless. After all, that's what makes aviation as safe as it is - there are backups for everything and backups to the backups for anything critical.

      Exactly what the procedures are depends on precisely what's broken, but "irregular operations procedures" exist for any situation where the kit ATC should have isn't fully operational. Usually it's along the lines of "back to pen and paper", but in a worst case scenario it can be "make sure everyone knows we're down to visual separation and onboard collision avoidance systems for all aircraft". These procedures exist in every country and will be drilled somewhat frequently.

      1. Anonymous Coward
        Anonymous Coward

        Re: major fuck-up I reckon

        This is why aviation is still safe - even the backup procedures have their own backup.

        If anyone wants to have an idea of just what an achievement that represents I would recommend you have a look at just what is simultaneously in the air at any given time on flightradar24.com. It is truly staggering.

  4. b0llchit Silver badge
    Flame

    Get it while still moving

    If our Group could be of any help to DOTr/CAAP, we’d be happy to participate...

    Wow!

    Translation: if we can earn a buck or two from this disaster, then we really want to be first in line to fill our bank accounts.

    Picking on corpses while they are still slightly alive. That was a fast response. Never let a good disaster go to waste to earn another boat load of monetary exchange leaflets.

  5. blackcat Silver badge

    Been there but not the one who caused the fault

    A few years back the company I worked for had finally decided to replace the batteries in the 3 phase UPS. Whoever installed it had done a bit of a 'quick and dirty' with the wiring and had not left much in the way of slack. The door that allowed access to the batteries was partially blocked by the output cabling but the service people could just about get the access they needed without the need to power down the server room.

    About half way through the job the tension on the output wiring resulted in the neutral cable coming out of its terminal and falling against one of the live phase outputs. Amazingly only a couple of bits of kit let the smoke out before the monitoring shut the whole room off. Unfortunately some of the PSUs to the out of support but still critical SAN let the smoke out and it would not come back to life with the remaining PSUs. We managed to get some spares next day (phew!) and got pretty much everything up and running within 24 hours.

    It took a few years off the IT managers lifespan though!

  6. Anonymous Coward
    Joke

    Aren't many APC UPS models made in the Philippines?

    This incident doesn't shine a good light on local management of such devices.

    1. david 12 Silver badge

      Re: Aren't many APC UPS models made in the Philippines?

      Does APC make rotating-machinery live diesel gen-sets? I think not.

      1. Anonymous Coward
        Anonymous Coward

        Re: Aren't many APC UPS models made in the Philippines?

        You missed the joke icon - still two UPS sets failed, accordingly to the article.

  7. ComputerSays_noAbsolutelyNo Silver badge
    Boffin

    Don't worry, be wiring.

    When my university extended its campus, there was a stuff-frying incident with the electrical installation. A normal outlet was wrongly wired, thus delivering 380V instead of 220V.

    It's so easy to make such a mistake, there are 4 or 5 wires in a cable: 3 phases, a neutral and ground/earth.

    If you connect a phase and the neutral to a normal power socket, it delivers 220V.

    If you, however, use two of the phase-wires, the outlet delivers 380V.

    Normally, there is some color-coding of the wires to avoid this type of mistake, but stuff gets stuffed up nonetheless.

    1. Potemkine! Silver badge
      Trollface

      Re: Don't worry, be wiring.

      Note also that one should test the output when connecting a new socket, just to be sure. In that case, a multimeter can help.

      1. Paul Crawford Silver badge

        Re: Don't worry, be wiring.

        Multimeters are not a good choice, too many ways you can get it wrong with potentially disastrous results resulting in an explosion (e.g. trying amps range to measure volts on a high energy circuit) or a deadly shock from checking for AC volts on DC range and seeing nothing then touching it.

        Meters should be CAT-IV rated if you do want to use them, but a "voltage tester" is a simpler and safer option. As a random example:

        https://www.test-meter.co.uk/martindale-vt12-voltage-and-continuity-tester

        1. An_Old_Dog Silver badge

          Minimum Required Competece and Care?

          Multimeters are not a good choice, too many ways you can get it wrong with potentially disastrous results

          If you're not sufficiently-educated, sufficiently-experienced, sufficiently-imaginative ("what could go wrong here?") and sufficiently-careful, you should not be messing about with mains power, automobile engines, or chemistry.

          Dumbing-down the test device to a McDonalds-style touch screen menu won't prevent the ignorant or foolish from meeting an eventual Bad End.

        2. cyberdemon Silver badge
          Trollface

          Re: Don't worry, be wiring.

          > Multimeters are not a good choice, ... (e.g. trying amps range to measure volts on a high energy circuit)

          That's what we have Darwin Awards for.

        3. Missing Semicolon Silver badge

          Re: Don't worry, be wiring.

          Don't most meters have:

          * A separate socket for the current range,

          * an input alert if you have got the red lead in the current socket, but selected "Volts"?

          1. Paul Crawford Silver badge

            Re: Don't worry, be wiring.

            Mistakes still happen. You might want to read the section "Hispanic Factory Workers Dies of Burns After Improperly Testing a 480-Volt Electrical Bus Bar." starting on PDF page 68 (page 53 on bottom of page) of this report:

            https://www.nfpa.org/-/media/Files/News-and-Research/Fire-statistics-and-reports/Electrical/RFArcFlashOccData.ashx?la=en

            1. Orv Silver badge

              Re: Don't worry, be wiring.

              Also worth highlighting that 480 volts RMS implies a peak voltage of 691 volts, which is above the insulation rating for most consumer-grade meters.

              1. Paul Crawford Silver badge

                Re: Don't worry, be wiring.

                Many consumer / cheap multi-meters claim to be rated to measure 1,000V but not with any switching spikes on top. But the usual reason for such a nasty end is having it on the wrong range (ohms, amps, low voltages, etc) and such cheapo meters are not able to safely fail, they simply explode.

                That is why the CAT-III/CAT-IV ratings are so important, they specify certain expected transient over-voltages to be survived, and that they fail in a safe manner if anything else happens (so a combination of decent parts used, adequate PCB clearances, and High Rupture Capacity fuses as needed).

                Not your 20mm x 5mm glass fuses, but proper ceramic HRC things that are often 38mm x 10mm and can interrupt prospective fault currents of 50kA or more.

    2. MachDiamond Silver badge

      Re: Don't worry, be wiring.

      "It's so easy to make such a mistake, there are 4 or 5 wires in a cable: 3 phases, a neutral and ground/earth.

      "

      Sure, one can make that mistake. Not going around and verifying your work afterwards with a tester that only costs a few bob is criminal. I worked for a large sound company and our power feeds were all separate lines with CamLock connectors. The only person that was allowed to connect power to the distribution rack was the owner of the company and he only did so after measuring everything first for the third time. Woe would be him who presented the system as ready to connect with a wrong feed. The main issue was that the power feeder cables would have been likely to have been tied into a panel by a union elechicken and they just really didn't give a toss.

      1. Orv Silver badge

        Re: Don't worry, be wiring.

        Once a company I worked at had a prototype built into a shipping container they were hooking up for a demo at a military base. Our field tech thought to check the power feed, and found that the military wires their 480V camlock connectors differently. Differently enough that had we just plugged it in, we'd have put 480V on the skin of the container.

    3. Orv Silver badge

      Re: Don't worry, be wiring.

      A wild-leg three-phase system also gives you some exciting ways to screw up. These systems use a transformer with one center-tapped phase, with the center tap tied to the neutral. With a 240V transformer this gives you 120V to neutral on two phases, but 208V on the third ("wild") phase.

      1. Paul Crawford Silver badge

        Re: Don't worry, be wiring.

        You don't really see that in the UK/EU, we are almost always LV star (wye) secondaries and HV delta primaries for end application substations. But the loss of N or swapping N for one of the L is just as exiting and easy to do...

        1. Orv Silver badge

          Re: Don't worry, be wiring.

          Makes sense, I think "wild leg" setups are pretty unique to North America and its 120/240-volt "split phase" power system.

          Another exciting situation here is when you lose the neutral connection. Now loads on half the 120V circuits are in series with the other half across 240V, and the voltage will rise or fall depending on which half is more heavily loaded.

          I used to regard split-phase as something of a kludge compared to the UK 240V system, but I've come to appreciate it; delivering 240V to high-power appliances and 120V to everything else, with no conductor being more than 120V above ground potential is actually quite clever.

          1. Paul Crawford Silver badge

            Re: Don't worry, be wiring.

            It has its plus and minuses if you pardon the pun. While safer, you can still get killed on 120V to Earth and probably more fires from more amps for a given kW load. In the UK (at least) we have 110V for building sites (using yellow "commando" connectors that are not interchangeable with the blue 230V or red 400V ones) and that is really 55V - 0V - 55V two-phase in a similar manner to the USA's domestic supply, but half the voltage. In the days before cheap RCDs that was a major step towards safety as what passes for wiring in building sites is appalling beyond belief.

            While rare, you also see 230V - 0V - 230V systems in the UK for the odd farm, but generally speaking if you need more than 230V single phase worth of power then it is 230V/400V three-phase you get. There are also LV sites on 400V/690V three-phase but personally I have never encountered one, I think that is only for big motors but not quite going to the 3.3kV sort of voltage that comes under different rules (as above 1kV AC is classed as "high voltage" and not under the usual wiring regulations).

  8. Will Godfrey Silver badge
    Facepalm

    No Excuses

    If you know there's three phase around you should always check with a suitable test meter. If you 'know' there isn't you should still check in case some numpty has been messing around. Also, especially check neutral continuity. The most lethal situation is a broken neutral that's just hanging there looking innocent because there's a star connected motor on it.

    1. Gene Cash Silver badge

      Re: No Excuses

      Or as a wise man once sad: "If God tells you it's locked out, double check with Jesus, then test it dead yourself."

      1. Anonymous Coward
        Anonymous Coward

        Re: No Excuses

        Would that be the father, son then holy ghost? But I'm trying to avoid playing the part of ghost.

      2. Press any key

        Re: No Excuses

        It's only locked out if you put the lock on yourself and you have the only key to the lock.

        1. MachDiamond Silver badge

          Re: No Excuses

          "It's only locked out if you put the lock on yourself and you have the only key to the lock."

          Until some numbty with a biz degree thinks it would be a good idea to have duplicates on hand of all the lock-out keys.

          The railroads ran into that sort of problem with track tokens. On single line track, a driver couldn't proceed without being in physical possession of the proper token. Some d-head thought that only having one could lead to slow downs and had duplicates made. A couple of head on collisions later..........

        2. A.P. Veening Silver badge

          Re: No Excuses

          And even in that case there is no guarantee, too many lockpickers around.

          1. Hubert Cumberdale Silver badge

            Re: No Excuses

            To be fair, you can probably even shim most of them quicker than you can use the key.

  9. Anonymous Coward
    Anonymous Coward

    Strangers in the night

    There's no power, what do we do?

    747 coming down in the night

    Tryin' to get a message though

  10. This post has been deleted by its author

    1. Korev Silver badge
      Coat

      Re: What The People are Saying

      > “Maybe now they will consider moving to cloud.evil. Isn’t that right Mr. Bigglesworth?” Dr. Evil, Cloud CTO

      Well on-prem had too much Austin Powers going to it

    2. Doctor Evil

      Re: What The People are Saying

      “Maybe now they will consider moving to cloud.evil. Isn’t that right Mr. Bigglesworth?” Dr. Evil, Cloud CTO

      I cannot recall saying that, but it does sound like me.

  11. phuzz Silver badge
    Alert

    A couple of weeks ago in Bristol, National Grid somehow ended up raising the mains voltage to 266V. Incredibly, basically everything was fine and kept on working as normal.

    We only noticed because the UPS feeding our comms rack was cheap and just passed on the 266V to the transfer switch. The transfer switch decided that 266 was too high and just turned off, so we only noticed when we lost internet access. The other UPS's coped just fine and conditioned the power, one recorded the initial spike at 272V! (Frequency didn't seem to drift any more than it usually does over a day).

    If you'd asked me before, I'd have assumed that most consumer equipment would start to die past 260V, but everything from desktop computers to lightbulbs just shrugged it off.

    1. Disgusted Of Tunbridge Wells Silver badge
      Mushroom

      UK electronics have to tolerate 253v, don't they? 260 isn't that far off.

      1. Phones Sheridan Silver badge

        For the UK that's correct, " In the UK, the declared voltage and tolerance for an electricity supply is 230 volts -6%, +10%. This gives an allowed voltage range of 216.2 volts to 253.0 volts." ( https://www.spenergynetworks.co.uk/pages/voltage_changes.aspx )

        I've rarely come across equipment, that can't tolerate 240 +/- 10% though, the few that didn't tended to be cheap chinese imported power supplies.

        1. Sam not the Viking Silver badge

          Mains Voltage

          I reminded our Electrical Engineering Graduate (alleged) that 240 volts a.c. on the mains is the RMS volts.... Peak volts is rather higher.... It is fortunate that equipment/devices etc. are designed to cope with this. They are, aren't they.......

          1. Roland6 Silver badge

            Re: Mains Voltage

            > It is fortunate that equipment/devices etc. are designed to cope with this. They are, aren't they.......

            They are when set correctly...

            I have killed several multimeters over the years because I had failed to rigorously apply the rule my father taught me as to what setting you should always set it to, to begin with, when dealing with an uncertain faulty circuit...

    2. Arthur the cat Silver badge
      Alien

      Years ago during winter, back when I lived in Norwich, the power supply started acting very strangely. The voltage was obviously cycling over a 30-40 second period, with the high voltage making all lights (incandescent in the house, sodium in the streets) become bright enough for me to worry they might burn out and then dropping so they were very dull and the mains power radio I was listening to shut off(*). This went on for about a quarter of an hour, enough to seem very spooky.

      I was a postdoc researcher at UEA in those days, with a side order of being one of the people responsible for the computer department's Vax and Prime computers. When I got in to work the next morning the Prime had shut itself off during the first brownout and was safe, but the Vax had been set to autoboot on power on so had turned on repeatedly in time for the over-voltage, and had ended up totally fried. The DEC engineers literally had to replace everything but the backplane.

      The National Grid blamed it on extreme icing on the 400kV lines running through East Anglia mixed with high winds causing the lines to swing resonantly, and said the consumer voltage had been fluctuating between 150 and 330 volts. We joked that it was actually a UFO trying to get a jump start for its FTL drive.

      (*) At which point I turned it off for safety.

      1. blackcat Silver badge

        I had something similar one weekend many moons ago where the village had lost an entire phase and my house was getting about 150-160V live to neutral. Anything that was SMPS powered and rated 110-240 was working just fine. Everything else was being very strange so out came the multimeter very promptly followed by the main breaker being flicked OFF.

        One of the 11kV lines had snapped at a T junction in such a way that it didn't fall to the ground and earth out but it wasn't connected any more.

        Right faff of an operation to fix it. Lots of calls to HQ, eventually a guy goes 1 pole down the line, shins up and earth bonds all 3 wires, goes up the line 1 pole and does the same, a unimog with a cherry picker comes along and 30 seconds later the line is repaired then followed by an hour of removing the earth bonds and more phone calls to re-energise the lines.

      2. DJV Silver badge

        Damn, that rings a bell that I had almost entirely forgotten about! As a Norwich resident (still!) I do remember a crazy evening when the mains went haywire. As someone who was into electronics, I had a multimeter and could see that the mains was fluctuating rather strangely. Was it late 70s/early 80s?

        1. Arthur the cat Silver badge

          Ancient and increasingly faulty memory here but winter 79/80 or 80/81.

          1. Roland6 Silver badge

            That would have been winter 80/81 - I remember delaying my drive back to Norwich (*) until the snow ploughs had cleared the A505 between Baldock and Royston...

            (*)I had only got my driving licence in the summer of '80.

      3. MachDiamond Silver badge

        There is a whole bunch of solar and wind near where I live and every so often it seems like one of those installations can't get synched to the line. My voltage gets buggered and I can see the phase all over the place as well. I've got constant line voltage transformers for my office, but I don't want to test how wild a swing they can handle so I'll shut things down.

      4. Orv Silver badge

        When the Great Northeast Power Outage of 2003 happened in the US, I was working for a company in Michigan. Our rack UPS logged some heavy voltage swings, both above and below normal, before the grid finally collapsed. Later I was able to roughly correlate my UPS log with the published chain of events, as different power lines tripped offline and created surpluses and shortages in different parts of the grid.

      5. Roland6 Silver badge

        The Prime and VAX at UEA dates you, I assume you were still in the Village and that the campus 1900 wasn't affected (or had this been removed by this time?)

        1. Korev Silver badge
          Windows

          If we're talking about Vaxen, then the chances are that The Village didn't exist at that point.

          Greybeard -->

        2. Arthur the cat Silver badge

          The Prime and VAX at UEA dates you, I assume you were still in the Village and that the campus 1900 wasn't affected (or had this been removed by this time?)

          Still in the Village(*) then, still a couple of 1900s in the main Computer Centre. I honestly can't remember what happened to the CC as my team (Robin Forrest's lot) were pretty independent of it at that point.

          (*) I am not a number, I'm a free variable(**).

          (**) Well, I was in Computer Science.

          1. Roland6 Silver badge

            >I'm a free variable(**).

            (**) Well, I was in Computer Science.

            Given you were at UEA "Do Different" suspect you were probably more of a quantum variable...

    3. Citizen99

      Hmm. Any correlation with our Virgin cable telly going down for a few days?

    4. Anonymous Coward
      Anonymous Coward

      Reminds me of a support call I once had from a person visiting a third world country.

      "We had a power cut. Then when the power came back on, all the lights were twice as bright as usual. Then I plugged in my laptop....."

  12. Eclectic Man Silver badge
    Unhappy

    Carbon Footprint

    BBC Radio 4 is currently broadcasting a series on global warming and reducing one's carbon footprint. (Weekday mornings at 9:00 https://www.bbc.co.uk/sounds/play/m001gkq8 *). I wonder what the carbon footprint effect of the ATC outage will turn out to be. After all, if every flight into Manila and other airports in the Philippines had to land somewhere else, that is a lot of extra flights to catch up.

    *It seems we are allowed one short haul return flight every 3 years, and one long haul return flight every 8 years, if we want to reduce global warming to a maximum of 1.5 Celsius in time to stop a runaway greenhouse effect.

    1. Eclectic Man Silver badge
      Unhappy

      Re: Carbon Footprint

      I'd be interested to know the reasons for the (so far 5) downvotes. I only reported the statements of experts. Global warming exacerbated by burning hydrocarbons is an established scientific fact. So maybe you could provide a reason for your downvotes, please?

      1. Missing Semicolon Silver badge

        Re: Carbon Footprint

        Because BBC. A series at 9:00 moaning about Global Warming again sounds too close to a morning sermon to me. Especially as the BBC have been caught out telling multiple porkies in their climate coverage (misattribution, exaggeration, frankly making stuff up sometimes).

        1. Eclectic Man Silver badge

          Re: Carbon Footprint

          @Missing Semicolon. Thanks, but why downvote (now up to 6) without comment instead of explaining that in the first place?

          I'd be interested in references to the BBC 'making stuff up sometimes' about climate change, btw.

      2. Orv Silver badge

        Re: Carbon Footprint

        It's not going to be solved by individuals deciding to not go on vacation. The main drivers of climate change are industrial emitters, who would dearly love to pawn off responsibility onto the rest of us while continuing to rake in profits.

      3. AlbertH
        Mushroom

        Re: Carbon Footprint

        Simply because it's total nonsense. The infamous "97% of scientists agree" is a figure made up ONLY of workers for the IPCC. The IPCC's name alone tells you that the expect "Global Warming / Climate Change".

        There's a saying in both the Scientific Community and in the Media: "Is it true, or did you hear it on the BBC?"

        The BBC used to be the world's premier broadcaster, but since the mid 1980s has been an embarrassment!

  13. Anonymous Coward
    Anonymous Coward

    They should put this sort of thing in the cloud. That way it's closer to the aircraft.

  14. Paul Hovnanian Silver badge

    Very loose

    ... interpretation of the term "technician", IMO.

    The equipment needs a warning sign:

    ACHTUNG!

    Alles Lookenpeepers!​

    Das machinen ist nicht fur gefingerpoken und mittengraben.

    Ist easy poppenkorken, schnappen der springenwerk und blowenfusen mit spitzensparken.

    Ist nicht fur gewerken by das dummkopfen.

    Das rubbernecken sightseeren, bitte keepen das hands in das pockets, relaksen und watch das Blinkenlighten. ​

    Danke​

  15. An_Old_Dog Silver badge

    Testing, Testing, and More Testing

    Diesel-powered backup generators need:

    * Under/over-voltage startup- and protection-testing; * Diesel fuel replenshment schedule; * Diesel fuel water- and bio-sludge testing (and remediation, if contamination is found); * Switchgear testing.

    Both Diesel-powered and battery-powered backup systems need switch-in/switch-out testing, and run-time testing.

    1. MachDiamond Silver badge

      Re: Testing, Testing, and More Testing

      "Diesel-powered backup generators need:"

      They need AVE. I haven't seen if the genset saga had ever been concluded. He and dewclaw bought a borked genset cheap with a bad engine. The cause is suspected to be routine testing. The generator is fired up for a couple of minutes every month or so. It was never run long enough for the lubrication to get distributed nice and evenly throughout. Lots of shiny bits in the oil pan and disturbing banging inside.

      The generators need to be tested properly and the fuel cycled periodically to keep it fresh. I know there are stabilizers for fuel that's on standby, but it's likely close to the same cost to run the gennie to burn up some fuel and top up with new. Tanks can even be drained into the service truck's fuel tank and brand new fuel put in the tanks every couple of months.

      1. An_Old_Dog Silver badge

        Re: Testing, Testing, and More Testing

        Our data center runs weekly tests where the gennies take the full load for 1/2 day, every Sunday. Our Poison Control Center runs monthly tests where the gennie takes the full load for an entire day. The P.C.C. nurses' workstation desktops are set up so they do not need Internet connectivity for their relevant case-management, poison database, and pharmo-kinetics software to work.

      2. AlbertH

        Re: Testing, Testing, and More Testing

        We used to get our standby diesel fuel "polished*" periodically!

        * It's a thing: look it up!

  16. ricardian

    The electronic organ in the kirk where I'm the organist was bought new about 8 years ago. About 5 years ago it started "locking up" at random intervals (usually in the middle of a service) then "unlocking" after a few hours. I managed to get the manufacturers of the organ to admit that the problem was caused by the advent of local wind turbines & solar panels. Eventually the manufacturer replaced all the internal electronic panels and I've had no problems since then.

    1. Tim99 Silver badge

      So the manufacture had under-specced the power supply, or the (privatized?) power company supply was outside specification?

  17. Arthur Daily

    Airport Departure and Flight Taxes - were they imbezzled?

    The Philippines has some pretty stiff flight and departure taxes.Everything should have been covered. Philippines has excellent and cheap talent, so there are no excuses for simple foul-ups, or not saying why there was no redundancy. I know generators and UPS's are often falsely blamed, to avoid the real reason. Critical stuff also has power 'conditioners' like big expensive copper coil chokes, and for radar gas overvoltage banks to take a full lightening strike. Thus overvoltage sounds like a false excuse to me. Who signed off on the testing drills? No, this is a cover-up story. I suspect the batteries were never replaced or tested (along with the ups). A bigger than normal blackout happened, and someone unqualified (cheaper to employ you see) bypassed the GPS and power conditioners, did a direct connect, and the local power company on startup - let a surge in, along with 380-520v startup capacitors in every airconditioner and fan in the airport. (380*1.414= peak ac voltage of 518 volts, and if 3 phase) Oh dear! Speculation that the power conditioners were 1) sold off, 2) failed years ago, and just bypassed), that there was no UPS, and the dude* have no 3 phase experience. We look forward to the official report.

  18. J.G.Harston Silver badge

    In the '90s I changed flights at Manila, and accidently went through customs. It was one of their regular power shortages, and they had everything turned off except ATC and facilities "apron side". I walked into a wall of heat, It was like walking into a brick wall.

  19. Anonymous South African Coward Bronze badge
    Devil

    Simon would be so proud of that team.

  20. AliBear

    Thought this was going to be some 15-year-old Who? Me? story...

    Just think of all those warehouses full of shiny new UPSes made in the Philippines...

  21. Roland6 Silver badge

    A red faced way of forcing a systems upgrade up the agenda?

    > both the CAAP and Bautista recognized the CMS/ATM was already outdated before it was even fully operational in July 2019.

    Frying the systems with excess voltage will mean replacements are necessary rather than simply repairs...

  22. mistersaxon

    It's just a silly phase I'm going through...

    Worked in a building where they made several offices into one open plan. That was how they discovered that some of the sockets were wired on one phase and some on another, after connecting a PC to one phase and the printer to the other...

    But generally the worst electrical foul-ups I've come across have been where a UPS is the cause: one that decided to put 400V on the output, another that switched the power to the IBM mini off and on every 30 seconds and one that thought "hold my beer" and just flat out burst into flames - and that one was a room UPS. The fire brigade chopped the cables with a (heavily insulated) axe and pushed it out of the computer room into the yard because it was on wheels and they decided that was the easiest thing to do, and who am I to argue with a fireman holding an axe?

    If you're lucky your UPS will fail to a safe state but it's more luck than design I feel. (and yes I know I'm being very very selective and that UPSes have saved WAY more bacon than they have flame-grilled - including mine. And yet I still have nightmares about the 20A circuit breaker that jammed half-open when a bulb blew out and tripped it and it burst into flames in my garage.)

    1. blackcat Silver badge

      Re: It's just a silly phase I'm going through...

      "That was how they discovered that some of the sockets were wired on one phase and some on another, after connecting a PC to one phase and the printer to the other..."

      I always thought that sockets needed to be marked if they were on different phases but it appears that the regs changed sometime in the 2000s as I moved into an office in 2011 where each row of desks was on a different phase and nothing was marked. There were a couple of places where people were sharing socket strips and equipment that was on different phases. We showed this with a meter between two lives.

    2. Orv Silver badge

      Re: It's just a silly phase I'm going through...

      Back in the 1990s University of Michigan had a fire in the UPS battery room of their main data center. It was a huge mess -- they were down for days and they took Internet service for most of the area down with them, since at the time they were a major peering point for the Michnet regional network. Around the time they got that resolved they suffered a fiber cut, which led some to suggest they needed to present an offering to the God of Fire and Backhoes* that they had so clearly offended.

      * diggers, I believe is the term, for those in the UK.

  23. StargateSg7

    While we DID send MANY kilovolts into ham-hocks, rump roasts and whole turkeys and chickens back in the day to test our data centre power surge/power spike failover systems which provided us with many a flash-arc-cooked dinner, we NEVER sent that electricity over to LIVE DATA CENTRE SYSTEMS! This type of bypass is LUDICROUS in the mere fact that it was even WIRED this way is unfathomable to us! Even in the early/mid-1990's when were were blowing up lots of dead food with electricity, there is NO WAY our building wiring and failover circuits would even ALLOW such jerry-rigging as was described in this story!

    Whoever originally designed and wired the UPS system and data centre building circuits has some SERIOUS self-introspection needed! There should have been MULTIPLE FAIL-OVER and breaker-circuits allowing AUTOMATED and MANUAL BYPASS of any such generators, power conditioning and power backup devices! Many times you have to do an intrinsicly safe LIVE REMOVAL and/or LIVE INSERT of OLD and NEW computers, generators, UPS'es and switching gear. The FIRST thing you do when you design and build a data centre is to ENSURE proper fail-over and automated/manual bypasses and switch-overs. This is Data Centre Building Design 101 and should NEVER have happened in the first place! Even building cabling grounding and multi-rack grounding has to be thought of and TESTED from the get-go! You just can't plug-in big power systems without some forethought and BASIC pre-thought-out design work!

    I don't care if you design a 5000 Euros data centre or a 50 Million Euros data centre! THINK and BUILD-IN all the PROPER cable designs, building power and rack grounding, UPS/power supplies backup AND bypass/failover circuits in BOTH automated AND manual modes in case everything fails all at once!

    I can't believe this is happening at this time where we can now look up modern electrical codes and MANY data centre design tips'n'tricks as a set of PDF files that can be found and downloaded in mere minutes!

    I am flabbergasted and kind-of ANGRY that things like this STILL happen! 380 Volts is a big amount especially since it was at something probably like 40 to 50 amps which 19,000 WATTS of power which can deep-fry ANY human body in mere seconds. Sorry if I sound a bit self-righteous BUT electricity is NOT something to be toyed with willy-nilly! I know far too many people who were severely injured and killed even by a just a few volts and amps! When it comes to electricity it's ALWAYS SAFETY FIRST! DO NOT JURY-RIG and BYPASS safety circuits that are there for a reason!

    V

    1. GrumpenKraut
      Boffin

      > 19,000 WATTS of power which can deep-fry ANY human body in mere seconds.

      Dear Mr. pants-on-fire: 19kW is what a typical electrical shower heater is rated. Still I have never been deep fried within seconds of showering.

      1. cyberdemon Silver badge
        Holmes

        Actually, 19kW would be a very atypical electric shower. Mine is rated at half that, and takes 40A at 230V. A 19kW shower would need to be on 3-phase AC since it would be not-very-economical to run 80A cabling up the wall of a house..

        9.5kW is enough to raise the temperature from 10C input to 40C output with the flow regulator almost fully open. At 19kW you'd be showering in 70C water.. Not quite deep-fried, but certainly parboiled.

        It would also depend on how those 19kW were delivered. Your microwave oven takes 1.2kW from the wall to deliver 850W of heat into your food. So if I put you in a 19kW (output power) microwave, I'm sure you'd feel deep-fried within a few seconds.

        19kW is also more than enough to run a commercial deep-fat-fryer of the sort you'd find in a chip shop.

        So don't call people "Mr. Pants-on-Fire" without checking your own figures, Mr. GrumpenKraut. :)

        1. GrumpenKraut

          Once had a water heater that maxed out at 16KW. During winter that was just enough power to make the water decently warm. "Deep fried in seconds" is off by about three orders of magnitude. You can do the calculation if you really care.

          1. bernmeister
            Alert

            "Fried" is a colloquialism used to describe damaged electronics. Dont take it too literally. It doesnt take many Watts at high voltage in the wrong place to wreck electronics. Fuses are not intended to protect electronics from failure, their main purpose is to pevent fire in equipment. Even surge arrestors may not prevent damage. Protecting against the kind of overvoltage experienced would require circuits specifically designed for that purpose. Either they did not exist or had been bypassed.

  24. Andy3

    Sounds like a mix of laziness, lack of maintenance and ancient, incompatible systems. Also a lack of over-volt protection and basic fusing. A shambles.

  25. bernmeister

    Start of a trend.

    Just recently there has been a rash of failures in airport communication systems. A chronological list is needed. This failure has been closely followed by an unrelated failure in the USA, closing down all USA commercial airspace. Who's next? Systems have neen critcized as being old fashioned and out of date but with design and implemention times of years parts can become obsolete before the system is up and running.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like