back to article Millions of APC Smart-UPS devices vulnerable to TLStorm

If you're managing a smart model from ubiquitous uninterrupted power supply (UPS) device brand APC, you need to apply updates now – a set of three critical vulnerabilities are making Smart-UPS devices a possible entry point for network infiltration. The vulnerabilities, dubbed TLStorm, were found in Schneider Electric's APC …

  1. Down not across

    Yet another pointless insistence on "cloud"

    Why on earth would my UPS need to connect to "cloud" or anything external?

    All my UPS and RPC kit are on their own VLAN and definitely without any access to internet. Only access is internal monitoring, logging and control.

    1. Anonymous Coward
      Anonymous Coward

      Re: Yet another pointless insistence on "cloud"

      Indeed. Another good reason why "IoT" devices should never have free unfettered access to external environments. SNMP, email, http, etc - all of which can be monitored and controlled internally, but automatic connection to cloud services. No chance.

      I have just this week purchased two of these UPS's, which I note support this "cloud management" as an integral feature. I have not yet opened them up to see if this can be disabled. Out of precaution I also bought the optional plug-in management cards, so that I can run internal management services without needing to connect the cloud port to anything.

      There is absolutely no reason why i would ever want devices on my internal network to "phone home" with their data and for APC to give me access to their portal to view my own devices.

      1. J. Cook Silver badge
        Boffin

        Re: Yet another pointless insistence on "cloud"

        We have at least one of these devices; the on-board ethernet is disconnected and we use the plug-in management cards on it for alerting and management of the device in-house. The firmware will not use the management card to phone home, only the built-in ethernet. Keep that one unplugged (or put a dust plug in it!) and you'll be OK.

    2. cawfee

      Re: Yet another pointless insistence on "cloud"

      Let's connect a giant flammable device to the internet, I don't see anything that could go wrong

    3. ThatOne Silver badge
      Devil

      Re: Yet another pointless insistence on "cloud"

      > Why on earth would my UPS need to connect to "cloud"

      Automatic status updates to Facebook & Twitter?

      (You don't really live if other people can' read about it.)

    4. Anonymous Coward
      Anonymous Coward

      Re: Yet another pointless insistence on "cloud"

      We had to add a UPS to some installations of some other equipment for our main customer (part of the US Army). Yes, it's probably on this list.

      But I specified NO network connection lest we had to go down that rabbit hole of tracking even more software/firmware versions, testing, etc. Customer prefers to just leave it alone until it beeps that it needs a new battery, or maybe they already have a UPS maintenance schedule.

      That's what they get for insisting on the COTS product due to schedule and cost. I'm (mainly) the cable guy and just had to make sure the power levels worked out.

    5. Dwarf

      Re: Yet another pointless insistence on "cloud"

      I came here to say the same, so have an upvote instead.

      There is absolutely no need for core infrastructure to have a connection to the cloud. Anything used for management of any IT component should be on a management network - one in-band (to the OS) and one out of band to the OS. There need to be enough of these management networks so that there is no way to move horizontally across systems and bypass other controls.

      Its dead simple to configure with a bunch of VLAN's and a management firewall, then with the support teams connecting via management (jump host) workstations.

      Any platforms that just stick everything on the same network is just asking for trouble, be that local LAN segment, or the public Internet.

    6. Graham Cobb

      Re: Yet another pointless insistence on "cloud"

      I completely agree in the case where this is a personal/home deployment or a data-centre deployment.

      However, I am sure there are many, many millions of these devices (probably a majority) which are just a tiny part of a managed service doing something else. For example, I am sure the (independent) petrol station across the road doesn't own their own cash register/credit card payment machine. They are in the car business, not the IT business - a lot of money passes through their tills, with a lot of regulations about tax, etc - and I am sure they contract that out.

      The managed service almost certainly provides the tills, the card readers, the network devices and the UPS for them all. This UPS will be connected to a cloud-based service for remote monitoring and management, scheduling of battery replacement, etc. The service provider probably accepts APC's assurances regarding the security of the setup (and may even subcontract the connectivity to the APC cloud service). Those are the cases that will be screwed by this vulnerability.

      1. AMBxx Silver badge

        Re: Yet another pointless insistence on "cloud"

        But that just makes it even harder to update. Why the insistence on cloud, when a secure VPN connection would be sufficient.

        1. Graham Cobb

          Re: Yet another pointless insistence on "cloud"

          Because this is the reality of the business world.

          The garage owner (tiny business) contracts with a PoS supplier. The PoS supplier is a managed service: they contract with someone who designs the particular solution for this garage from standard building blocks and shows them how to use the web interface to monitor the UPS batteries and error reports. The solution builder uses the boxes recommended by the manufacturers and connects them together following the designs and specs from the manufacturers. The UPS manufacturer recommends the cloud service: after all, they have spent a lot of time and money designing and provisioning it to make life easier for their customers than their competitors do. And, in most cases, it is adequately secure (the biggest security concerns in PoS are about the money flow, after all!).

          The fault lies with APC - not with their customers for using their deployment instructions.

          I am sure that in your enterprise projects you have the luxury of a Solution Architect who can look across the end-to-end solution and see the weaknesses. But the real world of the bulk of systems implementations, particularly in tiny companies and in retail, do not have that luxury. Profit margins are so tiny they can't pay for the architect, and even if they did, they wouldn't be able to afford a solution that was not straight off-the-peg.

          1. Dwarf

            Re: Yet another pointless insistence on "cloud"

            @Graham.

            Do you think that APC used an architect to design their solution, or did someone in marketing just make a fuss about having the word cloud in their product description and some developer lashed something up to meet the minimum cost and minimum time that would have been defined in the same meeting ?

            I agree that end customers should be able to consume a system in a secure and reliable manner.

            But it doesn't need cloud in the first place, a simple buzzer and an LED on the front that says "Fault" would do the same, having it send an email or alert to someone is also easy to do, as is providing a local app on the PC / server to see the UPS state and report it to the OS. There are many reference patterns on how to do this without requiring a cloud connection.

            On the flip side though, any PoS provider or other such company should have appropriately skilled people to do an install right and once installed, have the appropriate ongoing management of a system through its lift. After all, if the PoS system is down, then so is the company, hence they should want to protect their systems.

            1. Anonymous Coward
              Anonymous Coward

              Re: Yet another pointless insistence on "cloud"

              I agree that end customers should be able to consume a system in a secure and reliable manner.

              How in bloody hell do you consume a system?

              1. Dwarf

                Re: Yet another pointless insistence on "cloud"

                Consume as in use.

                i.e.

                consume your data allowance === use your data allowance

                consume cloud services === use cloud services

                consume a UPS capability === use a UPS capability

                1. Anonymous Coward
                  Anonymous Coward

                  Re: Yet another pointless insistence on "cloud"

                  I remember, a long time ago (less than 10 years, if my memory serves me right), that I used to read a book, listen to music, watch a movie, use a computer and so on.

                  Now it seems I'm consuming books, music, movies, computers, phones, cars...

                  BTW, do I also consume a piano or am I still allowed to say I play it?

              2. Crypto Monad Silver badge

                Re: Yet another pointless insistence on "cloud"

                > How in bloody hell do you consume a system?

                I refer you to M. Mange Tout. He ate an entire Cessna, if I remember correctly.

              3. Anonymous Coward
                Anonymous Coward

                Re: Yet another pointless insistence on "cloud"

                Plenty of hot sauce, salsa, and a side of chips?

          2. Anonymous Coward
            Anonymous Coward

            Re: Yet another pointless insistence on "cloud"

            "...not with their customers for using their deployment instructions."

            You're missing the point about UPS by using PoS as a contrast, while under the context of a "cloud" based UPS. At what point does your batteries need to virtually transfer to a home base for calculation?

            In fact, isn't it irresponsible to add any latency between communicating you may or may not have power or worse, a fire has started? How does the fire department feel about this? If "... this is the reality of the business world.", you may want to start to question your reality, you might save some money and/or lives.

            For the PoS angle, yeh, that's sticky. With automobiles though, I thought there is no lemon law on 2nd hand vehicles and "new" vehicles are supplied with all retail information by law (although dealership/franchise integration would be sticky, unless you take the McDonald's approach :-/).

            1. Graham Cobb

              Re: Yet another pointless insistence on "cloud"

              You are missing my point. UPS is mostly a business solution - very few consumers have one - and small ones are often bundled as just part of a solution that is bought for some other purpose.

              In my example, the garage doesn't buy a UPS. They don't even know they have one! They rent a cash register - it comes with a bunch of "stuff" installed under the counter or in the back office and connections to make it work. The company they rent it from manages it - including monitoring the UPS to see when they need to send an engineer out to change the batteries, just like they send someone out to change the card reader when the keypad wears out. That is the reality of small UPS use - most of them are paired up, 1-to-1, with something performing a business function and the customer is a small business buying the function, not the UPS.

              In this case, the end user isn't the one making the decision to use the cloud - it is APC making the decision in the devices they supply in this market segment. There are some good reasons to network them so that the people responsible for them (who are not the people at the location) can monitor and control them. But it should be secure.

        2. Anonymous Coward
          Anonymous Coward

          Re: Yet another pointless insistence on "cloud"

          I agree that a VPN may be a better solution, but no, because the units themselves don't really support a VPN link for one, and the server team and network team may not always be on the same page, so even adding one at the network level isn't something you can count on.

          That said I'd slap a wire-guard VPN QR code on them in a heartbeat if the units supported it. Not holding my breath as this company still forces me to shutdown all the attached load on the unit do an in place upgrade of the firmware on the network management card.

          As too this class of bug, I'd just block all traffic from the unit from ever finding the internet, and do that at the switch level, not by trusting the on device ACL/Firewall settings. Because I don't trust an APC half as far as I can chuck one.

      2. Anonymous Coward
        Anonymous Coward

        Re: Yet another pointless insistence on "cloud"

        "...a cloud-based service for remote monitoring and management, scheduling of battery replacement, etc."

        Nope, no cloud needed or even "smart" anything. I've personally assembled a 1200 array lug-to-lug UPS and the only software used was in the Fluke meters. Test a sample and rotate them out after X months regardless, then test the old/replaced batteries. You'll know immediately when the voltage drop occurs at the control, all this without anything smart. At home, you can do this with a USB charging adapter plugged in with a ATTiny attached to a buzzer. I don't go that far, but if you want smart you can achieve this many ways without a cloud or vendor supplied anything, just build it up as needed.

        Another field being sold the "smarter is better" crap is in the main control, so just because the UPS is fine, that doesn't mean your control thinks so. IMO all of this should be completely analog and while I'm no longer part of any of this, I've been hearing scary stories about fire systems. Apparently there's a few fire marshals out there doing too much work thanks to things being so "smart" (you're lucky if the fire marshal doesn't shut you down after 2 times, 3 times and you're on forced vacation).

        FWIW, with the smart/nic controllers on UPS's you can have access to things like N.U.T., but personally while I've read the readings, I never really have done anything with those readings as I simply replace a home UPS battery ever 18-24 months. So it's a neat feature, just not that amazing.

        1. Graham Cobb

          Re: Yet another pointless insistence on "cloud"

          "Home UPS" is hardly a market segment at all! This class of devices make their money living under every cash register in your local Tesco superstore and every shop in your town centre.

          To the extent that "Home UPS" is a market they consider at all, it isn't the connectivity features they add for that market. It is the lock-in proprietary batteries and things like that. The network and cloud features are there for business users, who do have a need for them - any benefit to the tiny number of home users is just luck from their point of view.

          1. Anonymous Coward
            Anonymous Coward

            Re: Yet another pointless insistence on "cloud"

            "...cloud features are there for business users"

            Do you work for APC, honestly? Everyone you're debating with has given an example on why a UPS should _NOT_ have cloud dependencies, but you haven't give a single one on why it should. Are you going to PM me on a "deal"?

    7. TeeCee Gold badge
      Facepalm

      Re: Yet another pointless insistence on "cloud"

      Because anything that isn't in "the cloud" is old, busted, obsolete and only purchased by companies that are out of touch with the world.

      Look, it says so in the tech pages of the FT.

  2. Anonymous Coward
    Anonymous Coward

    Use case

    I actually own one of these UPS's that's used at a cabin a few miles south of the Canadian border that's remote enough that the only low latency internet connection possible (until Starlink ever delivers after 16 months on their waiting list) is 1.5Mbps DLS over a land line. (And the phone company has run out of lines in the area, so I can't even upgrade to bonded 3Mbps service.) The remote access via the cloud isn't very useful but it seems that's required to get the thing to email alerts when the power at the cabin goes out or comes back on, which I want to know about.

    Otherwise, while I see the advantages of remote access via a LAN (which allows remote accessibility via a VPN) is useful, it's clear that Schneider has architected this cloud scheme for subscription revenue. You get 3 years "standard" service when you buy the unit. but after that you'll have to pay. And they have several levels of varying uselessness service at various gouging price levels.

    I guess the crappy security comes for free.

    1. Anonymous Coward
      Anonymous Coward

      Re: Use case

      "...power at the cabin goes out or comes back on, which I want to know about."

      There's Raspberry Pi projects that can do exactly this. In reality, a RPI is vast over kill, but the amount of LED/Light switch tutorials for RPi is crazy abundant. It might be more technical, but some of the difference in technicality will be eaten away by the fact that you still have to learn how to setup the "cloud" software (and probably need an app). With the RPI all you need is a daemon with an email capable of SMTP/POP3 (most e-mails), the physical setup is more or less connecting a number on one thing to a number on another (so 2 different types of learning curves, both easy).

      There's also an abundant supply of RPI/ESP32 tutorials if you want to setup remote video, but by the sound of your connection, you'll have to use the still picture option :-/ (although server side you can still save a 30fps video file).

      1. BenDwire Silver badge
        Facepalm

        Re: Use case

        As the MD of a small business I've used APC stuff for years, and have only ever used the serial or USB interfaces. When my company's PABX was replaced the installers recommended that it was run from a UPS. It was located in a basement room where some production equipment was stored, so very few people passed on a regular basis.

        One fateful day, the phone system just died for no reason, and by the time I went to see the blinken lights it was all back on again, working perfectly. It was only once I turned the lights out did I see the faint red glow from the dead battery indicator. The production guy then confessed that the irritating beeping had been going on for months, but he just ignored it ...

        Solution was a RPI running NUT plugged into the UPS port. That served up a webpage with the status, and sent emails whenever anything untoward occurred. It subsequently fed data to Grafana when a proper IT bod was employed. I'm guessing that the price of a RPI is substantially cheaper than any cloudy solution.

  3. Mike 137 Silver badge

    "the firmware updates are not cryptographically signed securely"

    Why on Earth not? This is so basic. One wonders who they got to develop the software.

    Oh, I keep forgetting - we no longer programme, we code (and when stuck we copy and paste blindly from Stack Overflow).

    1. Anonymous Coward
      Anonymous Coward

      Re: "the firmware updates are not cryptographically signed securely"

      Most developers these days have incredibly limited IT experience. The quality of the code they produce is often shockingly inefficient, shows a lack of understanding about networking, security and and other fundamental concepts. However it’s often very neat and easy to read but works in the way their mind works rather than reality. It’s not their fault, it’s what happens when you create an environment where they don’t need to learn anything.

      1. ChipsforBreakfast

        Re: "the firmware updates are not cryptographically signed securely"

        You are 100% correct. I've found myself having to explain networking basics (and I'm genuinely talking about basics here - networks, subnets, ports & protocols) to more developers than I can remember of late.

        I wouldn't expect a dev to be able to design me a full blown enterprise network but surely it isn't too much to expect them to understand what a subnet is when they're writing network-aware code?

    2. Anonymous Coward
      Unhappy

      Re: "the firmware updates are not cryptographically signed securely"

      > One wonders who they got to develop the software.

      They probably just copied the demo from the board manufacturer.

  4. Marty McFly Silver badge
    Thumb Down

    What does APC get out of cloud connectivity??

    Any bets they get telemetry on the health of the batteries? I'll bet the registered administrator gets a warning email with a convenient 'Buy Now' button to APC's overpriced OEM batteries.

    Watch for upcoming UPS versions to require cloud connectivity for just that purpose. Automatically receive new batteries every two years with eco-friendly recycling of the old batteries. Don't worry, this was all covered in the EULA when you provided your credit card number and signed up for the mandatory account.

    1. skizzerz

      Re: What does APC get out of cloud connectivity??

      They 100% get telemetry on battery health, they even advertise that connecting the UPS to the cloud will extend your battery's warranty by an additional year.

  5. Anonymous Coward
    Anonymous Coward

    Home UPS

    I have two APC UPSs at home. One supports my Internet Connection(in the old coal hole) and the other is in my office where I run a web server.

    These are essentially redundant now that I have a stonking great battery installed (48kWh) that runs my whole home inc all the heating via a heatpump. Yes, I do have solar.

    As more people get home battery storage, the market for small UPSs will drop quite a bit.

    Being a bit of a 'cloud sucks' sort of Grumpy Old Sod, those UPSs have never been connected to my Lan so I guess that I'm safe from this problem but I do know a good number of small businesses where this is not the case. I'll be ringing them up over the next few days and getting them updated.

  6. E 2

    This is so cool!

    Can they make them explode remotely?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like