University DDoS'd by its own seafood-curious malware-infected vending machines

A US university saw its network traffic slow to a crawl thanks to an IoT malware infection that hit, among other things, its vending machines. The unnamed university had its story told by Verizon Enterprise in a sneak preview [PDF] of its 2017 Data Breach Digest report. The story, as told by an also unnamed senior IT staffer …

[Anonymous Coward]

Lesson Learned

There's a valuable lesson for them to have learned: don't put crap on your net. We have wifi vending machines here at work, they take credit cards and NFC, and I have no idea what net they are on, but I know my security people would not allow any device on the network without checking it out thoroughly. But then, we have a dedicated security team and run online and normal retail ops, so there's no way an infected vending machine is going to appear on my work production networks. At that mysterious university? Yes, there is a way. The staff are not equipped to do anything or say anything about those fancy new vending machines until AFTER they cause a problem. 100% this is "hey, just install the ice cream machines and don't worry about the wifi security, kid!" One. Hundred. Percent.

[Flocke Kroes]

Re: Lesson Learned

Don't use plain text passwords to control your botnet.

[Stevie]

Re: Lesson Learned

All your Mars Bar are belong to lightbulb.

[Anonymous Coward]

1. Put the IoT (Internet of Trash) behind a firewall.

2. Change the frickin default passwords.

3. Put the IoT (Internet of Trash) on a VLAN separate from everything else. If you need to make several IoT VLANS, do so.

4. Write a script that scans all of your VLANs looking for IoT devices that allow default credentials. If you find one, take action to hunt the vermin down.

http://www.csoonline.com/article/3126924/security/here-are-the-61-passwords-that-powered-the-mirai-iot-botnet.html

[Mage]

Also

Disable uPNP on firewalls / routers.

The article is a catalog of incompetence.

[hmv]

Re: Also

uPNP is unlikely to be enabled on a University firewall - we don't run SoHo gear.

As to incompetence, well perhaps, but universities are weird places where IoT devices spontaneously appear on the network without the (in)security team having a say. This is not necessarily a bad thing in itself (academics are supposed to play with weird shit - it's in their job description).

What is also possible is that the IoT vendor agreed to comply with security best practices, and then ignored that contractual requirement whilst deploying the gear.

[Anonymous Coward]

Re: Also

Our university (hence the AC) is planning a network upgrade and one of the great new ideas is to do away with subnets-per-department so all machines can talk to all others.

Sure there are plans for trusted/untrusted IP ranges but nothing so far that indicates this sort of thing has been taken in to account. Apparently your single sign-on for windows will be enough to make sure you get the appropriate access...

[Adam JC]

Re: Also

Crikey, won't you run into IP allocation exhaustion issues with regards to IPv4 pretty quick?

And aren't your core-routers able to do inter-VLAN routing with ACL's? Surely that's a better way of going about it.

[Bandikoto]

Re: Also

16.8 million hosts aren't enough for even a large land-grand college? Sure, the public hosts will likely be on a Class B network, but everything else (e.g. IoT, phones, student laptops) will be on a private network and those are invariably on net 10.

[bombastic bob]

Re: Also

"Apparently your single sign-on for windows will be enough to make sure you get the appropriate access"

I _HOPE_ they're not excluding Linux and FreeBSD (or even mac) by requiring a 'single sign-on for windows'... or maybe I'm just reading too much into this.

Anyway, separately firewalled subnets for IoT might help. Or not.

[Anonymous Coward]

Re: Also

I don't think they will exclude them, many in the arts department use Macs, as do a few of the IT folk themselves, and many of the sciences use Linux boxes and the occasional (very occasional now) Solaris box.

I just think they have not thought it through. Or are thinking of using the underpants+profit approach to design...

[Anonymous Coward]

Internet of Tat

That is all

[Stevie]

Re: Internet of Tat

Your "w" key isn't working, AC.

[cd]

I was just reading today about an IOT water heater. "Only" $1000 for a 50 gallon. Which would send me updates about how it was doing. And potentially, according to the description, save me $4444 over its projected 10 year lifetime. Be curious how they arrived at that one, other than just holding the 4 key down, then Return, and going to lunch.

What would Pirx do if his water heater texted that it was in jeopardy?

[DNTP]

water heater in jeopardy

Probably find out that the IoT company that made the water heater had gone out of business five years ago, or that their "warranty" only covered hardware malfunctions and they had no interest in supporting a software or security problem.

[Youngone]

Reads like marketing nonsense

The linked PDF reads like an Ad for some Verizon service or other.

I'm going to choose to be skeptical about whether the described events even happened.

[razorfishsl]

Seriously WTF was this not on a separate VLAN with Firewalls.

Typical China like mentality, just increase the subnet mask and stick it all on the same network with the same DHCP.

As one of my staff asked today,

"I've been reading up on some internet tutorials, can you give me a config dump of the CISCO core switch & FW, I want to improve it"

.......

"NO"

[Doctor Syntax]

As far as I can see this network was specifically set up for the purpose, presumably by the University's own network team. Did it never occur to them to change the passwords?

[Denarius]

Ritchie and Kernighan again

Reflections on Trusting Trust. ITIRC ASCM decades ago. Trust nothing, trust no-one until at least basic verifications done. Techno-utopians could benefit from understanding the doctrine of the total depravity of humans from the Reformation. It at least gives a sensible starting point for risk assessment such as, "how would a baddie abuse this kit/process/document?"

[jake]

Re: Ritchie and Kernighan again

ACM, not ASCM. Written by Thompson. No Kernighan or Richie.

http://dl.acm.org/citation.cfm?id=358210

I'm not certain it supports your thesis. Perhaps elaborate?

[Steve the Cynic]

Re: Ritchie and Kernighan again

The point, I think, is about how you can know that a particular thing is trustable, and the conclusion is that unless you can trust the person (or people) who created it, you can't trust the thing.

[Anonymous Coward]

I can't help wonder...

If this is the same kind of university which would also easily hook up a students network with the internal administrative network and the teachers network, only to end up surprised that students managed to gain access to their study results and more...

[John Smith 19]

I'm guessing the slightly tough part.

starts once you've written a script to scan for the Mirai botnet list of default passwords.

You find 1000 devices all with the same password because this is the IoS we're talking about.

Do you a) change them all to new standard password (get one device, get them all) or b) Create and give them all unique passwords and keep them in an encrypted field in a database.

a) Is cheap and probably quite simple, but once it's compromised you're back where you started. b)Is more work to start with but you can leverage the result for all you future IoS devices.

Maybe I'm paranoid but a layered defense seems a pretty good idea to me. As others have said, firewalls and subnets are good. Limit the access. Find a way so once a device starts spewing packets at some ridiculous rate for no obvious reason (other than it's been infected) it chokes something and there's an obvious marker to investigate.

[Doctor Syntax]

Re: I'm guessing the slightly tough part.

"Do you a) change them all to new standard password (get one device, get them all) or b) Create and give them all unique passwords and keep them in an encrypted field in a database."

Because you want to act quickly, go for a) on the first pass. This gives you breathing space to implement b) when you've worked out a suitable strategy.

As this installation seems to have been intended to apply down to the level of every light-bulb in the place for easy of maintenance (yup, sure made life easy!) there might need to be a lot of people who needed access to the list so implementing b) might not be straightforward. It might even include a review of whether all the devices needed to be "smart".

[Jess--]

Re: I'm guessing the slightly tough part.

Trouble is on a lot of these things you can't change the passwords.

for example on a cctv dvr manufactured by dahua (and heavily resold under many brands) you can change all of the passwords apart from 2, the 1st is a remote view login (33333333 / 33333333) the 2nd is a root login (88888888 / 88888888)

the box itself is a cracking bit of kit but you can't deploy it because there is no way of making it secure.

[Stoneshop]

Re: I'm guessing the slightly tough part.

the box itself is a cracking bit of kit but you can't deploy it

... on a public network

because there is no way of making it secure.

Pickaxe, blowtorch, C4.

[RSteer]

Re: I'm guessing the slightly tough part.

I wonder if it's possible to put such devices behind a bastion or gateway that requires a strong login, making the individual device weaknesses not easily accessible?

[John Smith 19]

"It might even include a review of whether all the devices needed to be "smart".

Possibly the most fundamental question of them all.

And probably the least frequently asked until some poor fool has to support this s**t

The IoT is coming like a slow mo car crash. I could not believe that circuit breaker modules already have internal microcontrollers in them. WTF they do I have no idea. They certainly aren't net connected and they are robust against being zapped by high currents and voltages (by the standards of digital equipment). Maybe other systems could learn from them?

Hard coded accounts is a lazy, incompetent way to cope with dumb users. I do hope it won't take an actual law to stop this stupidity being included.

[Michael H.F. Wilkinson]

Seafood curious?

The IoT devices caught the crabs?

[Stoneshop]

Re: Seafood curious?

They misinterpreted "squid proxy"?

[Korev]

Re: Seafood curious?

They misinterpreted "squid proxy"?

These puns are awful, put a SOCKS in it...

[Hans Neeson-Bumpsadese]

requesting seafood-related subdomains

Obviously some sort of phishing attack then

[Stoneshop]

phishing attack

In this case, wireshark will probably have caught it.

[zaax]

And there is the problem. Vendling machine owners do not care about your network.

[Paul Crawford]

Set your IoT networking rules to only allow access to the vending machine companies network addresses then.

[Anonymous Coward]

Erm yes, Dave Lister's boss Rimmer has a static IP and the knowledge to tell you what it is...

Most vending M/C repair vans/companies I've seen are Mum (+ Pop) type companies, probably running out of a lockup/from home with BT Internet as the ISP (because their kids got fed up with the "who is best calls" and then arguing with the anything but BT answer)

[Infernoz]

Simples, just block DHCP requests from unknown devices, or otherwise restrict network access of unknown devices.

A 1st layer of controls could use a MAC address white-list (devices registered by authorised people) and/or MAC address to IP address mapping, extra control layers could include on-line auth. e.g. encrypted user auth.

I use MAC address to IP address mapping at home, to simplify device use and as a 1st security layer, I can then limit access via the IP address and encrypted user auth.

[EnviableOne]

separate vlan, separate interface on firewall, no access to the rest of the network, Bandwidth limited on the infrastructure and v short ACL

iotVlan_Access_In

permit IotVlan VendorNetwork

outside_Access_in

permit VendorNetwork IotVlan

there IoS their Problem

[Afernie]

Yes, but...

It would not surprise me in any way if the Vending Machines lacked VLAN support.

[EnviableOne]

Re: Yes, but...

The machine doesn't have to as long as the switch does.

the vending machine is plugged to the switch the port assigned to the VLAN, the network virtually isolated from the rest. 802.1q its a great standard

[Afernie]

Re: Yes, but...

Yes, 802.1q is a great standard, and no you don't need tagging support. I've nonetheless seen some very interesting behaviour with kit lacking it. Weird shit with bridging firewalls, for example.

[Anonymous Coward]

Internet scanned devices default passwords

Shadowserver reports supplied by your friendly hosting network should be supplied and investigated.

[Primus Secundus Tertius]

My new car

My new car is an IoT on wheels. I am moderately confident, but no more than that, in its integrity as it emerged from the factory.

But there is a USB socket by the gear stick. One clueless service mechanic wanting music with their tea-break....

[Anonymous Coward]

Coke machine...

The first IoT machine ever was a soda dispenser that could read its thermostat, so the students knew when the Coke truck had refilled it and when its contents were cold.

There is irony somewhere in there.

[Stevie]

Bah!

Makes you wonder how they sold machine-vended chocolate bars before the IoT doesn't it?

[zen1]

If that happened where I work, I would have the individual, who authorized the decision to stick that crap on my production LAN and NOT behind a fw, hunted down and lectured. And any monies spent mitigating that mess would come out of their budget. That's inexcusable, even if it was a university.

Secondly, I would be having a nice long talk with the vendor about them reimbursing my company for the expense, as well as what their plans were to prevent events like these from happening again OR a when I could expect them to get their shit out of my buildings.

[David Roberts]

Money talks

Vendors sponsor education establishments to use their vending machines.

The vendor turns up and plugs the machine into power and the nearest convenient LAN socket. IT involvement?

[W4YBO]

Lamp post?

"Short of replacing every soda machine and lamp post, I was at a loss..."

Why would a lamp post have a network connection? What practical value could that have?

[Andytug]

Re: Lamp post?

The "saving" as always seen by the PHB and their bean counters, is manpower.

No need to send a person out to check any readings, usage, how many choc bars are left or whether the bulb is gone, it'll tell you over the internet.

See also smart meters and a heck of a lot of IoT tat.