Why these cloud-connected 3D printers started making junk all by themselves On August 15, 2023, 3D printer maker Bambu Lab experienced a service outage caused by what it’s described as "abnormal network traffic" that caused some customers' cloud-connected printers to start creating plastic objects without user intervention. Following the phantom printing, some customers complained on social media that …
They were from the queue, but still went through the "cloud" :-/. Everything but what's on the SD goes through the cloud... sadly (although I think even what's on the SD can go to the cloud :-/).
Bambu clearly has a great printer but their decision to create dependency on their cloud killed a lot of purchases. Also what cost them was keeping their firmware closed as Marlin and Klipper are the clear winners in filament based 3D printer firmware and deciding not to use either is a failed decision. All in all it's seemingly the best pre-built filament printer sub $4,000, but there's still a lot wrong with it.
Using MQTT to start a print job makes me wonder about how much of the code on this printer is written by Bambu and not simply copied from other sources. Using MQTT for messaging is fine, but using MQTT for complete job control looks extremely amateur, like a jquery user does today. Regardless, Bambu needs better code.
Lastly the article writer wrote "...plans to monitor the temperature of printer..." Plans to? It already does, like every other modern 3D printer. Technically even really old ones do to at least feed the PID control. All of them now have thermal runaway monitoring, even though that issue was blown way, way out of proportion for modern materials used with modern printers.
We all know that cloud = someone else's server.
In this case, it would seem that someone else's server had an issue with restoring a proper job list after a connection had dropped (sounds like something that should have been tested properly).
That looks like there isn't sufficient job identification when printing. If the printer knew which job it was printing and if each new instruction was accompanied by the proper jobID, then if a new jobID was suddenly sending instructions, the printer would be able to refuse and set itself in error status.
Sounds like that is a precaution that was not taken because why think of making sure the printer knows what it is printing ? The CloudTM never goes wrong, right ?
Knew there'd be someone getting lathered up about The Cloud. Can't see how it has any real bearing on this, which could easily have happened with a USB-attached printer with rubbish firmware and rubbish desktop software/device drivers. Apart from apparently no-one's allowed to make things without their own wifi connection and TCP/IP stack nowadays.
For a shining example of software engineering excellence and robustness, I refer you to my local HP printer. The one which frequently requires a reboot to both PC and printer before it will deign to print a single page.
Rubbish software is rubbish, no matter where it's run.
But this wasn't your local HP printer, it was a Bambu printer controlled by the cloud and that seems to be where the error was. You're correct about rubbish being rubbish but the more complex you make things by going outside the local setup the more things are available to go wrong. If there wasn't a solid reason for it to be done this way then it looks as if an unnecessary risk was introduced.
The reason for doing this was to provide an optional set of features that some people may want to use and others not.
If you didn't want to use it, no harm and no foul: no internet connection, no job scheduling screw up.
> If you didn't want to use it, no harm and no foul: no internet connection, no job scheduling screw up.
...and no printing directly from the PC. You'd think in 2023 there'd be some middle ground between cloud-based-middleman* and sneakernet.
I was about to construct a snarky comment about how maybe someone should invent a standardised cable for connecting peripherals to computers - something universal - and low pin-count, so perhaps some kind of serial bus...
Then it dawned on me, for a significant number of today's developers, the head-in-the-cloud model the only model they know. It's arguably easier to cobble together an IOT device and associated service on a remote server than it is to create a USB printer class device, write a driver and supporting software for it, and then get those approved and driver-signed. It's also easier to go the cloud route than to deal with the support issues that go with getting people's computers to talk to the device directly on their own LAN. So perhaps my cynical footnote is just a bonus, not the main driving force after all...
[* AKA tollbooth-in-potentia]
Marlin and Klipper both support printing via some kind of universal serial bus connection.
And Octoprint has supported printing via local LAN since inception.
Of course, this one company decided not to use any of the above. One wonders why that might be.
I've had that exact problem with Epson receipt printers. The job prints fine, but for some reason that information never makes it back to CUPS, which keeps re-trying the job. As the printers are attached to cash drawers, this also results in the drawer going DING and shooting open, which can be a bit of a surprise.
(This is also fun for pranks, I knew a colleague had a cash drawer set up on a test bench, so I remotely sent a print, then messaged him to ask if I'd managed to surprise him. I certainly had, especially when the cash drawer had sprung open, and shoved the computer right off the desk :)
That reminds me of when the internet was a new thing and lots of drivers etc were downloaded from manufacturers' dial up BBS.
We had a bank of USR Courier 56K modems connected to a Netware Comms Server to "network" them.
Used to have great fun - at a colleague's expense - sitting at my desk in the IT office a few feet from the "target", using terminal software to connect to one of the modems (which were in the server room) and repeatedly dial his desk phone.
The hard part was keeping a straight face while watching him progress through puzzlement, irritation, annoyance, frustration to slamming phone down anger.
When the penny finally dropped what was happening, it was still very hard for me and the 3rd person in the IT office (who was in on it) to not laugh as the "target" loudly demonstrated his extensive knowledge of swear words and repeatedly questioned my parentage LOL.
Why does a 3d printer need to be cloud controlled? That would seem to be over complicating things massively.
The cloud does offer the advantage that the device is accessible wherever the user happens to be in the world, but with any 3D printer, you need to ensure someone is there to remove the print from the plate.
The only advantage I can think of is if you are out, you can trigger a print job (these can be quite lengthy), so it should be ready by the time you get to the printer.
But that still does not explain why you would need a "cloud connected printer". All of it can be done locally.
You do not go on holiday and think "Oh hey, I need to put some stuff on my 3D printer". It is a solution that is desperately searching for a problem and creates a whole set of problems you do not want.
And for "I am not patient enough to wait for my print job to finish... Well, you could just start it in the morning and have it ready when you get home in the evening? That is, if you insist on leaving this stuff live and active without anyone in the immediate vicinity.
Well, you could just start it in the morning and have it ready when you get home in the evening?
Well, it works for sensibly slow bread (low yeast or sourdough types). Hmm, is there a marketing opportunity there? Cloud connected bread makers for extra fluffy bread? Oooh, 3-d printed extra fluffy cloudy bread? There's lots of scope for buzzword enthusiasm there!
You need to add a little AI too. How about:
Your new cloud connected SlowBreadMaker adapts to its environment and the ingredients using the latest AI. You control your SlowBreadMaker from a modern app and select your preferences on the fly from everywhere you are. The SlowBreadMaker learns your dietary requirements from your choices and SlowBreadMaker adapts intelligently to assist you in your daily healthy quests. SlowBreadMaker automatically orders supplies and will assure the healthiest composition of ingredients for your health and convenience. Never run out again. Your SlowBreadmaker will feed you your whole life(*).(*)Terms and conditions apply.
Your SlowBreadmaker will feed you your whole life
I assume life being defined as however long till it decides to introduce one of the more interesting ingredients.
Naturally, it would allow you to obtain your unique daily antidote by watching 20 hours of advertisements, paying for another BreadCartridge, or both, depending on your subscription tier.
Contacting the company by the form found in the bottom of the locked filing cabinet in the disused lavatory with a sign on the door saying ‘Beware of the Leopard.” will summon a free antidote in 4 to 7 business years.
> you could just start it in the morning and have it ready when you get home in the evening
And you don't see *any* utility at all in being able to keep an eye on the status of that print during the day?
Like maybe seeing it has a problem and you have time to pop back and get it going again at lunch? Or being able to give it some other command to keep it going (e.g. the spool of red has kinked and jammed, but these aren't going to be visible parts, so abandon the part-done piece and continue the rest of the run in black).
We use a bunch of "hobbyist" 3D printers (Prusa etc) for in house manufacture of an assortment of bespoke "widgets" for use internally.
Remote - across a network - monitoring of jobs complete with webcam view of the build area are very useful.
Simple solution - a bunch of Raspberry Pi 3B+ running OctoPi (suitably locked down), one for each printer.
And if we need to keep an eye on a many hour print job, connect in securely via VPN.
No cloud needed.
Things like the Spaghetti detection that can stop the printer before a huge blob builds up around the print head (can happen when print comes off bed during printing, layer shifts, model issues etc.) would biggest reason - The Spaghetti Detective AI
its printers "can only reach a maximum temperature of around 400°C [752°F]," a temperature it assured is too low to start a fire.
There's a piece of classic literature called 'Fahrenheit 451' , where the clue is in the title. A temperature of 752F is more than enough to start a fire.
And cloud connected by default as well? FFS why?
> Fahrenheit 451
You can auto-ignite some substances way below 400°C, but the general idea is to consider what materials you sensibly allow on your printer. If you decide to pop some white phosphorus onto the print bed and ram the hot end into it, don't expect much sympathy.
More sensibly (!) running the machine whilst it is wet with diethyl ether (really gets the stains out) is contra-indicated as well. Ditto ethanol. Absolutely not silane.
But if you keep all the obvious materials away from the hot end - and that includes paper - running at 400°C should not[1] cause a 3D printer to catch fire.
> consider what materials you sensibly allow on your printer
Kids off school during summer holidays... raining outside... could be _anything_ on the printer bed - clean clothes from a dryer, homework, dolls, chocolate cake, hay from the hamsters, hamsters eating chocolate cake...
A real risk of a nasty fire.
Um, monitoring - indeed, controlling - the temperature of the hot end, and other bits, such as the hot plate (notice a pattern in the naming here?), is a fundamental function of a 3D printer! You can even have gloriously nerdy conversations about the best thermistor to use, whether the design of that hot end located the thermistor too far from the nozzle/to close to the heating element (or vice versa)...
If they are not already monitoring those temperatures then something is very wrong!
Because print jobs can take a long, long time to complete and it can be useful to monitor them from wherever you are and, yes, even schedule a new print job: you have an idea, start a small job there and then, piece is ready to use when you get back that evening.
Clever IT type people set their printers up with OctoPrint then carefully allow the right bits through their firewall so they can monitor it during the day. Other people use a third-party server that the printer can reach through the NATting home router without opening up holes to the outside: AKA "the cloud".
"But it is far too dangerous to walk away from your hot and highly inflammable 3D printer while it is running!"
Aside from "you can always keep it safely inside a fire pit" how about considering that there may be someone around to notice the raging inferno and hit the Big Red Switch but that someone need not be otherwise bothered about what your stupid machine is doing all day, can't you at least do something about that high-pitched whine it makes?
There are even collections of privately-owned 3D printers connected to a third-party service that allows you or I to buy time and get a small job run off (getting away from this particular manufacturer, AFAIK).
"But why as default?"
Because all the flash features are always on by default these days! They assume you buy a gadget that talks to a server (when there are plenty that don't) because that is something you would like to use.
> There are even collections of privately-owned 3D printers connected to a third-party service that allows you or I to buy time and get a small job run off
Full disclosure - I've just checked the services listed in my old notes and, I guess no surprise, they have been taken over and are run purely commercially now. Not to say that there aren't still community run services, just no longer the ones I knew of.
But in their day, the community run services were getting use and were scheduled via someone else's server, so...
Clever IT type people set their printers up with OctoPrint then carefully allow the right bits through their firewall so they can monitor it during the day. Other people use a third-party server that the printer can reach through the NATting home router without opening up holes to the outside: AKA "the cloud".
There are a lot of potential solutions that don't involve cloudybollocks. So insert SD into printer and drive everything from a front panel. Ok, that requires a front panel that gives the user an illusion of control. Or printer sits on a LAN and the app sits on the user's PC or server. User can control that server however they want. Except I don't think the Bambu printers have a physical LAN port, so users better have a decent WiFi setup. Basically there's zero reason for stuff like this to insist on cloudybollocks, and force the dependency on it, especially when it can and does lead to issues like this.
Of course if users read the small print, the vendor will accept no liability for loss or damages. No dount lawyers will disagree and there's probably already a couple of class actions looking for fees. I mean compensation for wasted print media etc.
Aside from "you can always keep it safely inside a fire pit" how about considering that there may be someone around to notice the raging inferno and hit the Big Red Switch but that someone need not be otherwise bothered about what your stupid machine is doing all day, can't you at least do something about that high-pitched whine it makes?
I'm guessing the answer is also lawyers. There may be disclaimers stating printers should not be left unattended, but many users run print jobs overnight because they take a long time. Sensors may fail, thermal protection may not work and vendors don't want to get sued. But SNMP's been around since 1988, and other network/device management protocols even longer. Both Windows and *nix have integrated this stuff into their OS and messaging and management apps. So again, there's no need for any cloudybollocks, other than probably someone in the vendor's marketing and management loved the buzzword.
Mildly confused here.
>> Clever IT type ... carefully allow the right bits through their firewall so they can monitor it ...
> here's zero reason for stuff like this to insist on cloudybollocks
You mean, precisely like the scenario you just quoted?
> User can control that server however they want.
Just so long as they don't want to control it remotely, appears to be today's argument.
> and force the dependency on it,
Citation, please.
>> ... how about considering that there may be someone around...
> I'm guessing the answer is also lawyers ... printers should not be left unattended
I EXPLICITLY described a situation where the printer IS NOT BEING LEFT UNATTENDED! It just isn't being attended by the person who fetishistically wants to monitor its every move! How does your response relate to the sentence you just quoted?
I EXPLICITLY described a situation where the printer IS NOT BEING LEFT UNATTENDED! It just isn't being attended by the person who fetishistically wants to monitor its every move! How does your response relate to the sentence you just quoted?
It's a product management thing. You develop a product, and want to sell it to millions of users. Not all users are created equal, and some will do unexpected things. That will probably result in litigation, so you have to do risk assessment based on users being idiots. My favorite example was Ruger being sued for milions, and being forced to stamp a safety warning on the side of their pistols. Product liability avoidance has to assume users have no common sense. Sometimes it is also common sense, ie if the printer operates at very high temperatures, thermal safety is a good thing. Sure, you can say not to leave it unattended, but people are going to and you can't force a user to sit and watch a 5-6hr print.
Because print jobs can take a long, long time to complete and it can be useful to monitor them from wherever you are and, yes, even schedule a new print job: you have an idea, start a small job there and then, piece is ready to use when you get back that evening.
And how do you remotely remove the printed piece from the printer in order to print something new?
Ah, so the feature is entirely useless because you every time you have an idea you have also left old prints on the bed.
Gotcha.
There is an awful lot of stretching going on today to ensure that any possible utility from an entirely optional feature is totally driven into the ground.
> No, but, as with my computer, I do not let my 3D printer on if I'm not using it.
Your choice. Has nothing what so ever to do with what may or may not be useful behaviour for anyone else.
I, personally, leave my desktop PC off when I'm away from it - but my other computers are all running away, doing server-type stuff. So I do choose to have computers on when I'm not using them (just in case I - or anyone one with privileges - decide they'd like to use them).
I also let the washing machine run on an auto cycle while I'm out, ditto the dishwasher. The PVR is continually switching itself on, especially when I'm away from home for more than a few days.
...doing server-type stuff ... washing machine on auto cycle... dishwasher. Funny, so do I.
Do you also let your 3D printer on just in case you want to print something while you're at work? Fully knowing it's connected to the cloud, aka not under your control? Damn, you're braver than I am.
> Fully knowing it's connected to the cloud, aka not under your control?
Ah, there we have it - one job scheduling glitch (which sort of issue is not intrinsically cloud-related) and the whole argument comes down to "I don't believe I have any control, I do not have any trust at all in any third-party service, I do not believe it is worth the risk, it would be brave (or stupid) to ever do so."
And the *real* risk being? A day's wasted printing (I'm going to ignore all this "it will burn your house down, just you wait and see" noise), the cost of the filament and electric. Response to this horrid, ghastly, world-shattering event that'll give you PTSD until the end of your days? Contact the service provider, have a word and, if necessary, go talk to small claims.
If it happens regularly (which even this story isn't claiming happens - it is only newsworthy because a number of people's printers were affected) then time to change the setup. Grr, what an annoyance.
>I also let the washing machine run on an auto cycle while I'm out
While your washing machine uses quite mature techonology and probably has extra protection to prevent water leaks insurance companies still like to claim leaving them unsupervised is grossly negligent.
Most people won't want to print anything when they are not at home and switching off the printer will not only make it safer but although safe power. I know it will probably be only 3-5W in standby but that is still energy that can be used for something more productive.
In my opinion it is ok to have an option to control your printer from the cloud for people that think it is a useful feature but it should definitely not be the default setting.
"Because print jobs can take a long, long time to complete and it can be useful to monitor them from wherever you are and, yes, even schedule a new print job: you have an idea, start a small job there and then, piece is ready to use when you get back that evening."
That's the utility. You have to consider the cost of providing that. The cost is the risk that the increased complexity brings. Anything from a screw up like this via malware being introduced to the printer through that carefully crafted hole in the firewall that allows the printer to become a staging post to attack the rest of your network* to Bambu going TITSUP** and no more printing.
* Did you remmber to put the printer on its own VLAN?
** Terminal Inability To Service Users' Printers
Bloody Norah, we really are intent on totally trashing even the *IDEA* of an entirely optional feature like remote 3D printer control. Sigh.
> Anything from a screw up like this via malware being introduced to the printer through that carefully crafted hole in the firewall that allows the printer to become a staging post to attack the rest of your network
So, to your knowledge, the ONLY way to provide this kind of service is for the firewall to have a hole open? Note that by this point you've totally passed the point of discussing how Bambu in particular have implemented their system.
You do know that it is possible for a machine inside your firewall to make, say, an HTTP request out to a remote server without opening a hole in the firewall? And that HTTP request can pass information both ways? And another host, like, say, your phone or your PC at work, can also send requests to the same remote server (again, no holes in anyone's firewall) and... Doesn't even have to be HTTP, other protocols can be initiated from inside.
> Bambu going TITSUP** and no more printing.
Care to cite where all of the Bambu printers are 100% reliant on the Bambu servers in order to print? Not whether they provide nice-to-haves via their services, but basic functionality (as you are claiming 100% lack of printing).
Bloody Norah, we really are intent on totally trashing even the *IDEA* of an entirely optional feature like remote 3D printer control. Sigh.
I don't think anyone is really doing that, only how those features are implemented, why and the vulnerabilities they introduce.
You do know that it is possible for a machine inside your firewall to make, say, an HTTP request out to a remote server without opening a hole in the firewall? And that HTTP request can pass information both ways? And another host, like, say, your phone or your PC at work, can also send requests to the same remote server (again, no holes in anyone's firewall) and... Doesn't even have to be HTTP, other protocols can be initiated from inside.
I'm hoping it's HTTPS, but to permit an outgoing request, you have to open a hole. This should ideally be temporary, ie a stateful firewall but often it's down to trusting some ALG that the app is secure and the session won't get hijacked, corrupted or whatever. It's much the same with phones. Status stuff should be pushed, initiating a session from the phone just needs a way to find it when it's behind DHCP and/or NAT. Pretty much every OS can do this, and you shouldn't be forced to depend on some 3rd party server that may be unreachable, switch to a subscription model or all the stuff that happens when users are forced into the 'cloud' for no good reason.
> to permit an outgoing request, you have to open a hole.
My point being that you do not, as was being implied, have to open up an incoming hole. Anything that has to happen for any outgoing will happen for *every* outgoing, and if anyone is worried about that they simply have to stop using the Internet - having a 3D printer calling out is otherwise no worse than anything else.
> and you shouldn't be forced to depend on some 3rd party server that may be unreachable, switch to a subscription model or all the stuff that happens when users are forced into the 'cloud' for no good reason.
True. Very true.
Can you show where Bambu are actually doing any of that? As far as I can find out, their printers are quite capable of being driven by the good old fashioned front-panel. You get fancy features via their server but have not seen where it is a necessity and hence being, as you say, forced.
My point being that you do not, as was being implied, have to open up an incoming hole. Anything that has to happen for any outgoing will happen for *every* outgoing, and if anyone is worried about that they simply have to stop using the Internet - having a 3D printer calling out is otherwise no worse than anything else.
You don't really seem to understand basic IT security principles. One of the important ones is you want to minimise both outgoing, and especially incoming connection requests. Those can, and do get abused. A common method is to get privs on a device on an internal device, then jump from there to other devices on the 'trusted' side of the network. The 'smarter' the device, the easier that can be. If the device isn't designed with security in mind, that becomes even easier.
Security gets even harder when users are forced to open incoming holes, especially when many users won't have a decent firewall that allows explicit permit/deny rules based on IP addresses, ports and protocols. Then even harder when the source address range is probably a public cloud service like AWS that's widely exploited by skiddies to launch attacks from. It is almost always better to only permit from 'trusted' addresses, eg ones assigned to the vendor.
And then it's back to why users are forced to do this. I want the ability to have a printer connected to a local server. That server can be on a 'dirty' VLAN, or ideally a dedicated 'dirty' LAN. I can then secure (or try to secure) connections from that segment between public and private networks. If I want to have remote access, I should be able to do this via a trusted VPN. Basically there should be no need or reason to be forced to proxy any communications via a 3rd party server.
Can you show where Bambu are actually doing any of that?
I'm not talking about just Bambu, although this specific issue is a result of a forced dependency. Maybe there'll be future attacks where some skiddie decides it'll be funny to make every printer they can root print dicks. And only dicks because they've rooted the printer. And the printer doesn't have any kind of console access to allow users to re-flash the firmware. But it's a common problem ever since vendors jumped on the 'SaaS' and 'cloud' revenue models.
I sometimes do photography, which often involves going to a remote location to do a shoot. There are costs involved, ie transport, plus models, makeup, assistants etc. Back in the good'ol days, I used to be able to have Photoshop on a laptop. My camera(s) have WiFi, USB and even Ethernet ports, so I could let clients look at previews, or shots as they were taken. A handy thing to be able to do to speed up my work flow, and thus cash flow. Then I figured it would be neat if clients had iPads for this, rather than have them huddled around 1 screen. How naive I was. Even though an iPad has connectivity, to get images to/from them, you're forced to route them via Apple's servers. And then of course Adobe did the Adobe thing, so no offline mode for 'Creative Suite' because you can't possibly be creative without relying on their servers and connectivity.
Oh, and if I wanted compensation for the thousands a shoot may have cost because their Internet was down, that's just too bad. It's a cost and risk of doing business the way vendors like Apple, Adobe etc force you to.
Again it's a widespread industry problem. For example, I thought this looked neat and useful, until I looked at the FAQ-
https://glowforge.com/faq/wifi-and-the-cloud#does-glowforge-need-a-wifi-connection
Can the Glowforge print without a Wi-Fi connection?
No, a Wi-Fi connection to the internet is required to print.
Which being marketing, is of course bollocks. Glowforge has forced that requirement on their users for no good reason, hence I'd never buy that product, however good it might be. This is especially an issue for anyone looking to make money. If you're looking to produce prototypes for a client, or maybe working on a classified design, there will be NDAs. They'll almost always have clauses regarding security and file sharing, which if you've got a 'cloud' dependency, you just can't honor.
So that you can't use your expensive thing unless you keep paying the subscription.
Back in the day, I worked on stuff that involved connecting to (big, industrial) CNC machines via serial cables. Even with my crappy soldering, that was pretty reliable. A device that requires a subscription and I am G0 X<away> Y<away> F<really fast>
Otis invented an elevator that without power would not drop. The default state is clamps locked. When power is applied, magnets release clamps to move.
Default state is safety, and this design is now a common standard.
Bambu clearly doesn't subscribe to this design by setting off end users printers when in a failure state. If anything in the way of obstruction or flammable items (paper instructions etc) had been in the way it would have been a fire, potentially in many households.
I don't trust a company that supposedly designs a safety critical system, yet doesn't follow the Otis principle (if queue fails, don't send messages at all), or tells me they will be remotely monitoring hot end temperatures, from a remote service that has already had multiple outages.
I assume there's local overheat detection, but who knows, safety clearly wasn't though of first.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
