Surely not XP Embedded? :-/
Steven "don't most remote access tools have a 'blank the remote screen' option these days" R
Poor old McDonald's can't seem to catch a break. No sooner had it decided to rename the Blue Screen of Death to "a reboot", more imagery has come to our attention. This time the company's new kiosks were flashing their unmentionables. In this case, eagle-eyed Register reader "Dafydd F" noted the touchscreen in the act back in …
About 17 years ago I did POS support for a restaurant chain that had about 140 franchised units. (not McD's) The company was too cheap to upgrade anything on their POS systems (in both senses of POS), but they did run Linux and SCO Unix at the time. Despite the age of the equipment, which was mostly NCR, with a sprinkling of IBM, it was pretty rock solid reliable and one unit had a system uptime of about 3 years before that was ended by an extended power failure.
I shudder to think how much more busy I would have been (as it was me and one other guy supporting these), if everything had run on Windows.
It's an interesting measure of "progress" that a machine which has been running for 3 years solid would now be flagged as a risk because it has likely not had security patches installed and their requisite reboot during those 36 months.
[I know not all updates on all OSes require a reboot. Just a depressingly large number.]
It IS "progress" that we should not let systems run for years on end without getting patched. If you don't like it, it's probably because you did not design your system to tolerate hardware failures/reboots. More fool you for that nowadays - this age of microservices, containers and clusters where any given hardware node is as expendable as the next (thus your application barely bats an eyelid), if only you leverage the multitude of free tooling to implement it.
This architecture could even work on Windows if you're masochistic enough.
> It IS "progress" that we should not let systems run for years on end without getting patched.
But it's the exact opposite of progress that you can't just get and apply security-only patches anymore. That gives a pretty strong disincentive to keep stuff up to date security-wise for a lot of people and companies.
Not true for POS systems, which are almost always tied to a specific location due to peripherals that are tied to a specific location if not bolted in place.
The best you can hope for is that problems are solvable with re-imaging, else it's screwdriver time.
>It IS "progress" that we should not let systems run for years on end without getting patched.
How does that work with nuclear power stations with a life measured in decades?
The main problem with having systems running for long periods of time is that it really tests the quality of the applications. I seem to remember that many patches over the years have been for applications leaking or hogging memory and generally degrading system performance...
How does that work with nuclear power stations with a life measured in decades?
I'd expect the actual SCADA systems controlling the reactor would be run as a hot failover set of at least two and possibly three systems, so that apart from the actual failure resiliency you could upgrade one, then cut over to another. If you actually had to do so outside of a planned reactor downtime, that is.
If the reactor is cold you can even rip out the entire control system and bring in fresh hardware and software if necessary.
And the stuff that deals with what's outside the dome is roughly similar to what other process industries are using (non-nuclear power plants, refineries, etc.), are probably implemented the more or less same way, and will likely have the same life cycle management.
This is a very good point. If a nuclear reactor near me relies on a system never going down, I should probably move away from that reactor. We need to tolerate system failures if not tolerating them could signify death. It's still a good sign if a system can have a long uptime, as that means the system is probably well-engineered, but if it becomes a requirement that a specific system continues to run without ever rebooting, it's a bit concerning.
There is a difference between relying on a system never going down (ie. unplanned outage) and hence (should have) been designed to be resilient etc. and letting a system run for years without being turned off.
In general I suggest the more reliable and resilient a system is and the bigger the pain of doing an upgrade, the more likely it will be left to run...
Personally, I wouldn't want to be living within a 100km radius or less than 2000km downwind of any nuclear reactor that was running critical systems based on an MS OS and receiving updates every month.
Interestingly, I suspect, if you walk through all the updates to any MS OS (eg. XP, W7) I expect you will only come across possibly one or two that actually fix truly broken OS functionality, that impact the functionality of an application tested on and running on the original OS release.
I've never noticed any traffic lights rebooting...
That's because they tend to be designed so that on any error they'll show red in all directions. Then as the controller has restarted and found itself to be operating normally (which shouldn't take more than a few seconds, so most people won't even notice) the sequencing resumes.
"I've never noticed any traffic lights rebooting..."
You probably have, but didn't understand what you were seeing.
"Are you saying they are insecure?"
They are very insecure for the simple reason that nobody ever bothered securing them. Unless you consider a simple key lock on a street box that is accessible by anybody to be secure.
"They are very insecure for the simple reason that nobody ever bothered securing them. Unless you consider a simple key lock on a street box that is accessible by anybody to be secure."
Indeed, and quite often all lights in a given region will have the same key. I had a friend in high school who found one of the keys in a "hidden" box and started handing out copies to all of his friends.
I agree. For such terminal-only uses I would think that Linux would be the ideal platform.
The only reason I see is that there still isn't enough developers that work on Linux, in other words, it's much easier to find a halfwit that can bungle some code on Windows (not to mention managers that have heard of Linux are still rarer than hen's teeth), so companies use Windows on stuff where Windows most definitely should not be.
I wouldn't have thought there would be much reason not to do something like that as a web app, that way anything with a recent browser from a single board computer (such as a Pi) to a full on workstation costing thousands could be powering the screen. There is the good reason that you might not want a web server accessing your POS system, and I don't know if the credit card companies would have anything to say (after all, there are chip and pin card readers on these machines)
That said, does Linux lend itself to large scale distribution well? Microsoft's System Center is, frankly speaking, a pain in the arse at times, but it does scale well, and it is feasible to set it up to distribute software to thousands of machines at the push of a button.
I know you can control Linux with an MDM solution, but is it easy to push an update out to thousands of machines? Is it easy to monitor that deployment?
Not knocking Linux at all, On the contrary, we have a project I am looking at where we would like to have standalone info screens, and I personally favour a raspberry pi based solution, but have the problem that we have no method in place for managing (including updates) the Pis remotely.
"I wouldn't have thought there would be much reason not to do something like that as a web app, that way anything with a recent browser from a single board computer (such as a Pi) to a full on workstation costing thousands could be powering the screen."
Yes, but web standards change very quickly, and web developers always want to have the newest technology to fail in. Also web browsers are hugely complex systems (more complex than operating system kernels) which are therefore likely to fail in inpredictable ways.
I think the problem is that we do not have propper "graphical multimedia terminal" standards. Sure we have VT100 to which we have added truecolour and mouse support, but if you want to display a photograph or play a sound, your choices are severely limited.
Let's not exaggerate. A web browser install file is a hundred and some megabytes, an OS install is a DVD or three. Not to mention that the web browser does not manage access to the network, nor does it have to bother about how to write to the screen, or printing, or accessing the disk, or storing things in RAM. Those jobs are handled by the OS.
It seems to me that that indicates that an OS is a hugely complex system, much more so than a web browser.
An OS *is* a hugely complex system, but when it only exists to launch a web browser, why make it more complicated than it needs to be? Saying we support X browser at versions Y-Z is much more flexible than supporting a specific OS. The thin client way is valid.
That said, support contracts tend to be for the whole stack, not just the top bit. Reducing the number of systems supported reduces costs and support headaches. Older OSs that are known quantities, that are locked down without internet and the hardware being relatively physically secure makes for an easier life than say, having to know that SVGs don't display correctly in safari.
Let's not exaggerate. A web browser install file is a hundred and some megabytes, an OS install is a DVD or three
Now it's you who's exaggerating. An OS install that just needs to boot up into a GUI, run a browser and some other bits and bobs like drivers for the payment gear, and trimmed to the specific hardware it's going to run on should easily fit on a single CD.
And the original comparison was to an OS kernel. That's massive and very complex, but it is pretty small as disk usage goes. The DVDs usually include lots of much bigger things. For example, you'll probably find the following things on most desktop OS distributions' install media:
1. The default applications, which are probably at least a hundred megabytes. Since many OS distributions include a browser of their own, make that at least two hundred megabytes.
2. Foreign language translation files (not always). Those can range dramatically in size.
3. Fonts. There can be very many of these.
4. Hardware drivers so you don't go through the awful search if a generic driver is sufficient.
5. Extra image data such as icons, desktop backgrounds, etc.
6. Documentation files about various aspects of the system that never get read.
Take all of these away, and the disk usage of the distribution will be cut by a very healthy chunk. There's almost certainly even more to remove before you're left with the kernel alone.
"An OS install that just needs to boot up into a GUI, run a browser and some other bits and bobs like drivers for the payment gear, and trimmed to the specific hardware it's going to run on should easily fit on a single CD."
We were doing all that with Coherent (less a browser) in the 1980s. Substitute curses for a gui, but otherwise much the same to the end-user. Installation for the whole thing, one boot floppy and four installation floppies. If you insisted on a GUI, X11R5 was two more floppies. For the developers, the GNU C compiler system added four more. If you insisted on proper documentation, ghostscript was two more.
Yes, networking was rudimentary ... but UUCP works fine over serial ports (including modems, for your WAN needs). Where is it written that a POS system requires Internet access?
Feaping creaturism is going to be the death of civilization as we know it.
> I know you can control Linux with an MDM solution, but is it easy to push an update out to thousands of machines? Is it easy to monitor that deployment?
I have used Ansible to do that (being SSH based, all you need is a running SSH server on the target and a login, which pretty much every remote administered *nix machine has).
The biggest update push I did was to circa 130,000 Linux servers/workstations at a previous place I worked at. Took about an hour, which is a lot faster than if we had to do it manually (or even write a script to do it for us). The operation is atomic on a per host basis, so at the end you get a summary of success or failures.
Also, outside of the Windows world, it is perfectly possible to update a machine (including Kernel) without needing a reboot. So having a system uptime of years does not preclude the box sitting unpatched. Worst case is you have to restart some services so that updated libraries are loaded.
"does Linux lend itself to large scale distribution well?"
"is it easy to push an update out to thousands of machines? Is it easy to monitor that deployment?"
Yes and yes.
"we have no method in place for managing (including updates) the Pis remotely."
Hire an admin who knows what s/he's doing to implement it?
I'm a big fan of Nerves (https://nerves-project.org/) in that space. OTA updates without downtime if you want to. Downside is that you need to develop in Erlang or Elixir, upside is you get an extremely reliable and cost efficient system.
But yeah, I guess it's easier to stick with the Windows knowledge you have if your primary business is cooking burgers, not IT.
What well-supported Linux windows managers have touch input as standard? Windows has been touch-aware out of the box ever since Windows 8 and touch will be supported well into the future. Any sort of touch input for Linux is going to be an add-on, custom writtenand manufacturer-dependent, won't it?
I worked on retail POS systems a long time back and I remember seeing Linux on devices like bar-code reader terminals in back-of-store locations. Sometimes the display of the custom terminals had a sad-looking Penguiny command-line interface rather than the simplified GUI-plus-keypad the program usually presented to the minimum-wage shelf-stacker droid logging stock changes into the shop's back-office inventory system. The usual solution was to power it off and back on and hope the firmware wouldn't go bonk! again.
I did ask about window managers, Gnome and the like rather than Linux itself. How do they take touch-screen inputs and translate them into HID inputs like key presses and mouse movements, click and drag etc.? Are they functional with touch out-of-the-box or does it require tweaks and configuration of the installation?
The driver presents the touchscreen to the WM as another HID, not unlike a touchpad, so it's not the WM that needs to care in detail what hardware the touchscreen runs on. And for instance Debian 9 comes with about 30 of them, for the diverse chippery involved.
There are two Thinkpad tablets here, an X41t running Mint, and an X61t running Debian. I know there was some wrangling to get an on-screen keyboard running in the X41 (but that was not about the touchscreen itself); the X61 just worked right away.
"I did ask about window managers, Gnome and the like rather than Linux itself. How do they take touch-screen inputs and translate them into HID inputs like key presses and mouse movements, click and drag etc.? Are they functional with touch out-of-the-box or does it require tweaks and configuration of the installation?"
Years ago I worked on POS systems and it was a huge pain. Every POS had to be calibrated and the Linux commands to get that working were annoying (it didn't help that the POS manufacturer couldn't even be bothered to orient the touchscreens the same way each time) so it was with much nervousness that I got a touchscreen laptop 2 years ago.
I hunted and hunted for documentation and couldn't find it, so I tried catting the input device and touched the screen to see what would happen. The mouse jumped to my finger position. It seems every Linux distro for the past few years just enables touch by default.
My local Pret has just had a shiny new POS installed by that lovely Larry Ellison (at a very reasonable price, I'm sure). It seems to work exactly as the old one, except it crashes every lunchtime and requires 20 key presses to ring up each item. #progress
"restaurant chain...SCO Unix"
Pizza Hut? I know when I was a PFY, the Pizza Hut I worked at had 386s running SCO as Point of Sale. This store opened in 2001/2 I don't know they got a hold of hardware that was ancient even back then.
Years later they eventually upgraded to touchscreen units running XPE.
SCO was very useful back in the early 90's, it allowed the deployment of commodity hardware into locations where such hardware stood a reasonable chance of being repurposed, given that whilst PC's were 'cheap' compared to pure Unix boxes, they were still expensive for Joe Public...
For some reason putting an unfamiliar or non-MS OS on the box massively reduced the likelihood of the box being either relocated or repurposed to run games etc. - massively increasing systems reliability...
My Taco Bell back in the 90s had a Unix machine in the office, which I am almost certain ran SCO, connected to the Par register system. In the late 2000s I did some call-out IT work for a McDonald's contractor. In each office I found two machines, one running Windows Server 2003 and the other running some flavor of *nix (not sure if Linux or good ol' System V Unix,) with some really cool integration between the two.
There were tools, which I cannot remember the name of now, that allowed one to remove, or at worst permanently disable, all the unwanted crud that came with Windows 9x. Doing so one had an OS that was actually rather fast and stable and as long as the drivers were of an acceptable quality (WHQL? tested, and you could also run the independent test tools for further validation).
The eventual crash based on elapsed time was usualy due to the integer size/wrapping of the boot tick timer. It didn't necessarily cause the OS to crash, but some applications tended to fail as a result of it and on occasion these included drivers. I remember having to write code to detect when close to the wrap-around time and to extend the counter programaticcaly to ensure that the counter (timer) based events were still triggered as expected.
In a previous life I worked on a fancy card machine for the US market that, for PCI compliance purposes, needed to reboot every 24 hours.
Except what we then found was an influx of logs, configuration and wallpaper (told you it was fancy) downloads all at the same time when thousands of devices came online at the same time at Xam in the wee hours their time zone.
So I had to build a random factor, a random number between 0-3600 as an offset to try and stagger all of the device reboots across an hour.
Except that then meant the possibility of a device being alive for 25 hours, an hour more than compliance allowed.
So this random factor needed to be generated one time and persisted in permanent storage (and later included in the configuration received after reboot), such that it rebooted at the same random time within the hour after Xam every night.
Most US-based companies use embedded Windows for these applications, I'm not sure why... Perhaps major US corporations would rather have the backing of another major US corporation over the community of Linux developers. I mean, the US military even runs their warships on Windows.
On another note, McDonald's might just be rolling these things out in the UK but they've had them over here for over 4 years and have largely worked out the issues. It's surprising that they'd have so many issues over in the UK, maybe they hired the wrong IT contractors.
XP Pos probably, which is essentially a lovecraftian love child of xp embedded and tablet SKU's with ongoing patch releases, has better touch support and will run on the cheapest nastiest components you can find, and there was a panic buy of thousands of licenses when it was announced as end of life, because most of the POS software vendors in use have prohibitavly expensive upgrade clauses to bump OS version.
That said the McD's kiosks are some of the worst in the business, the screens are terrible and dont respond, then again thats why you usually dont see screens bigger than 18", but they have 32" panels
Used to build a shitty kiosk system in use for cashless catering in use at many high street name bank hq's and other large enterprises, whole sector is about 10 to 15 years behind the curve, and thankfully the rise of tablets has left a lot scrabbling for new business models and failing in there incestious swamp of chummery
I supported a touchscreen kiosk of training courses in Job Centres and similar venues 20 years ago. Those had 21" Illyama CRT touchscreens (they were bloody heavy) and other than recalibrating during my quarterly visit they were responsive and required no maintenance. Unlike the terrible HP printers also included in the cabinet which would usually stop working roughly 5 minutes after I left a site.
Ahh, touch sensitive CRTs.. We used to have one where I work (in fact, i think we still do in a store cupboard somewhere). 17inch Phillips job.
Was bought for one specific project, which never came to anything and ended up just floating around in the labs being used as a normal monitor. That said, I installed Quake on the machine, and it was brilliant.. To kill something, you just needed to touch it.
There were some Philips touchscreen CRTs that had a foil over the screen front. It had a mesh structure that you could distinctly feel, with a raster size of about a mm. A serial port was used for touch data.
Some displays for the HP150 used a rather coarse IR grid in front of the CRT, with LEDs and IR photodiodes fitted in the bezel. You could select without even touching the actual screen.
Tower Records had infrared grid touch-screens in the mid 1980s. I had the contract to service them in the Bay Area for a year. Removing the gunk from the detectors on the bottom of the screen was a near weekly necessity. Awful, awful job that required near constant driving from location to location. Only contract I ever lost major money on.
That said the McD's kiosks are some of the worst in the business, the screens are terrible and dont respond, then again thats why you usually dont see screens bigger than 18", but they have 32" panels
You should try the KFC / Burger King equivalents, at least McD's have tried to make theirs usable, the others have a horrible mess of UI and an excruciatingly slow process to be fought through.
Still, it reduces my likelihood of wanting to go in any of those places, so every cloud and all that.
I used to work for a company that serviced PoS systems such as these. As someone that was seen as little more than a "remote controlled screwdriver" I can attest this is pretty typical. I believe that the kiosk is in the middle of a re-imaging procedure. Its very typical for these to have extremely high levels of automation that include plaintext passwords, but these passwords are typically only used in very specific scenarios during set-up, then the set-up script will change them to something unique at the end of the configuration process.
Most PoS systems receive regular abuse with being powered off incorrectly and therefore frequently get corrupted. In some locations where employees were particularly careless I would have to re-image a terminal ever other month until management caught on to the extra expenses form repeated repairs not covered under the service contract. (Software corruption not due to hardware failure was generally not a covered repair).
Additionally, these systems are locked down so hard that its usually impossible to run normal repair tools such as SFC and Chkdsk, making a re-image needed in any scenario where there is software corruption.
The BSOD in the other article looks to be on a KDS (Kitchen Display System) controller or similar piece of hardware. These usually have a version of Windows embedded that is stripped of every unneeded file and service so that there is no way to even access a desktop. In fact, they may not even have drivers for a keyboard installed they are locked down so hard.
Al of these devices run some sort of stripped-down, embedded version of Windows. Most are on PoS Ready7 now, but some older ones are still running PoS 2009 (a.k.a. Windows XP Embedded).
Linux is almost never used, unfortunately. Microsoft has bent over backwards to include special, not well advertised features in Windows Embedded versions. For example, look up XFS, and I don't mean the file system, this is eXtensions for Financial Services. XFS was pretty much created solely to entice ATM vendors to Windows and away from OS/2 instead of switching to something like Linux, Unix, or BSD.
When I first saw systems being called (with a straight face) POS, I laughed and wondered what marketing genius managed to sneak through naming a class of systems "piece of shit". Because that used to be the only thing it meant.
Then I had to work on them, and learned that I underestimated that person's accomplishment. Those systems are both "points of sale" and "pieces of shit" at the same time!
One better, I recently shutdown a remote system when the payroll officer was using it! A minute later, I visited the office, they were just putting down the phone(!) The coin dropped immediately... the look on my face must have been priceless! They had added a working day to their schedule, 3 days a week to 4!
I must have missed the memo regarding them changing their working days!
mcdonalds uses software called "newpos" on their terminals/touchscreens the clue can be found in the domain the kiosk is on "newpos till" which includes remote support capabilities and runs on windows xp embedded here is a picture of the pos with an error screen: https://imgur.com/r/PBSOD/V5tELUs the version of newpos they are using in that error screen is 6.1 (which is the latest version). newpos is the same software they use on the counter terminals but with a different interface. before the switch over to newpos they used dos software called "pcpos"
I dont work in retail/industrial tech but I like to spot cracks in the facade, like watching the seatback screen on United flights boot Linux. It's incredible how long these systems last for.
My local supermarket here in an Atlantic seaboard state is called Giant, and I swear they had OS/2 underneath their POS up until VERY recently, maybe 18 months. I saw the charmingly clunky font and window chrome and it looked exactly like Warp Presentation Manager.
As for SCO, if they hadn't impaled themselves they would have been remembered for a truly solid POS UNIX.
Ah, the 90s. I miss em sometimes.
I can answer the questions here:
The normal tills use either XP POS ready 2009 or Embedded standard 7 for newer ones, and the kiosks use embedded standard 7.
The back office though is a whole other beast. The store server runs some windows server sku, the store I worked at ran server 2008, but the actual interface for the back office work (GUI is a generous description of it) is from 1985 and probably runs in some sort of dosbox type thing.
McDonald's software development basically involves just piling stuff on top of other stuff. The kiosk software is basically just the till software (called NewPOS) with a new GUI and some modifications thrown on top. And newPOS itself is just some horrible monolith which takes about 15-20 minutes to start
I volunteer in a charity shop and our software till went TITSUP fairly recently and we were on the phone to support who was telling us about the reset button. It was apparently under a flip up door. We could not find it. Because someone had previously removed the door, doh! I've watched as a PFY has logged in remotely and debugged things.
The till is misbehaving a bit, adding digits so a 99p sale becomes £9,999. I had to get the manager (with appropriate till clearance) to remove it. Fortunately the customer saw the funny side. Between it slowing down so much you dare not press the numbers too much lest you ring up such figures. I think the biggest sale I ever made was $36. Though we did sell a donated ride-on kiddy car for £200 in the summer.
Then there's the guy who throws a strop if we put £15 on a really nice leather jacket or a pair of fashion trainers as though we're a pound shop. We have a responsibility to get as much as we reasonably can and £15 for a leather jacket is an absolute bargain in anybody's book.
Biting the hand that feeds IT © 1998–2022