Testing times: Between some IoT code and a hard place

Every company has its ups and downs. Those downs could be exploding phones or a sudden unmovable overstock of Clinton merchandise (or conversely an uptick in Trump-the-statesman t-shirt demand). Bigger organisations can better absorb the illness of a member of staff or a surge in demand beyond all expectations, although no one …

[bazza]

Some OSes like VxWorks have long had nice facilities for post-mortem debugging. With that you can write your software to log various things in a circular buffer, as well as a ton of OS / task runtime scheduling data. If it ever goes bang you simply connect the debug environment (or extract the buffer content somehow) and see what had happened in the time leading up to the crash. Very nice. And being a nice, tidy, tiny OS makes it well suited to an IoT device.

And you can achieve similar results with Linux these days; stuff like FTRACE and kernelshark spring to mind. Maybe Linux needs more resources to run than VxWorks, and this matters a lot for price conscious consumer products.

These are all nice things to have, but in microcontroller land we don't really get an OS. We know we want that kind of post mortem capability, but it's a pain to have to develop it all oneself.

[Anonymous Coward]

Back in the days when ICE was an expensive indulgence ( ~ 30 years ago [ aarrgghh ] ), I remember cobbling together a remote reset facility so that if a test in a suite running overnight locked up the board you could record the fact and move on to the next one rather than having to restart the whole thing in the morning. Essentially our test host sent a command to a separate piece of kit whose one job was to turn the power to the board off and on again - just be thankful that computers are so much more sophisticated nowadays.

[Anonymous Coward]

Any process that runs autonomously for a considerable period needs a watchdog.

I have Excel VBA analysing web pages' contents via Selenium. At various points in the code a UDP message is sent to a VB.Net watchdog application in the same PC. The message indicates what is being processed at that instant.

If the watchdog gets a message with the same state/data information several times in succession then it assumes the current page's HTML has triggered some unexpected loop.

If the watchdog doesn't get a message within the allocated time then it logs the last message's state/data.

In both cases it kills the Excel instance. It then restarts Excel to continue with the next page - having marked the failing page for a retry later.

If the Excel run completes then the watchdog puts the PC into hibernation mode - and the log has recorded any anomalies..

There are obviously other OS related failure modes that the watchdog wouldn't catch. In several years it has never been hit with a situation that the above didn't circumvent automatically.

[Bob Rocket]

Sell me one

I understand it is a product still in development, that there will be endless code updates and occasional track cutting and soldering on odd parts (and later unsoldering as that wasn't the source of the bug), this is the fun part of product development.

I know my way round a schematic and been playing with flash based microcontrollers since their infancy (try debugging on ceramic windowed evaluation chips).

[Brian Miller]

Testing is only effective with effective people

All right, look: if you have an incompetent group, testing doesn't really mean anything.

The group has to contain competent people in order for the whole development cycle to succeed. If you have idiot developers and idiot testers, you have trash output. If you have bright developers and idiot testers, you have someone rubber-stamping code as it goes, with the product being spotty. If you have idiot developers and bright testers, then the testers find all sorts of bugs with no hope of them being fixed.

The only time this works is when there are enough bright people in the group for the entire product cycle. You have to start with good design and architecture, good implementation, and good testing. Fail in anything, and the product is going to have serious problems.

When it comes to embedded devices, the testing can be horrible. Imagine your company white-labels a device. You receive the device, test it, and it simply falls down flat. So you send the bug reports, and can only hope for a fix. And of course, the device doesn't have any interface for automation. Maybe it's embedded Android, or maybe it's embedded who-knows-what. All you know is that you can't humanly scratch the surface of the bugs that are there, yet because your company's name is on it, you're on the hook for what that thing does.

Yes, it would be great to have an AI harness. But of course, all of that has to be set up. And you still need competent people to do that.