Please check your data: A self-driving car dataset failed to label hundreds of pedestrians, thousands of vehicles It's a long weekend in the US, though sadly not in Blighty. So, for those of you starting your week, here's some bite-sized machine-learning news, beyond what we've recently covered, if that's your jam. Check your training data: A popular dataset for training self-driving vehicles, including an open-source autonomous car …
> Thousands of vehicles, hundreds of pedestrians, and dozens of cyclists were not labelled.
It seems to me that there are two sorts of mistake. There is labelling something incorrectly, such as labelling an elephant as a bicycle. And there is a different sort of error which is not labelling something at all.
In the first case, mislabelling obviously leads to errors in the system that is trained on the data. But what about the second?
If something is not labelled, then it doesn't form part of the training. It is as if the unlabelled object doesn't appear in the image. ML systems are not trained to identify every single thing in an image: every leaf on a tree, every piece of litter. So things that are not labelled will simply be ignored: not forming any sort of association.
So the question is: does it matter if some images don't have some things labelled?
Yes it matters. Particularly with [semi]autonomous machine learning. Such a system misclassifying an object as "looks like, but isn't" puts that object squarely in the path of an autonomous vehicle seeking to avoid a collision with another object clearly classified as "must avoid".
We generally don't know exactly how a neural net builds it's recognition matrix, so it's conceivable that it might pick up on some minor/irrelevant detail common to untagged objects in it's training data and end up putting all such objects into its ignore bin.
The real fun is with edge cases such as the occasional unicyclist, trike or recumbent bike. A system trained on data containing untagged objects offers the real and dangerous possibility of a "bin" for objects that don't neatly fit into any of it's trained classifications.
Um, yes... if it's something that should be seen, like, say, a pedestrian or another vehicle: the purpose of that training set is (was?) to allow a categoriser to identify things and see how closely it matches the training set.
Would you like to be the one in three pedestrians that the system doesn't know it can't see because it was never told about its error?
> the purpose of that training set is (was?) to allow a categoriser to identify things
Yes, we agree. And as long as those same things appear in other parts of the training set, it doesn't matter if they are missed out in one or two single frames. The AI will still "learn" them from the other frames where they ARE labelled.
The article neither says nor suggests that entire classes of object (all traffic lights or all prams, for example) are systematically omitted. Missing from the entire set. Just that the occasional thing that is labelled in most frames has not been labelled in a few.
The dataset needs to contain a wide range of objects if we expect it to recognize them in all circumstances. What if it only contained images of pedestrians wearing black coats, or with long hair? It may never recognize one with a green coat or a bald person. So the absence of labelling is an issue in that it can deprive you of significant diversity in the dataset.
I am sick of hearing of this scum bothering people that are actually trying to make a product.
You have a patent ? Good for you. Are you making anything with it ? No ? Then shut up and fuck off.
Patent law should be changed to include an article that states that a patent is valid only if the patent holder can justify that product is being made and sold using that patent. Doesn't need to be the holder, who can grant usage to whoever he likes, but product must be made or you have no right to complain.
That would clear out quite a few portfolios that are in zombie mode right now.
Patent law should be changed to include an article that states that a patent is valid only if the patent holder can justify that product is being made and sold using that patent. Doesn't need to be the holder, who can grant usage to whoever he likes, but product must be made or you have no right to complain.
Made ... or at least in active development ...
... but what do you do about the inventor who has a brilliant idea but can't afford to develop it and is looking for a partner or sponsor to help with the development?
I agree that it's not helpful to patent something and just sit on the patent and use it to prevent anyone else from developing any product that it might be thought to cover. That's not in the spirit of the original idea of patents, which was to ensure that ideas were published, but to allow the original inventor some opportunity to profit from an invention before opening it up to all and sundry.
The DOE’s Advanced Research Projects Agency-Energy (ARPA-E) will be particularly hard hit: not only does the proposed budget effectively eliminate the agency, it must pay back $311m to the treasury.
Even though the agency was proposed and first funded under Bush it didn't open its doors until Obama had just been sworn in. That fatal association, plus the fact that it insists on looking at new energy technology rather than just saying "Coal good, coal good, oil good, coal good", means that Trump has to shut it down. The only surprise is that it took him so long.
Whenever I've encountered a "check you're not a bot" test on logging into a website in the past few months, I've always been asked to select the images that contained a pedestrian crossing or a bus, or some other street based scene. If this is considered a good AI defeating test, then it doesn't say much for the prospects of this use of the technology.
And they're all from the USof A, so some of the bits of traffic lights or whatever that are seen in the corner of an adjacent square may be hard to recognise this side of the pond. Most UK traffic lights are on a pole; many in the US hang from wires. Where does the 'traffic light' stop and the network of supporting wire begin? Pedestrian crossings have some differences, too.
> There seems to be one missing word and one erroneous apostrophe in the extract above.
Dear WhileI'mHere,
Thank-you for bringing this to my attention. I am the lawful and legitimate owner of the genuinely innovative and original invention known as "someones bungalow". This attempt to subvert my patent by adding an apostrophe is clearly just that and I have instructed my attorneys to sue for infringement.
I also own the patent for "subverting patents by insertion of extraneous apostrophes" and I have instructed my attorneys to sue for this second infringement.
Please do not reply as I might have to sue you.
ha ha 2=2=5, nice try. But prior art.
My Dad used to visit a bungalow, and as he popped inside, we'd be left waiting in the car for what seemed like hours (probably 20 minutes). Eventually one of us asked who was in the bungalow. "Ah, just someone i know" he'd say. Always seemed to set off his asthma too.
to the rest of us. Sadly our universes have a junction that surrounds him. He only sees his version of reality.
Spending billions on something that is already pretty well sorted yet calling time on the US Military Newspaper, Stars and Stripes at a cost of $15.5M per year shows just how out of touch he is. Never mind, he'll be off spending another $5M next weekend to go and have a round of golf.
Surely in a proper capitalist country, The Market should be relied upon to make this investment. This government funding smacks of (shudder) Socialism to me.
If other companies can't keep up with the market leader, then they deserve to wither on the vine.
Could actually be bloody lethal, as could using an educational training data set for (even experimental) production. The sloppy thinking that drives web development has clearly migrated to the life-critical development domain. In the words of Private Fraser "we're doomed".
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
