How a Kaggle Grandmaster cheated in $25,000 AI contest with hidden code – and was fired from dream SV job A Google-backed competition to develop machine-learning software to help abandoned animals find loving homes turned ugly – when it was revealed the winning team cheated. PetFinder.my – a non-profit that lists profiles of dogs, cats, and other small creatures for people to adopt – devised the contest to “save homeless animals …
One corollary of this is the AI can detect which animals are never going to get adopted and they can euthanize them early to save resources...
...is an example of when AI can actually be scary, because their models presumably literally do predict which animals will be put down and it only takes a human to decide this is the logical outcome.
Not to sound glib about it, but who's to say there isn't someone making those decisions in animal shelters already? They don't run on infinite amounts of money, after all. One would hope it doesn't happen too often, but animals must get put down for economical reasons, and part of the decision about which animals get that fate would presumably contain at least an element of whether they are likley to get adopted or not.
Given that most repsonsible shelters are going to be spaying the cats and dogs that pass through their doors (you know, to combat the problem of irresponsible pet ownership which is the cause of their needing to exist in the fisrt place), the decision may well involve determining whether it is worth getting an animal neutered when they come into the shelter.
Long and short: folks, unless you are planning on breeding your pets, get tose cats and dogs spayed. Cats especially; the songbird population will thank you by continuing to reward you with a dawn chorus.
There was a Star Trek: Voyager episode, Critical Care, where care was allocated on a "Treatment Coefficient" in a hospital that looks a lot like how AI could go wrong.
..if you want really scary, consider the sort of parallel decision-making process that goes on in hospitals, especially when the resources fall short of those required to treat all patients.
In the US, we use a much simpler decision matrix:
Does patient have a shit-ton of money?
YES: Lives; potentially bankrupted by the cure, but that's their problem.
NO: Too bad, so sad.
The solution to that isn't death panels or a death algorithm. It's an algorithm that intelligently raises the tax rate or shuffles some dosh from one service to another.
Remember: the AI crunches the numbers, WE set the priorities on how we act on that data.
"Mrs. Marjorie Smythe-Snow is in and out of hospital twenty-four times a year. She's 101 years old and can, on her good days, tell you about every war and global crisis she's lived through. The algorithm says she's going to cost a lot more if she keeps on ticking. What do?"
"Obviously we remand this drain on society to hospice care!"
"... No. In fact, so no that you're fired, security, get this clown out of here. No, what we do is raise about... punching in some numbers here... One cent in tax for a month should cover it, or we can short the city of Dover's cleaning supply budget."
Your comment made me incandescent with rage. It comes from a 1970's understanding of dog behaviour. Fortunately there has been a lot of research into both the psychology and the physiology of dogs in the past 50 years. My own research into the subject has concentrated on males as I had a male pup. But some points are applicable to dogs of both sexes
1/ Responsible owners of pedigree animals do not allow random mating's to occur
owners of male dogs can tell when they scent a bitch in heat and will put them on a lead. Responsible owners of female dogs don't walk them in public parks etc when they are in heat
2/ Early neutering of males used to be seen as a responsible option. This is no longer the case, dogs should not be neutered until they are fully grown, early neutering will increase the risk of joint issues, hip dysplasia arthritis dues to joint wear and tear etc. in larger breed dogs they are not mature until 2 years old. By this age most dogs will have finished training and so neutering does not bring any benefits
3/ Neutering was believed to reduce the risk of cancer in dogs, this is now shown not to be true, obviously removing the reproductive organs will prevent that cancer but research now shows it will increase the risk of other cancers
4/ Neutering of males was thought to provide a docile dog. In reality juvenile dogs often suffer periods of fear, its a development phase, neutering during that phase will 'fix' the dog in that state
5/ neutering was thought to prevent 'humping' this is a juvenile behaviour which can be trained out. Dogs are either trained to stop hunting or carry on regardless of whether they are neutered or not. Female dogs also hump it is driven as much as dominant behaviour as sexual.
6/ there is a small but real risk that neutering a dog may result in personality changes. I have read heartbreaking stories of males who have changed completely after castration. Anyone who has owned a dog knows they quickly become a valued friend and family member, you love your dog for who He/ she is, you know their personality and quirks, seeing them radically change is akin to losing them.
There are male dogs who become distressed when they scent a bitch in heat and may not be trainable to behave appropriately but these are a small minority of males.
Yeah no, while some points have some merit, you assume responsible is the default state of pet ownership.
It isnt, given the stray cat problem every where, or the dumb asses like my neighbour who get a kitten then let it go feral because it scratches carpet, and havnt bought cat food in 3 years as evidenced by our shared bin and recycling boxes, and ridiculous get rich quick puppy farms i feel that the responsible thing to do is to neuter by default, and only allow non neutered animals to legit licensed breeders, not that would ever happen...
THIS.
Here in the US, a lot of the major cities have a serious feral cat problem. One solution is to trap, neuter, and return (TNR) the cats to the area were they were captured. It stabilizes the population, reduces propegation of disease, and it works.
@PM from Hell: "1/ Responsible owners of pedigree animals do not allow random mating's to occur"
There are no responsible pedigree owners, because the entire concept of pedigree is a hideous affectation. There are working dogs from good stock, and then there are designer abominations with hereditary conditions. The latter are 'pedigree'.
Meanwhile, my first two rescue dogs came into being because the owner of the mother deferred neutering, and then said poor mutt was impregnated by the neighbours German Shepherd who destroyed a fence to get to her. Maybe if you keep your 'pedigree' dogs in some kind of prison you can remove all risk, but it's no life for the dogs.
Now, I'm on my sixth rescue, with a current pack size of four, which is as many as my wife and I can look after. All are neutered, and none suffer from it. The current oldest just had her 14th birthday, and one of the first two pack members was 15+ when he died, which is pretty good innings for a 40Kg dog. See, I have anecdotes too.
1. if you are "incandescent with rage" at anyone spaying and neutering, I know one animal that may need to be neutered to cope with intractable personality problems. Yes, I assume you are male, that is the human group most uncomfortable with the procedure
2. you are harping on about "responsible pet owners". Would that there were more, but we cannot even get responsible voters, see the current US and British heads of state.
3. there is "research" that proves the Earth is flat and that vaccines cause autism.
Long and short: folks, unless you are planning on breeding your pets, get tose cats and dogs spayed. Cats especially; the songbird population will thank you by continuing to reward you with a dawn chorus.
Was ready to upvote you for the first sentence, then downvoted you for that second, provably false (sub-)urban legend second sentence.
But thanks for playing...
Sorry, are you claiming that cats don't predate birds?
Yes, there is the argument that "birds die anyway", but that argument only holds true if the ecological niche filled by birds is full (in which case, the stongest outcompete the weak).
Just because it's hard to demonstrate that there is a causal link between predation of birds by cats and decline in numbers (correlation doesn't equal causation), doesn't meant here isn't a link.
If you went round randomly shooting people and used the argument "lots of people die anyway", I don't think it would hold up to scrutiny, any more than the "no evidence that cats cause a decline in bird numbers" argument.
The RSPB did a study a while back that concluded that cats don't affect bird populations much as they tend to only capture prey that would most likely have died soon anyway.
https://www.rspb.org.uk/birds-and-wildlife/advice/gardening-for-wildlife/animal-deterrents/cats-and-garden-birds/are-cats-causing-bird-declines/
If you were a feral cat, which dinner option would you choose: Ground-dwelling small animals you can actually compete with (mice, rats, squirrels, moles, etc.), or birds with lightning-quick reflexes, able to instantly fly away?
Yes, yes, the young ones, trapped in their nests. But here too, why bother climbing all the way to a lone, shaky branch of a tree, instead of just going after the litter of mice right here on the ground? Opportunity is indeed the decision-maker, and individual situations might vary a lot, but the generalization of "cats feed on birds" is a silly "Tweety vs. Sylvester" grade stereotype...
And yet, you have weirdos like my wife. When we went to pick up another dog, I found some really nice ones that I liked. She picked out the ugliest, meanest, unhealthiest looking cur that she could find there. It was apparently massively abused, nearly blind and limped a lot. It had cigarette burns on its skin and matted fur. My guess was that it was headed to the chamber that evening. But the little rat of a dog lasted 2 more years in our house. I wanted one that wasn't broken. She wanted one no one else did. Guess who won.
The dog.
The AI will never account for those types of decisions.
There's a rather excellent book by a rather excellent author which precisely skewers this syndrome, via the literary device of an independent robot. "Amusingly" one chapter for which the book nearly got banned, is now the default behaviour in American hospitals. It's just simple logic, after all: if you create a system/structure a certain way, it will tend to end up... at its logical conclusion.
Tik-Tok by John Sladek.
Unreserved recommendation, as indeed for all of Sladek's works (Roderick!), for sheer intelligence if nothing else.
(great Amazon review here, worth reading in full :
Tik Tok is two ideas in one slim fast read:1 - Asimov's laws don't work (and never could).
2 - People are idiots.
Sladek launched his career in the SF New Wave of the sixties, and spent most of his time examining Robots and AI -- in fact in the mid-nineties he was a technical writer for a software firm.
)
What worries me the most, is that the code was available to view and the cheat code was not quickly identified. I think this is a major problem with software development today, that code is so complex with multiple layers of dependencies, people simply pick up code blocks and just assume it is secure and functioning correctly. Even open source code is not guaranteed safe until thoroughly reviewed.
Many of the amazing AI advances in recent years just have poorly paid workers in cheap locations pawing through your data.
Expensify could magically tabulate your receipts. They were just flogging it out to fellow humans on Amazon Mechanical Turk.
Spinvox was using the power of AI to turn your voicemail messages into texts that could be discretely read, using human workers. (19 years later another startup called "Vxt" is recreating the same idea.)
https://www.theregister.co.uk/2009/07/29/spinvox_mechanical_turk/
There's a tiny robot that drives around Berkeley doing deliveries. Kiwibot drivers are sitting in Colombia.
Unless I am very mistaken, the source code developed by competitors is not public (it could not be during the competition and neither is after). Many of our students use kaggle competition and datasets and try to reproduce the results as part of their curriculum, we would not allow that if the algorithm were already known.
But is is possible that the one who provided the data for the competition have access to the programs at the end.
I'm not sure which aspect of this report causes me the most despair.
Is is that someone genuinely sat down and decided that the way to solve the problem of abandoned pets was to artificially enhance their social media profiles?
Is it that whenever you make an ordered list of people, some people will be determined to be first, regardless of the nature of the ranking or the methods employed.
Is it that those same people are being hired on safety-critical projects, like driverless cars, seemingly on the basis of their ranking on an easily-manipulated high score list?
Is it that I'd never previously heard of Kaggle and am so clearly out of touch with current trends, yet can't really seem to care?
Just take me behind the barn and shoot me.
"Is is that someone genuinely sat down and decided that the way to solve the problem of abandoned pets was to artificially enhance their social media profiles?"
Definitely this one for me. The rest is just a bog standard machine learning problem combined with the usual result of the current trend of gamifying everything and turning into a competition with a top spot. What gets me is not just the idea of lying about pets in order to get them adopted faster, but the idea that there is even a problem here that needs solving. Some animals are less desirable as pets than others. A mangy arthritic mule is not going to get adopted as quickly as a playful fluffy kitten. Short of just claiming the former is actually the latter and hoping no-one notices when they come to pick it up, there's nothing you can do about that. Humans cheating in a competition isn't something anyone should be surprised by. But I'm am very confused why anyone would think the fact that nice pets are preferred to shit ones is something that could be fixed by throwing machine learning at social media optimisation.
"There are people in the world who will choose to adopt a less attractive animal out of compassion."
Yup. One of my cats has severely damaged back legs and hips from an accident when he was a stray kitten. He's also an unpopular colour, since black cats are deemed difficult to photograph by the Instagram generation. Then there's my other cat, an emaciated, flea infested, pee soaked mess when he first appeared in my yard. Both are now healthy, happy and bonded to me.
@james_smith: one of the best cats I have had survived a serious RTA, and was taken in by the RSPCA. When I saw him he was on his own in an enclosure - all but blind, a couple of wonky legs, going around in circles, brain damaged and unable to use a litter tray. He'd been available for adoption for several weeks despite being a beautiful ginger and white with huge eyes (due to the blindness. I just had to have him, and was rewarded by sharing my life with the happiest, funniest, most loving person* I have ever met until he died eight years later.
*Yes, I count some of the cats I've had as people.
It's not clear that "lying about homeless pets" was considered. Presentation evidently is a factor, an appropriate photograph... vaccination was mentioned, so if that's important to prospective owners then it may be worth doing for animals in waiting. (They may or may not get a different injection instead if not adopted.)
My local newspaper has a "homeless pet of the week" feature; often these are "rescue" animals and they variously need to be in / not in a family, kept indoors / outdoors, with or without other animals, things like that... but another option, perhaps, is to foster-home unowned pets temporarily and make them more sociable for rehoming.
”the idea that there is even a problem here that needs solving.”
Shelters are effectively in competition against other shelters to offload the animals before they get killed. Also, some potential pet owners may look but not adopt.
Both these things mean improvements to the way a dog is presented are worthwhile for the shelter.
I'm unsure where you got the lying bit from. The way information is presented can have a big impact.
Is it that those same people are being hired on safety-critical projects, like driverless cars, seemingly on the basis of their ranking on an easily-manipulated high score list?
Well, the high-score list shouldn't be that easily manipulated, the idea is that a test set is held back and your algorithm is tried on it blind once at the end of the contest. The mistake here is getting a bit lazy or too trusting and not properly keeping it under wraps. A bit like leaking the test answers on the internet.
"The goal was to create an algorithm that could predict how quickly a pet would be adopted based on its profile details, from its photo to its breed, sex, size, age, and whether it had been vaccinated or not."
Yeah, didn't we used to call that Statistics? Didn't need anything "artificial" to it, it was just confidence-based Mathematics. Computers were just there to do the calculations more quickly.
> Is is that someone genuinely sat down and decided that the way to solve the problem of abandoned pets was to artificially enhance their social media profiles?
Have you seen this recent story about a 12-year old in the US who hand makes bow ties for dogs in animal shelters because it helps them get adopted more quickly?
That's a very good point. I have some sympathy for Pleskov, I've been in environments where 'winning' was fetishised to the point that cheating, lying, and sabotaging others was practically encouraged.
That said, this was no such an environment so that feeling was down to him alone. It does sound like he's learned that lesson though, the hard way.
"I've been in environments where 'winning' was fetishised to the point that cheating, lying, and sabotaging others was practically encouraged."
This is a well proven human trait (I'm sure there is likely some name for it). If you reward the individual e.g. with bonuses, then many will go that way. If you reward the whole group involved, then they work together.
This is way many sales people are utter wankers. Companies reward the individual, whereas if you have say a profit share scheme, you get fewer arrogant self righteous pricks.
I understand what you're saying, but in this case - the gold medal is accompanied by gold.
Not so much the Kaggle gold, but the extra work and pay that the #1 position yields.
So it isn't necessarily an ego thing as Pleskov says as it more likely is a financial thing:
Kaggle Grandmaster, hire me!
The dataset used was biased. They used previously public information from a related source. If they had used undisclosed information, it would be clear the winners' code would do poorly. This guy was actually intelligent to predict WHERE the dataset would originate, and use that to cheat his AI.
Volkswagen used the same principle on their cheating diesels: they knew the test conditions beforehand, and how to detect them.
What I think is: people testing AI must learn to use undisclosed sets of data, and not to reveal their test methods.
Like, using Nürburgring to test how fast a car is. Any vehicle can carry a hidden GPS, and unlock a self-destruct mode that allows the car to go faster than anyone else's at the cost of durability, just to win the fastest lap in a vehicle that couldn't possibly hold that performance for long, and still blame "misuse" to avoid warranty issues if one of said vehicles ever broke down.
In fact, I heard of some manufacturers having speed limiters that would automatically unlock if the cars were near the Circuit of Suzuka, or any other known circuit of a list, as a positive feature, so it is totally doable.
There's a number of cars that use that technology. I think it first came to my attention in connection with the Nissan GTR.
The old version of a car knowing a test condition was dyno testing for emissions way back in the bad old days--if the car had a small light under the hood (bonnet) ostensibly to help you see to check the oil or something, it could just as easily be tied into the undoubtedly crude fuel injection system to make the car run cleaner, for example. Cars were emission tested on a dyno to simulate load, and invariably had the hood opened for connecting to the ignition system to measure RPM.
I think of that line of thought, I need a car that can measure when radar or laser signals are picked up in the vicinity, and slow me down enough to keep me from getting written proof of my car's performance.....
"Volkswagen used the same principle on their cheating diesels:"
Why don't we stop with referring this issue back to VW all the time - Opel, Chevrolet/GMC/Buick, Daimler, Fiat Chrysler, Mitsubishi, PSA (Citroen and Peugot) and Renault have all been found to be doing something similar.
I've little doubt those manufacturers not on the above list are just yet to be caught cheating, as opposed to not actually doing it.
Using VW is just a shorthand. We all know that most, if not all, of the others are doing the same.
I knew someone at Ford who said that they knew VW were cheating, as they all buy and test each others' new models when they come out. However they could never quite figure out HOW they were cheating.
In Japan all domestic cars are limited to 180kph. Until 2004 they were also limited to a maximum 280ps (275hp). Except for a couple of exceptions, the speed limit on Japanese motorways is 100kph (62mph), and a maximum of 60kph on other roads unless a lower speed is posted. All Nissan did was to remove the speed limiter if the car was deemed by GPS to be at a place where it is allowed to go over the normal legal speed limit. So not so much cheating to get extra speed, more like just allowing the car to perform as it was intended.
> All Nissan did was to remove the speed limiter if the car was deemed by GPS to be at a place where it is allowed to go over the normal legal speed limit.
That's how it was originally marketed, but the feature was never implemented. In reality, Japanese owners who wanted to race, had to take their car to a dealer "to be checked" before and after the race. The dealers manually enabled or disabled the limiter.
Amazing what people will do. Nice detective work.
It is sad that pets are only adopted by some because of their 'profile'. Apparently black cats aren't as adoptable because they're not as photogenic. One of our rescue cats has only one eye but doesn't miss a trick. Hence I call him a pirate.
If it hadn't been for those pesky kids.....
"These predictions would be used to optimize and tweak future critters' profiles so that they are adopted as soon as possible."
We need less of this tech and more honesty in the world, and facing facts such as not all pets can be successfully re-homed.
Kaggle is a bit of fun, but it should be obvious to anyone with any experience of online competitions that anything where something can be won will attract cheats.
Companies see Kaggle at its ilk as a cheap way of getting solutions to problems and that is their problem. And any company that values ratings in such competitions is also asking to be screwed.
... All it takes is one person to be curious to find the cheat.
Reminds me of something I read a decade ago where someone in motorsport won a big truck in a competition for novices. Except he wasn't a novice in his country, just a novice in the country where the competition was held. It all came to light after a member of the media saw the photo and said "that's not so-and-so". The whole thing blew up massively and the truck had to be returned. It is just amazing what people do to win.
I was going to have a right old whinge about the usual misuse of country TLD's:
"PetFinder.my - Oh so the 'My petfinder' company is Malaysian, then? *said sarcastically, with a smug obnoxious look*
Only to later read "Malaysia-based PetFinder.my,"
Bugger. A chance to be an angry old moaning git, and you took it away..... I must moan about that...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
