Turns out humans are leading AI systems astray because we can't agree on labeling Top datasets used to train AI models and benchmark how the technology has progressed over time are riddled with labeling errors, a study shows. Data is a vital resource in teaching machines how to complete specific tasks, whether that's identifying different species of plants or automatically generating captions. Most neural …
Yes. Mechanical Turk pays the participants so little that they have a lot of data pollution problems. Also, they have the problem that the people doing the work are either bored people who will give up in fifteen minutes or people who really don't have better ways to get money, so you can't expect consistency or strict attention to detail.
It's along the lines of all those studies they do at universities where students are paid to participate in research with an amount of money which could be used in a vending machine in 1995. Especially the economics studies which effectively boil down to "Would you take this action if we cut your meaningless money to even more meaningless money?". I participated in a few research programs while studying, but always because I was bored and didn't mind wasting a few minutes. I did actual work to earn money.
Also, they have the problem that the people doing the work are either bored people who will give up in fifteen minutes or people who really don't have better ways to get money, so you can't expect consistency or strict attention to detail.
Long before the computer age I had a job with a mail order company.Our work was filng little slips of paper - coupons or something- with people's name and address they'd sent in. These had to be packed really tightly into stiff plastic wallets in alphabetical order. It was tedious and painful, the plastic would cut into your fingers and forcing the paper apart to insert the new slip was really difficult. None of us had been there long. And it was obvious why. The wallets were a mess. Sections would be in order, then they'd be random. As a new bunch of underlings were recruited, got bored and were fired. As we were.
The crap pay and the mean fisted decision to force as many slips as possible into the envelopes as tightly as possible meant that the filing system was close to useless..
The spirit of those days (early 80s?- the Woolworth's fire happened across the road while I was there) apparently lives on. And since there is a phrase "Spoiling the ship for a happ'orth of tar" it seems to go back a long way before that too.
If that's the case, they're not very good at their job. Most of the economics studies involving meaningless money contradict many other theories. Behavioral economics really likes these. Whether that's because the previous theories were wrong (probably), because people act different when they actually care about incentives (probably) or because the researcher is deliberately messing with the results (probably not), they don't tend to be blatantly confirmatory.
"You get what you pay for. GIGO!"
Yup.
There's currently a project commissioning human translators to translate a fairly large amount of text, for use as input for a Machine Translation system. Not a bad idea, but there are some issues.
Offering 1/2 to 1/3 of the going rate probably doesn't help attracting competent translators. Asking the translators to provide two alternative translations isn't a bad idea. However, that fails when you have to translate the Dutch 'Het gras is groen' into English, as 'The grass is green' is essentially the only sensible translation. An alternative translation would simply add misleading input. Finally, all the sentences to be translated for this project are completely unconnected from each other. That's a major issue, as anyone who's done more than a day or two of translating will tell you that 'context is everything'.
Different Road Traffic Acts in England than in Scotland. As far as I know, "Special Roads" in Scotland (namely, motorways and some others under similar special legislation (with restrictions on what sort of vehicles can use them) such as the Edinburgh city bypass) have to show the "70" sign instead of the "national speed limit applies" sign. Exactly why this bit of signage hyper-precision exists, I don't know, but the folks at SABRE and the like probably do…
Current AI is just pattern recognition on steroids, there is still no real intelligence in any AI system. The single word tagging of pictures has been known to be suboptimal for a long time now and yet it is still done. If there was any of the I in AI then multiple word tags should not be an issue as it should be capable of recognition of multiple items in one picture.
Additionally, and I can't find the fine article that was in this exalted place, you can carry a tag around with you and be misidentified as the tag because the AI 'learns' the tag as well as the picture.
Certainly artificial, definitely not yet intelligent.
Well, if only we could agree on a (non-circular) definition of "intelligence", that would be a step in the right direction. Until then, I don't see how we can hope to get anywhere.
Notice how no-one talks about the Turing test any more? That's because it was passed, and so everyone promptly decided "oh no, that's not intelligence after all". As long as we're allowed to keep moving the goalposts like that, they're not going to make it.
"Notice how no-one talks about the Turing test any more? That's because it was passed, ..."
It was? When? How? Who? I've never seen any computer system get anywhere close. Come to that, I've seen *people* fail it (usually working in institutions as customer-facing staff and following rule sets, to be fair). Or did they re-define Turing test, so that I'm not allowed to ask questions that any 5-year-old could answer but that fall outside the computer's domain of expertise? (For example, does this picture show baseballs or a bucket? Obviously a 5-year-old could answer that, but apparently an AI researcher can't.)
It's unclear. People like to hold Turing test challenges, and programs in those challenges have been ruled human before. That might not be a great basis to declare the test passed, but it is what the test specifies. It also depends a lot on what we want them to do. The original Turing test didn't include sending images to the other party, therefore not requiring the AIs to see. Also, a program trying to pass the Turing test, because it sends back text, has the ability to say "both" where the programs here which are just identifying things have to pick a single one.
In my opinion, the Turing test is a rough test that is likely to have too much uncertainty to prove intelligence. There were probably a lot of people who liked to talk about it back when it seemed impossible, but now they're pointing out defficiencies in the concept. The only problem is that a lot of people attack anything termed AI without even trying to define what they think AI is, or provide an unrealistic explanation which means something is only AI if it acts entirely human and attained sapience itself without ever being programmed.
To state the obvious, the labels need to consist in more than their complex relationships with other mere labels. The actual things need to be represented by sensory data, and the AI system needs to be located, discretely, in 3D space --- that is, if we want a system to actually *know* what things are.
... lychee labeled as Hedgehog Eggs. Down the street and around the corner, the local Lola Market has occasionally had their Spiked Choyote (Sechium edule) relabeled as Porcupine Eggs. Given their proximity, I suspect the joker is the same person at both stores. No, it's not me.
I learn something new every day from this site - much of which has nothing to do with technology.
===
Warhol would have run rings around AI with his placement of everyday objects as art. Bananas reminds me of Nico's distinctive vocals...
https://www.youtube.com/watch?v=AkDJcUCyjCU
Mark192: bananas mislabelled as "small, off-duty Czechoslovakian traffic wardens".
That is ridiculous, don't they know that the Czech republic and Slovakia split apart in 1993?
https://kafkadesk.org/2018/10/30/why-did-czechoslovakia-break-up/
"On January 1, 1993, Czechoslovakia split into two independent states, the Czech Republic and Slovakia, in what is now known as the “Velvet divorce” (in a reference to the Velvet revolution) due to its peaceful and negotiated nature."
"And while Slovak nationalism sentiment strived for more autonomy, Czech nationalism embraced Czechoslovakism, mainly due to their privileged position within the federation."
-- https://kafkadesk.org/2018/10/30/why-did-czechoslovakia-break-up/
As the article itself alludes to, I'm sure we can all think of certain other multi-ethnic countries where similar feelings apply…
Bias is endemic to society, right down to the very roots of the evolution of the languages we use. Cultural biases of thousands of years are embedded in every culture on the planet, and it results in biases in the data sets, even if they are "correctly" labelled. Prejudices in society that lead to fewer members of a minority being successful in particular fields leads the machines to conclude that people of such background are *unlikely* to be qualified, rather than that they are under-represented.
It is a thorny issue, and one that isn't easily solved, because it begins to tread on concepts of morality and justice in society, and the one thing we can all agree on is that no one agrees on those subjects.
With automatic translation software, it either just looks at the words and tries to do them literally or it finds an idiom in a big dictionary. When I did some translation, I would always replace them with factual statements. It had less flavor, but at least I knew the reader would understand it without taking the risk that the closest idiom I could come up with was regional. Then again, I was not a professional translator, just a person who spoke multiple languages and hadfriends who didn't.
Garbage In Garbage Out. A saying as old as computing. When I was spending a large amount of time merging data from other sources into the databases of a company I was working for I made sure my juniors and the SMEs spent a bit of effort vetting those data. All our data had to be defensible in court. It is the cavalier attitude to data and assumption all data are perfect which caused to avoid those areas. There are no standards of quality though they wouldn't be hard to to develop.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
