Amazon, Apple, Google, IBM, Microsoft speech-to-text AI systems can't understand black people as well as whites

Speech-recognition software developed by top tech firms struggle to understand black people compared to white people, according to research published this week. The study, led by academics at Stanford University and Georgetown University in America, probed five “state-of-the-art” cloud-hosted automated speech recognition (ASR …

[Elii]

Racist, really?

Not sure how a computer can be “racist” exactly. Maybe there were more white voices sampled than black voices, so it’s inherently more accurate with white voices. That sampling may just reflect the demographic of the population. It could be that someone writing the software specifically wrote or such that it wasn’t able to recognise black voices as well, but I think that is pretty unlikely.

Unfortunately with the current ridiculously over-PC atmosphere, if someone has the slightest opportunity call something or someone racist, they’ll take it.

[cornetman]

Re: Racist, really?

Indeed. Many intersectional feminists say these days (although obviously they are full of it) that racism is all about power. I'm not sure what kind of "power" a speech-to-text system can be said to have. It's just a program.

I can't understand a lot of Scots. Does that make me racist?

[diodesign]

"I can't understand a lot of Scots. Does that make me racist?"

I can't understand most northern UK accents, I must admit. That doesn't make me a racist.

However, if I was training a speech-recognition AI and left out a load of accents because I don't care for nor understand them, didn't think about them, or thought it would just work without them, then you might well be able to call me biased.

C.

[big_D]

Re: "I can't understand a lot of Scots. Does that make me racist?"

It is the same with German. A lot of regional dialects are very difficult to understand. Platt has some English pronunciations, but also a lot of Dutch, so I understand more of that than many other dialects.

Platt, Ostfriesisch, Schwäbisch, Bayerisch, Hessisch, Ruhrpott, Österreichisch, Schweizerdeutsch can be very difficult to follow, for a non-native speaker and also for many native German speakers. You will often get subtitles when people with regional accents are interviewed on TV, for example. Some of it is strong accents, but a lot of it is regional words and grammar that only exist there and have no bearing on "German". I suspect the same voice recognition engines would have similar problems in Germany, without even taking into account people of different race.

Heck, Alexa was totally hopeless for me. About a 20% accuracy rate. It was usually quicker to "type" a search into the FireTV with that dreadful remote than it was to speak to Alexa. My British accent probably didn't help me, but my wife and her daughters are native German, but come from a Platt region and it didn't have much success with them either.

[Cederic]

Re: "I can't understand a lot of Scots. Does that make me racist?"

No, I'd call your work incomplete.

Did you include people from rural locations in Tennessee too, because they have a very distinct accent and speech pattern. I won't even mention their skin colour because - get this - it just isn't relevant.

Which is why it's a valid comparison.

[big_D]

Re: "I can't understand a lot of Scots. Does that make me racist?"

Or get a couple of people out of the Louisiana Bayous. They can be very difficult to understand.

[Anonymous Coward]

Re: "I can't understand a lot of Scots. Does that make me racist?"

Watching some Canadian TV... I noticed one particular area, was not hard to understand, just had a habit of speaking very very quietly. :P

[jospanner]

Re: "I can't understand a lot of Scots. Does that make me racist?"

It's not just 'incomplete', though, is it?

Certain groups get favourable coverage in these things for a reason, and others get ignored.

It's because you didn't think to include them. And you didn't think to include them *for a reason*. It's because you didn't consider it to be that important, if you even considered it at all.

[Anonymous Coward]

Re: Racist, really?

I can't understand a lot of Scots. Does that make me racist?

No, but it does mean you have a lot in common with voice-activated lifts or elevators...

https://www.youtube.com/watch?v=tbgKDR9aIRA

[onemark03]

I can't understand a lot of Scots.

Pedantry on:

Sorry but your example is a bit off.

The dialogue in the Scottish lift skit is Scottish English, not Scots, which - depending on your point of view or which textbooks you have read - is either a dialect or a variant of English or even a separate language. (I will not go into that debate here.)

Pedantry off.

[jospanner]

Re: Racist, really?

Yes, racism implies a power relationship.

The words "<Ethnicity> sucks" have a very different function depending on the power of the person saying it, and the power of the ethnicity it is being said about.

You might not *like* being challenged, but it doesn't equate to someone having legit worries about their security or safety.

[Anonymous Coward]

Re: Racist, really?

Cool, so if we drop David Duke in the middle of Nigeria, he's suddenly not a racist?

This is breathtakingly obvious nonsense, and I mean that in the sense that it's ridiculousness sucks the oxygen out of actual important racial issues.

[katrinab]

Re: Racist, really?

If you only recruit white people for your sample data, then that is probably racist.

[a_yank_lurker]

Re: Racist, really?

I would not say racist as much idiotic. There are numerous North American accents. Some are difficult to understand the first time you hear them if you are not a local. The Left Coast accents are pretty bland as US accents go but some regions have a very strong accent even for whites. Now add in all the other English accents and I would like to see the error rates for say New Zealand, Wales, or Scotland.

[Mage]

Re: Racist, really?

Not just "black" people, but the entire Tech Industry tends to be White, Male and US Centric in the testing, default options, possible configurations etc.

The USA is less than 13% of the world population. A lot of their citizens speak Spanish. Why does the stupid one key less USA layout exist, resulting in badly designed non-US keyboards on laptops, BT and USB mini UK Keyboards that are really the USA one overprinted, poor non-USA support till recently on Android external keyboards, only Latin-Roman fonts on early Kindles, though that was a problem solved 10 years earlier on the underlying OS.

Absolutely everything having US Region, US Dictionary and Letter Size paper by default.

So the fact that all these USA developers ignore the rest of the world and only train their speech matching (it's NOT recognition) on a stupidly narrow subset of speakers is no surprise at all. After all it would cost much more to use proper training data.

[jospanner]

Re: Racist, really?

Do you also think that segregated drinking fountains can't be racist? You know, cos they're just drinking fountains?

[Imhotep]

Re: Racist, really?

It's hard to believe you are being serious. Yes, I do think drinking fountains can't be racist. I do believe that people who mandate white only drinking fountains are racist.

[The Oncoming Scorn]

Re: Racist, really?

I remember South Africa opening its first mixed race swimming pools.

The whites get the top half.....

With apologies for ripping off Not The Nine O'clock News archives for that one, who also brought us Constable Savage.

[Another User]

Re: Racist, really?

Looking at your registration date I tend to think that you are fairly young. This means you do not have first hand experience of segregation. You saw this in the Apple movie ‘The Banker’.

Cool deduction with limited data available?

Here is something regarding facial recognition https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/

[Imhotep]

Re: Racist, really?

In my experience "black" speech is mainly a function of environment - especially country - and not race. In college my macroeconomics professor was from Nigeria, and I had a hard time understanding him at first. His enunciation was not the problem, but the rhythm of his speech. Even later in the course, it took a couple of minutes for me to get in "sync" with him to understand what he was saying.

I have noticed in my lifetime that the regional differences in speech are going away somewhat - perhaps because the media is bringing us to a more universal accent.

[Anonymous Coward]

Re: Racist, really?

I’ve met some blank people from Africa that actually speak better than a lot of white people I know!

[Imhotep]

Re: Racist, really?

Well sure, if you start with a blank slate....

[General Purpose]

Re: Racist, really?

The word "racist" isn't in the article. The journalist didn't use it, the academics didn't use it. The only one taking "the slightest opportunity call something or someone racist", as you put it, was you.

[Dr.Flay]

The study is flawed as it assumes race creates an accent and speech patterns.

The researchers should have directed their attention to the many hilarious videos in youtube, of mostly white people in the UK that do not have a BBC English accent, where they are pleading, shouting, screaming and swearing at Alexa, Siri or Google.

[diodesign]

"it assumes race creates an accent and speech patterns"

Sure, there are plenty of people, millions, many, many millions, who are non-white, but due to their upbringing, speak and sound exactly like the surrounding white population.

This study isn't about people like them. It's about AI not having enough training data to recognize a *range* of accents. And it happens to be that the accents lacking coverage are those from black communities. There's the bias.

C.

[Anonymous Coward]

Re: "it assumes race creates an accent and speech patterns"

I think the point is that there are an awful lot of black Americans in the US, whereas the counter examples people are coming up with are small, isolated populations. Louisiana, for instance, has under 5000 people in 3/4 of its towns. Getting training data on potentially hundreds of different accents with tiny numbers of speakers would be hard.

I will just note one example of the kind of problem that can arise. A colleague was visiting Kentucky in the 1990s on a QA assessment for a supplier. I get a phone call.

-"They're talking about 'layered wire'. Does that make sense? I can't find the spec sheet."

I dig out the spec sheet.

-"It's lead."

-"Led what?"

-"Lead as in atomic number 82."

Sudden illumination.

-"Gotcha. Now I think of it, yesterday someone said I needed to 'hayung a rayut'."

[Mage]

Re: "it assumes race creates an accent and speech patterns"

Like the "black" Kenyan girl bullied in a USA school by "blacks" for "speaking white".

A lot of what is called racism is plain bigotry and ignorance. Really so-called "racism" is a subset of prejudice, bigotry, xenophobia, ignorance and simply people that want to bully or control anyone not like them. So they belittle them. Race doesn't actually exist.

[Anonymous Coward]

Re: "it assumes race creates an accent and speech patterns"

"Race doesn't actually exist."

I agree with everything you wrote but race does actually exist, and I hope it continues so. As a heterosexual, it's very nice to have many races.

[Mage]

Re: "it assumes race creates an accent and speech patterns"

There is negligible difference genetically between a tall fair, blue eyed Scandinavian and an African bush person. There are more genetic differences within Africa.

The tint of the skin or your height, eye or hair color or kind of hair is irrelevant to Speech Recognition. The very idea of Race was only invented to justify exploitation of various ethnic groups by more powerful groups.

[jospanner]

Re: "it assumes race creates an accent and speech patterns"

Just sounds to me like people are more upset about being called racist than actual racism.

[Drew Scriver]

Re: "it assumes race creates an accent and speech patterns"

How about we refer to homo sapiens as the human race and stop calling each other black and white?

Look at the gentleman in the picture above the article. The device in front of him is black. The color does not match his skin.

On a related note, I resent being labeled "white". When I put my hand on a piece of white printer paper it becomes quite obvious that my skin is not white. According to a paint scanner at the home improvement store it's a hue of red, actually. Yes, I did put my hand in it.

This division of people into different groups called "races" is scientifically wrong-headed, especially if you look at the genetic make-up of different groups. It can also be exceedingly dangerous, as the theory of evolution dictates that different groups would have evolved differently and possibly at different rates. And yes, I'm sidestepping the debate about species.

"I have a dream that my four little children will one day live in a nation where they will not be judged by the color of their skin but by the content of their character."

Dr. Martin Luther King Jr., August 28, 1963

[Anonymous Coward]

Eleven?!

https://youtu.be/sAz_UvnUeuU

Scotts Voice Recognition Lift.

[TheProf]

'I say, my good man, could you tell me where the bathroom is?'

On holiday in America a few years back I discovered that I got a better response from people if I used a phoney American accent when asking them a question

I'm not sure who was being racist.

Also: I have a voice controlled model Dalek and that only seems to understand me if I do my Captain Picard impersonation.

[Sandtitz]

Re: 'I say, my good man, could you tell me where the bathroom is?'

"Also: I have a voice controlled model Dalek and that only seems to understand me if I do my Captain Picard impersonation."

Shaved head, Picard Maneuver and everything? You could try Jar Jar Binks as well.

[TheProf]

Re: 'I say, my good man, could you tell me where the bathroom is?'

"Shaved head, Picard Maneuver and everything?"

I have the Picard haircut courtesy of nature. There's nothing natural about JJ Binks.

[The Oncoming Scorn]

Re: 'I say, my good man, could you tell me where the bathroom is?'

The still not quite ex Mrs Oncoming Scorn, had loads of fun* did trying to get On-Star voice recognition system to recognize her accent** in her car.

*Well the rest of the car's occupants did, especially when it finally recognized what she said & asked for confirmation & the youngest shouted out No!

**A mix of Essex & Devonshire.

[Sawari Ma]

They forgot to load the Jive language pack.

[Anonymous Coward]

Obligatory Airplane! clip

https://www.youtube.com/watch?v=RrZlWw8Di10

[The Oncoming Scorn]

Re: Obligatory Airplane! clip

I saw a rastafarian, crying on a street corner, bemoaning the fact his woman had left him, he looked depressed so I suggested he see a Doctor.

A few days later he was in the same corner, dressed in a smart suit, full of confidence.

I asked him why the change, what did the doctor say & had his lady returned?

"No mahn.. I went to de doctors, he say I waz impotent. Hiz words made me tink, so I changed me clothing & life, because if I am impotent, then I gotta dress & look impotent."

Accent from the Jim Davidsons book of politically incorrect material.

[The Nazz]

Racist , really (part ii)

And what were the scores for other groups, dare i say the Browns, the Yellows, The Reds etc? NOT the Greens, they oft speak with plummy accents.

Asians, Latinos, Hispanics, real Africans etc.

Omitted completely.

[cornetman]

> Black speakers are more likely to use African American Vernacular English (AAVE), a style of English that has its own grammatical rules and vocabulary.

So, erm not English then.

I'm sure if I used a lot of b*llshit words instead of well established and understood ones, it would have a hard time understanding me.

[FIA]

So, erm not English then.

There is no one definition of English.

I'm sure if I used a lot of b*llshit words instead of well established and understood ones, it would have a hard time understanding me.

What's a 'bullshit word'? You seem to expect everyone to talk like you do, the world doesn't work like that. Regional/cultural variations exist; they need to be dealt with.

[Anonymous Coward]

It is a matter of infinite regret to this interlocutor that thou thinkest that the manner in which thou expressest thine intellectual processes verbally is the only nice and accurate formulation of the language utilised by the people of England, and which is further required to be used semper, ubique et ad omnibus.

It also peeves me that you think the only form of English allowed is the one you happen to have learnt.

(Most Portuguese speakers neither live in Portugal nor speak with a Lisbon accent. And most English speakers do not use the Home Counties RP I grew up with either. But speech-to-text systems can generally understand me without any problems, other than the one the other day that repeatedly misheard my carefully articulated "ess" as "z". That is unusual; more often due to that bit of American weirdness they tend to hear "see" (third letter of alphabet) as z which they pronounce "zee". "Zed" is much simpler for an automated recognition system.)

[joeW]

Yes, proper words only. Such as "erm".

[jospanner]

It is English, you just didn't bother to consider it cos your idea of English comes exclusively from white, middle class English southerners, because you are actually quite ignorant.

[Anonymous Coward]

Third form of what I wrote, above. Except that I doubt he's a white middle class English southerner because (insert joke icon) we write better than that. And we know that the Register doesn't have a nanny censor for words like "bullshit". (Chaucer refers quite happily to a "shitten shepherd").

[joeW]

Based on my experience

I'm guessing that it would still do better with Black American english than with my Irish english.

[Nige]

Re: Based on my experience

Same here with a South Wales Valleys Welsh accent.

I've given up on all the voice recognition phone lines when I try to call my bank or credit card providers. Its easier to just repeatedly pressing '0' or '*' or find the menu option to actually speak to a person.

[martinusher]

Sigh.....

Its not race, its patios, accents and figures of speech that these systems have yet to be trained on. I'm not black but I quite often have trouble with my friend in the corner, she's OK with American and RP English but if I slip into my native dialect (Cockney) then she gets really muddled. It could be that the folks at Amazon have designed the system deliberately to discriminate against me but I suspect its more to do with my failure to enunciate my words clearly.

Learning to interact with these devices is like learning the rules of the road so you can drive. Its a nuisance and may not be the rules youi originally learned when you grew up elsewhere but its how we all get along and -- hopefully -- avoid colliding with others.

[TheMeerkat]

Re: Sigh.....

It probably the same as with Black faces - lack of contrast makes the recognition more difficult.

Some accents tend to have better contrast between words and sounds. I guess it makes “posh” accents easier to understand - it came by in a part of society that moved around more than the rest and, consequently, learn to speak in a way that more people understand them.

[Edward Noel]

Re: Sigh.....

I don't think it's fair to blame patios. Patois on the other hand...

[jospanner]

Re: Sigh.....

Speech tech is designed by people who speak like that. Why should we accept that there is only one "valid" accent because the designers were lazy and were used to everyone else bending over to suit them?

[John Brown (no body)]

Re: Sigh.....

It's the Apple and Microsoft way. Their way or the highway!

[John Brown (no body)]

Re: Sigh.....

"Its not race, its patios,"

Patios? Aren't they the things you bury the bodies under?

(don't you hate it when your fingers try to anticipate what your brain wants!)

[Randy Hudson]

Siri: Cut me some slack, Jack! Chump don' want no help, chump don't GET da help!

[TheMeerkat]

Accent is not an attribute of race. It comes with cultural background.

[Giovani Tapini]

Now, what about

Boomhaur from King of the Hill..

[Draco]

"Black speakers are more likely to use African American Vernacular English (AAVE)"

Is this true in Nigeria, Kenya, South Africa, Barbados, etc or is it only more likely in the USA?

One can argue that the data set is insufficient broad or biased. One might also argue that a minor accent or dialect has less representation in a sample. There might also be a discrepancy in who is willing to give a voice sample. I was recently validating audio clips from Common Voice - Mozilla's initiative to sample as much human voice as possible and of the 120 audio clips I evaluated: 20 were female, 97 were male, and 3 had no voice. Given this is an open-source project and open to anyone who can navigate to it, why aren't more women contributing? I did not track accents and dialects - which are, mostly, beyond me.

I am also willing to bet these speech-to-text systems fare poorly on English spoken with an accent - whether indigenous or "foreign" like Russian, or Chinese, or Filipino, or Spanish, etc.

[Mage]

Re: "Black speakers are more likely to use African American Vernacular English (AAVE)"

Most of the darker skinned people in the World are not USA Citizens.

Most actual Africans, or non-Americans of African origin do not use African American Vernacular English (AAVE).

Lots of Black people in the USA don't seem to use AAVE either. At least not the ones I met. Perhaps it's particular urban socio-economic groups in the USA that use AAVE.

Personally I think Speech recognition has hardly improved since the 1990s and over 15 years ago when it was on devices locally. I suspect a major reason for the Speech Recognition using the remote server now is the invasive capture of your behaviour. Maybe it's really much better for a certain narrow class of White Americans.

[Mark192]

Conflating accent and race

If they didn't control for accent/dialect, then they're conflating accent with race... a peculiar thing to do in a study... it's most often done as a lazy generalisation by people that haven't thought about the subject much.

[ragnar]

Is this actually a fair conclusion to draw?

"CORAAL contains black speakers who use AAVE to various degrees, and hail from Princeville, a city known for its historic African American population, as well as Rochester, New York, and Washington DC.

VOC is made up of white speakers from Sacramento and Humboldt County, California."

Wouldn't it have been a better comparison to have a dataset made up of white and black speakers who are all either speaking English or all speaking African American Vernacular English?

Since wikipedia says AAVE has its own unique grammar, vocabulary and accents, it's unsurprising that an AI dealing with standard English struggles. The AI may be equally good at recognising black speakers who are using conventional English grammar, vocabulary and accents but this study doesn't seem to allow us to draw a conclusion on that front.

[FIA]

It's otherwise great for white people, or people who sound like them.

What do 'white people' sound like? Do you mean 'Americans from a paticular geographical area?'

I'm white, but with a reasonably strong regional (non American) accent and there's quite a lot of things Siri simply can't understand me saying. Ya gets me?

[Anonymous Coward]

<tongue in cheek> I bet those voice recognition systems are ageist, too, and only recognise the vernacular of a particular age range, too!. </tongue in cheek>

Definitely not a matter of race IMO, but of culture. There's a heck of a lot of English variants and dialects that I can't understand either, but that doesn't make me racist or not interested in accessibility and inclusivity. It just means my limited processing power isn't up to the job of understanding much more than the dataset it was trained on when I was a nipper.

[jospanner]

If you didn't think to train your system on the set of data that actually represents your customer base because you just assumed everyone was like you?

Yeah that's gonna be racism, classism, ageism etc.

[John Brown (no body)]

Back when I was very young and just starting out at school, we were taught to speak "properly", ie a reasonably standardised English that every English speaker would be likely to understand. But we were also allowed to speak our local accents and use slang words too. We learned to adjust how we speak based on the situation.

I'm not sure where UK schools currently stand, but there was a phase some years ago where English was seen as "not a problem, so long as the meaning is conveyed", which resulted in at least a generation coming out of school with very poor communication skills other than their local "street" English, let alone barely able to spell.

[Nige]

Learn To Spoke Proper England

The brain has an amazing "universal translator" capability. Whether its due to the amount of training it has - lets face it, we have a lifetime's experience of listening to speech and speaking to others - or if its just an in-built ability when we have to interact with others who speak slower, faster, or with a different tone, inflection, or accent. And they're trying to train these voice recognition systems on a few 1000's of hours worth of recorded dialogue? No chance!

I always cringe when I'm listening to the Radio and you hear some 'hip & trendy' reporter saying "wiv" and "vis" and "vat" and "sumpink", or people who can't seem to pronounce a 'T' and replace it with a glottal stop. FFS!

When you're speaking to a wider audience, or maybe someone who isn't a native speaker of whatever your language is, it makes perfect sense to me to try to speak 'properly' and clearly pronounce and enunciate rather than the usual 'vernacular', shortcuts, slang, and lazy pronunciation that all too easily creeps in to common speech.

[jospanner]

Re: Learn To Spoke Proper England

You just seem to be very upset that working class people exist and want to speak in the way that's most natural to us.

[John Brown (no body)]

Re: Learn To Spoke Proper England

He specified a "reporter". By definition, they are speaking to a wider audience, not their mates down the pub. They can talk "street", or "working class" if you prefer with their mates and no one will care. But when doing their professional job, which is primarily communication, then they should not be speaking with a local street accent and language.

After all, it would be plain rude if I wrote this in my native Geordie accent since many here wouldn't be able to fully understand it.

Gan canny man, Ahm awee hyem noo.

[Abdul-Alhazred]

Implicitly, the Turing test is for the AI to pass for a white human, and this is exactly the sort of error a white human would make.

Turing test passed again.

[Anonymous Coward]

Understands most Indians well.

I’ve been using the captions on hangout meetings to clarify what many of my off shore colleagues are saying to me. Works a treat on many but not all voices. It’s also not perfect and does go back and correct itself.

[BGatez]

been done

https://www.imdb.com/title/tt5688932/

[jospanner]

Genuinely great watching the white middle class southerners show how they just expect everyone else in the world to bow down to them. The sheer arrogance of people for who society is engineered around throwing a hissy fit when asked to consider that other people exist.

You designed the system based on your own biases. The problem is that it isn't being used by *you*, it is being used by a wide set of English speaking people. Stop being so self absorbed and understand that different regions and cultural groups have different accents and vocab.

If that upsets you, tough. Grow up. Other customer facing jobs have to deal with the difficulties of human interaction. You're not special.

[Yorick]

Plight of the legal alien

German living in the US - Siri can't understand me worth shit. "Route me to the Volvo dealer in West Springfield" ... "Calculating route to Audi dealer, is that correct?"

I've given up. I only type now, my speaking very slowly to or worse, yelling at my phone is not a good look.