AI models still racist, even with more balanced training

AI algorithms can still come loaded with racial bias, even if they're trained on data more representative of different ethnic groups, according to new research. An international team of researchers analyzed how accurate algorithms were at predicting various cognitive behaviors and health measurements from brain fMRI scans, …

[me212]

Or maybe, just maybe the AI is telling the truth. Maybe the world isnt a nicely ordered place where everything is equal. Seriously everytime AI doesn't give the results they want it's muh racisim. Then we arguments it conflicts with results conducted not using AI, but we live in a world where anybody who even slightly suggests research shows groups with certain attributes have negative traits will be hounded out of their profession.

Now I'm not saying some 'ism is never behind results you get, but in all of these stories they seem to be looking for a predetermined set of results that show equality and reject any results that differ.

[Anonymous Coward]

Or maybe fMRI is not a good way to go about predicting cognitive behavior.

The answer could also be found in questionable assumptions that are taken as axioms by the people running the experiment, and god forbid that one has to end up writing "The results of the research funded by this X million $ grant indicate that our idea does not work".

There are a lot of hidden variables that are all too human and permeate throughout the process, from the design of the experiment, through the data and their processing to the final interpreation based, as you mentioned, on confirming what one is already thinking (and written in the grant application) rather than on challenging assumptions.

In a word, a snafu. And very human at that.

[Filippo]

The point isn't that the AI is telling things we don't like. The point is that it's apparently telling more answers that don't match the answers in the test set, when run on non-whites. This is problematic regardless of what those answers are.

If you got an x-ray and the doctor told you that he's not sure whether your leg is broken or not, because you're not white, you would be rightly mad at him - regardless of whether you're white or not, regardless of whether your leg is actually broken or not, regardless of whether he eventually makes a guess, and regardless of whether the guess is right.

The fact that he finds a correlation at all between ethnicity and his ability to diagnose a fracture from an x-ray is sufficient to question his competency. While in theory ethnic differences could impact the ability to make a diagnosis, in practice that would be an outlandish claim in this case. Maybe, in some cases, in dermatology, but x-ray are another thing entirely (and so are brain scans). It's far more likely that the doctor is incompetent - which calls into question his ability to make accurate diagnosis on whites too, because who knows what the hell he's basing them on? It's the same thing here.

[Anonymous Coward]

>>they seem to be looking for a predetermined set of results

You're badly showing your ignorance here - both in understanding ML, and in the subconcious prejudice that stops you from questioning your biased thinking.

They train on a set of data, and then use the produced model to produce an answer for a sample from the out-of-training data set. They check the error between the AI predicted result and the expected or measured and known to be correct result. They already have the right answer to compare against.

For eg if they have 1000 samples, they train with 750 and check how well it works for the 250.

What the paper says is that the training methods still show an increased error, that is racially correlated, which they do not understand. Perhaps the data scientist, who has to tune the model, was the source of the error, and was constraining the range of the weight format in such a way that the model is forced to produce results in incorrect output that is racially correlated.

For eg forcing only 2bits, where to be unbiased and to represent all correct outcomes, 3 bits are required. In other words, a 3 bit weight is required to be accurate and unbiased, whereas a 2bit weight is an incomplete and thus non-representative model. What ever a non-representative model works on, can present as racial, gender, age bias.

Spend some time understanding ML and the article before ranting.

[LionelB]

> Or maybe, just maybe the AI is telling the truth.

What "truth"?

> Seriously everytime AI doesn't give the results they want it's muh racisim.

Well, the "results they want" is good predictive accuracy. It seems to me that no-one is suggesting racism here. They are simply looking for an explanation for a puzzling discrepancy in predictive performance; puzzling, since the way the study was conducted there's no a priori reason to expect any difference between racial groups in the ability to predict the factors they tested.

[LionelB]

> It seems to me that no-one is suggesting racism here.

Except, perhaps, the author of the clickbaity and thoroughly disingenuous headline.

[JassMan]

As I commented in an earlier story on AI

It is time to stop giving ML the title of Artificial Intelligence unless you can prove it is open-minded and unbiased. Otherwise it should be called Artificial Bigotry

I'm pretty sure that the human equivalents in a past US president and the current Russian leader consider themselves Intelligent but the rest of us have another name for it.

[I am David Jones]

Re: As I commented in an earlier story on AI

Given that nobody is totally open-minded and unbiased, you’re setting a high bar there. I’d settle for “as good as or better than an average, trained human”.

[Yet Another Anonymous coward]

Re: As I commented in an earlier story on AI

It is open-minded and unbiased, but the data isn't.

If you feed it a data set of mugshots where 50% of the convicted are young black men and let it loose on a population where 5% are young black men - your CCTV is going to over detect young black men.

Similarly if you give it routine MRIs of healthy 50 year old white men with good insurance and 0.1% young black men whose condition was weird enough that somebody ordered an MRI it's going to assume that everyone in a certain population is likely to have medical weird conditions.

[LybsterRoy]

Re: As I commented in an earlier story on AI

How about a current POTUS

[LionelB]

Re: As I commented in an earlier story on AI

> It is time to stop giving ML the title of Artificial Intelligence unless you can prove it is open-minded and unbiased.

You mean open-minded and unbiased like human intelligence? Oh, wait...

[I am David Jones]

Isn’t this just down to the fact that judging facial features will always be easier when the skin tone is lighter?

And by “fact” I mean “my totally unqualified assumption” :)

[Anonymous Coward]

So what you're suggesting is you have a better chance of getting away with a crime in today's world of surveillance cameras and facial recognition systems if you get a tan first?

[Yet Another Anonymous coward]

I was doing some work on a skin cancer / weird mole detection imaging app.

Does this have a problem with skin colour, was a question that came up in the clinical trial design. Until somebody pointed out that, in general, indigenous Australians / South-Africans had rather less issues with sun linked melanoma than red haired immigrants

[Anonymous Coward]

Hey, hey you're a Monkee ....

Or is that just my AI that's on the blink ?

[gnasher729]

White faces being more “camera friendly” doesn’t exactly help, but doesn’t seem to be the biggest cause of bias.

[Anonymous Coward]

I expect the issue boils down to the quality of the data that was collected. I suspect you'll find more "less than caring" doctors treating AA's than WA's, and therefore not taking the time to do as thorough and proper a job, resulting in more noise in the dataset.

It is a very common issue here in Canada for the First Nations when it comes to dealing with the entire medical community right across the country. They often get downright callous service in some districts and hospitals, unfortunately. And I'm talking today, not some long-distant past best forgotten.

[Anonymous Coward]

Racism is still there ... everywhere .... just more subtle than it used to be !!!

msobkow,

Your comments regarding Canada are in general equally applicable in other countries.

I have never wanted to assume that the care given is variable because of existing racial biases BUT it is becoming more difficult to believe that the general racial biases in society are somehow NOT impacting the services we all use in the UK.

I have experienced interactions where it is impossible to exclude the possibility that the people you are interfacing with are not as even handed as you would like or expect !!!

Racial bias is not so in your face as it was in the 1970's but it is still there and can be quite subtle at times !!!

[martinusher]

Re: Racism is still there ... everywhere .... just more subtle than it used to be !!!

Bias may not be the result or prejudice but simply the problem of communicating across different societies. There also may be different expectations based on those societies. The fundamental problem with this type of report is an expectation that someone is doing a wrong to some group that needs remediation with the effort needing to be 100% on the provider and 0% on the wronged group. There's no attempt to meet half way, to find out what appears to be biased and how the model could be fixed. Instead, we're all supposed to retreat into our own walled enclaves and spend our lives griping about how the other group's getting more than we are.

To give a mundane example of how treatment differs within ethnic groups you just need to seek medical care in the US. Here the differentiator is not race or color but money, pure and simple. Providers struggle to provide adequate care for all among the masses but if you've got the wherewithall then your care and treatment options expand considerably.

[Falmari]

Social/economic factors

From the article:-

“In order to probe how racial disparities impacted the predictive models' performance, they tried to minimize the impact other variables, such as age or gender, might have on accuracy.”

Maybe age and gender are not the only data variables that have an impact. Could not social/economic factors also have an impact. Maybe their models do take these factors into account, but if they don’t the disparities of their modes could be on social/economic lines.

[unimaginative]

Re: Social/economic factors

Exactly what I thought.

It would be interesting to repeat this in other places, and to use a stratified sample to balance social and economic differences.

[Aitor 1]

Re: Social/economic factors

In the US white poor commit way less crime than black poor,and black rich also do more than black rich.

Part if the problem is the self reinforcement that comes from police looking quite closely to black people, and judges being harsher.

We could go on an on on the why, but without any context, and not caring about causes but facts, a model (not AI) will be "suspicious" of some minorities, etc.

[Eclectic Man]

Re: Social/economic factors

Umm don't you mean "In the US white poor are convicted of way less crime than black poor"?

Historically (and, I suspect, currently) in the UK and, I understand in the US, the law enforcement and judicial systems have been consistently biassed against black people. Black people are more likely to be arrested, convicted and get more severe sentences than white people with equivalent evidence.

[Anonymous Coward]

Re: Social/economic factors

I agree completely, save for the inner cities. Unfortunately, those trapped in the inner cities often have no choice but to resort to petty crime just to survive, and whether they do time has a lot to do with how they present in court and how lucky there are with getting counsel that actually cares a rat's patoot.

[Anonymous Coward]

Maybe imputing psychological and behavioral characteristics from images of undamaged brains just doesn't work? The phrenologists spent lots of time on it, too; for every negative result they had a positive one to show.

[Ordinary Donkey]

The publication actually goes one further and suggests that maybe the psychometric tests they are using to measure these characteristics are also a load of bunk, albeit using longer words. If your ground truth data is nonsense your AI will be nonsensical.

[Filippo]

My gut feeling is that maybe it works, maybe it doesn't, but either way it's a type of research that can very, very easily get skewed by subconscious prejudice. Unless every study they used in the training set was very accurately double-blinded, both in the quantitative measures and in the interviews, there's bound to be some skewing.

[LionelB]

Well, looking for neural correlates of behaviours and cognitive phenomena via neuroimaging has indeed attracted criticism as a "new phrenology".

The comparison is not entirely fair though, since the shape of your skull almost certainly has no bearing* on your cognitive phenomenology, whereas patterns of neural activity in the brain almost certainly do. There are certainly statistically-reliable studies which can detect, e.g., your conscious state (wake, sleep, coma, etc.), or the influence of certain drugs from neuroimaging data (I have been known to work in this area). It doesn't feel like a giant leap to propose that neural correlates of behavioural phenomena might be present and detectable.

*except, perhaps, in cases of severe injury, or some medical conditions

[Anonymous Coward]

Crazy

To reach for the term racist when talking about an algorithm is just plain wrong.

If the algorithm is not giving the answer you require, then are you asking the right question or is the algorithm just not good enough.

To imply there is a bias is just wrong it is a logic pattern and that is all.

The heading should have read

“Current AI / ML is still developing and will be for the next 500 years”

Still doesn’t really give the “click bait” effect

[gnasher729]

Re: Crazy

Sure, an algorithm is just some software running without the ability of being racist or nor (currently).

However, if trusting biased results of an algorithm has the exact same effect as being racist, then any non-racist person would want a very, very, very big warning sight attached to the algorithm. For example: “Warning: Racist algorithm”..

[Warm Braw]

Re: Crazy

To imply there is a bias is just wrong

Why? The results are demonstrably skewed. As the article says, there are lots of possible causes, including that there may be implicit biases in the available data. AI "algorithms" are essentially the result of their training data so if the data is biased then the algorithm will necessarily inherit that bias.

It doesn't have to be a specially racial bias, it could be a correlated factor, such as income: in the US in particular you'd expect there might be some discrepancy between practice in private and public hospitals either in the care with which the imaging was done or the attention paid to its analysis by clinicians.

This is obviously an important - as well as a fascinating - finding and to suggest it's futile even to look into it implies a wilful lack of curiosity.

[Ordinary Donkey]

Re: Crazy

The most obvious bias, given the trends persist even in the monoracial examples, would have to be in the Ground Truth. Somehow the people measuring and appraising the AA subjects are being less accurate than they are to the WA subjects, and not in ways that are predictable to an AI.

[LionelB]

Re: Crazy

The implication of the study, or interpretation of results, is that the issue here is not with the algorithm, but the data.

I do agree that the headline is dumb clickbait.

[Pascal Monett]

There is a "white" brain template ?

That is a dangerous thing to say, even if it is exact. A brain is a brain. I can't see how black people would have a brain that is structurally different from white people.

Of course, I'm not a doctor in the field, but it's a hard pill to swallow, and it brings us two steps away from "Jewish" brains and "terrorist" brains and then it's eugenics all over again.

They say they have a "common understanding" of how these systems normally work. I say they don't, because they cannot justify the results. It's just "machine says this".

That is because they have no log of how the procedures behave. It's a black box and, when said box spits out results that we find acceptable, we say it's working.

When I am confronted with a piece of code that I don't understand the behavior, the first thing I do is set up a log of its functions. When the function starts, what are its entry parameters, what results it sends back. I do a couple of test runs on different data sets, and then I analyze the log results.

In that way, I can understand how the code gets to its results, then I know what it is I need to modify to obtain the desired output reliably.

They're not doing this for their wonderful AI, so they don't know anything about it except what they expect as a result.

That is no way to manage a project.

[John 104]

Re: There is a "white" brain template ?

Sounds a lot like climate models...

[Khaptain]

A fully non white model

So why dont they do a model that uses only data from non white people ?

[Anonymous Coward]

Re: A fully non white model

That would be racist to the non colored people.

Anon because I'm sick and tired of snowflakes who can't take a joke.

[TeeCee]

Accuracy imbalance?

Accurately analysing for slight irregularities a dark blob on a light background makes it more likely to spot said irregularities than doing the same for a dark blob on a dark background?

Of course that would be unfixable if so[1] and thus not to be countenanced.

[1] As in the perceived bias actually being an inherent feature of the problem.

[wub]

Peeling the onion

OK, so folks decided to use AI to measure some human characteristics.

Then they looked at the data and said, "Oh, yeah, the training data sets are not diverse and introduce bias."

So they went back and trained using more diverse data sets.

Then they looked at the data and said, "Oh, yeah, our standardization methods use data that is not diverse and introduces bias."

I presume they are now going back and trying to eliminate bias in the tools they use to set the baselines for their measurements.

I wonder what they're going to find in the next cycle?

[LionelB]

Re: Peeling the onion

Well, hold the front page: Scientists Seek Explanation for Anomalous Result!

Out of interest, what would your explanation be?

[Version 1.0]

It's just human failure ... well maybe.

I've see this style of error for years now. I work in the human gait analysis field and there has been a very accurate study that shows that a young child's gait is different from an adult's gait ... I agree, I've seen this but the study was originally only based on child vs adult. As a result of biomechanics students being taught this, they assume that it's correct but don't remember the original testing. And so they assume that these days walking across the lab in a straight line is the same as walking on a treadmill and making sure you don't fall off it - it's similar but a totally different environment so the gait is only similar, not identical.

The errors described in this article are because AI is written by programmers/researchers who are just trying to get answers, when the answers look good then it's assumed that they are accurate - but nobody is verifying it. My approach, to students in these areas, has always been, "Show that the data is bad!" ... "and if you work at that and fail to show that the data is bad, then at last you have some evidence that the data might be good."

[Terry 6]

Algorithms

If my understanding is correct, an algorithm still functions in a 1+1=10 way, like any other programme.So if there is a bias in the underlying sample measurements, which could be just be something as simple as language use, by the subject or the researcher , this could/would reflect in the comparative score values that are collected.

[Alumoi]

Re: Algorithms

1+1=10

I see what you did there!

Have one on me.

[jemmyww]

One thing that it doesn't mention is that maybe it would be better to have skin tone specific models rather than one ML that is used for all tones. What if disease has different characteristics for different melatonin levels? (Obviously this should only be applicable to skin imaging MLs).

[M.V. Lipvig]

I'm no doctor,

and I don't play one on TV, but I don't see anything that precludes a brain difference based on other physical characteristics. There are plenty of precedents already. Digoxen for example is used to treat heart disease in men, but will almost certainly cause women to die of a heart attack. Malaria drug primaquine is highly likely to cause hemolysis, which causes red blood cells to self-destruct, in black people than white people. Certain diseases are far more virulent for some races and not others. Therefore, physical characteristics plays a part in everything. If they didn't, all diseases would affect all humans the same way. People need to accept that a, people are NOT all 100 percent equal in all things and b, there's nothing wrong woth it and c, it's not racist to accept that. Mother Nature is not a real person, and evolution is nothing more than a long series of mutations across generations that has no idea that race is a thing.

If the models are inaccurate by race, and the inaccuracies remain when the modeling data is equalized, I would look at the base brain assumption as the problem. Either they need a base brain by race, or the base brain isn't as "base" as they think.

[Eclectic Man]

Differences in brains

Just a thought. In the USA, black people have historically been restricted to particular neighbourhoods with worse infrastructure, worse healthcare and more pollution, particularly exhaust fumes and particulate matter from vehicles. Before tetraethyl lead was removed from petrol / gasoline there was considerably more airborne lead pollution near major roads. Each of these can affect brain development in any people who inhabit those locations (whatever the colour of your skin). If the models were generated from AA and WA populations with different development histories, maybe there could be subtle brain differences which might show up in scans.