Surprise, surprise: AI cameras sold to schools in New York struggle with people of color and are full of false positives

A Canadian security company apparently lied to officials at New York’s Lockport City School District about the accuracy of its facial recognition cameras, when the technology was installed across schools last year. Documents, obtained by Vice, show that SN Technologies’ CEO KC Flynn claimed the algorithm, id3, running of its …

[Shadow Systems]

Public confidence?

There IS none. AI is a load of bollocks in general, deep learning & machine learning algos are a bad joke, & facial recog is just another way for the Powers That Be to arrest you first & claim a reason for having done so after the fact.

The whole thing is a right clusterfuck that should be scrapped, restarted from scratch, & done correctly from the get-go.

Nobody should be allowed to use it until the false posative rate is below 10% at the worst of times & under 1% at the best. Those it flags should then be gone over with a judicial fine toothed comb to verify/reject if the person flagged is the person sought. "You're looking for a white male about 20 years of age, 1.98 meters tall, 80 kilos in weight, with a tattoo of a dog on his cheek. THIS is a 75 year old black man who's all of 1.25 meters tall, 50 kilos, & has no facial tattoos at all. On what planet does this make a match? Get out of my office before I have you arrested for being a fucking idiot."

*Cough*

Bin it, scrap it, start over.

[sgp]

Re: Public confidence?

Might as well leave out the start over.

[Joe W]

Re: Public confidence?

"[...] make sure the systems in place are accurate, transparent, understandable [...]"

Hahahahahaha.

Yeah. Right. Transparent. Understandable. AI.

[Keven E]

Re: Public confidence?

That whole executive order is full of *it.

Especially the "our Nation's values" part (capitalized "N" for some reason) which morphs into "Government use of such products..." (common commercial products) "...must nevertheless comply with... *American values; and..."

East coast? West coast? North and/or South American?

Sounds like plausible deniability.

[BinkyTheHorse]

Re: Public confidence?

The Dunning Kruger effect in this post is off the charts. You do realize machine learning is significantly older than deep learning? Or that most "machine learning algos" have a degree of interpretability that makes neural networks an outlier in this regard? Or that significant work has been put into making deep learning more debuggable and interpretable (the "dog in snow" paper is 4 years old FFS)?

This is essentially like wanting to scrap the entirety of software development after experiencing the multitude of shitty mobile apps. Don't blame the tool for the tools that use it wrong.

[Pascal Monett]

When the tool is not fit for purpose, there is no way to use it right.

[Lusty]

Re: Public confidence?

"scrap the entirety of software development after experiencing the multitude of shitty mobile apps"

Honestly, most days I'd vote for that, seeing how bad pretty much all modern software is. We've had half a century to get it right and we're further away now than ever.

[Fruit and Nutcase]

Re: Public confidence?

"Bin it, scrap it, start over."

Where did you get that slogan? Did you break into the computers at No. 10? That's supposed to be Top Secret - to be announced by Boris around the 2nd of January 2021

[Yet Another Anonymous coward]

Re: Public confidence?

>On what planet does this make a match? Get out of my office before I have you arrested for being a fucking idiot."

You show it a police training set where 90% of the targets are black and let it loose in a population where 10% of the population it sees are black, it is going to choose the wrong targets. It isn't the software's fault.

[Paul Hovnanian]

Re: Public confidence?

"You show it a police training set where 90% of the targets are black"

You don't train a facial recognition system off of the eventual search set. You train it with a representative cross section of the general population.

[Yet Another Anonymous coward]

Re: Public confidence?

>You don't train a facial recognition system off of the eventual search set.

Obviously. But the problem is all school shooters are white.

So given that school shooters are also criminals you can train it on a test set of photos of criminals and it will find school shooters - simple.

[IGotOut]

Eh?

Facial recognition stops school shootings?

There was me thinking it was mentally unstable kids having easy access to guns.

But hey no, it's they look eerrr like eerrr...ok how does it tell the difference between a pupil at the school going to class and the same pupil at the school ready to kill someone?

[DwarfPants]

Re: Eh?

must be detecting some sort of frown.

[iron]

Re: Eh?

Well if you'd read the article you'd have seen that it has a false positive issue where broom handles are detected as guns. So obviously it helps prevent school shootings by detecting any kid carrying guns as they walk into school.

Ok so its more like catch them before they shoot too many people rather than completely prevent shootings but short of Minority Report what else could it do?

[Falmari]

The Janitor did it

"false positive issue where broom handles are detected as guns"

Will suck to be a janitor.

No longer will it be the butler did it now it will be the Janitor did it

[John Brown (no body)]

Re: The Janitor did it

"No longer will it be the butler did it now it will be the Janitor did it"

And he'd have got away with it too if wasn't for those pesky kids and their dog!

[Falmari]

Re: The Janitor did it

No scooby snacks I am afraid so have an up vote.

[Paul Hovnanian]

Wait, what?

"it has a false positive issue where broom handles are detected as guns"

But perhaps not false if it classifies them as weapons. I can't remember a time when I went to school where I needed to carry a broom handle onto campus.

[doublelayer]

Re: Wait, what?

There are various long cylindrical things people might want to take to school with them, and many more things that could look long and cylindrical if the camera doesn't get a good picture. For example, people who walk in a rainy area might carry umbrellas with those long poles so you can carry one over your head. Or maybe they're carrying sporting equipment to use the recreation areas of the school. Or a camera or microphone stand they used to record something elsewhere and now need to return. People who have withstood injuries could use crutches with the long pole part. Meanwhile, someone who did plan to take a weapon could, if the weapon is small enough, put it in a bag; most students I know carry bags. With these provisos, there's a serious question if the cameras really serve a useful purpose. Even more so as they're also scanning faces. I don't know why they're doing that; most school shootings are perpetrated by people who previously went to the same school, just not with weapons. Given the reliability of other facial recognition systems, I hope it's not to verify that the students entering are all known by the school; there'd be queues of students waiting forever for the camera to recognize them.

[Michael Wojcik]

Re: Wait, what?

I can, though I admit it was attached to the broom head.

Brooms do have uses, you know.

[Keven E]

Holy cylinder!

It seems a lot of attention could be on the baseball team on this side.

[a_yank_lurker]

Re: Eh?

It's not going to stop school violence. At best it can sort of identify a person but it cannot determine what anyone plans to do. There are ways to lessen school violence but they all require adults to take responsibility and act like adults.

[AMBxx]

Take a broom handle to school

Easy way to defeat this - let all the kids know that broom handles are detected as guns. Next day, they all turn up with a broom handle. Broken system. Job done.

Or have children changed in the last 20 years?

[Denarius]

Re: Take a broom handle to school

Err, no. this is MerkinLand. Kids would probably get shot at door, you know, for sake of the children

[Michael Wojcik]

Re: Take a broom handle to school

At the very least they'd be suspended, because Zero Tolerance.

[Flywheel]

Stage 2 will be incorporating these...

Automated Machine Gun Targets People from 1.5 miles. They'd never get past the school gates.

Win!

[ITS Retired]

Re: Stage 2 will be incorporating these...

Oh goodie. Another way to kill people. Just what the world needs. /sarcasm

[John Brown (no body)]

Re: Stage 2 will be incorporating these...

Apparently, the Iranians are claiming something like that, controlled from a satellite, is what killed their top nooklear scientist.

[trevorde]

Interesting edge case

https://www.theguardian.com/us-news/2016/aug/25/cocks-not-glocks-texas-campus-carry-gun-law-protest

[Falmari]

Re: Interesting edge case

So rather than raise you hand to ask a question as its Texas you raise your gun above you had and fire a shot in the air.

But if the cocks-not-glocks protest get their way it it will be something else you raise to ask a question.

Now that would certainly be called an interesting edge case.

[Cuddles]

Grading on a curve

"It ranked 49th out of 139 in tests for racial bias"

If all 139 systems suck, it really doesn't matter how they rank relative to each other.

[Eclectic Man]

AI and satellite controlled gun

From the Guardian:

https://www.theguardian.com/world/2020/dec/07/mohsen-fakhrizadeh-iran-says-ai-and-satellite-controlled-gun-used-to-kill-nuclear-scientist

"

The assassination of Iran’s top nuclear scientist last month was carried out remotely with artificial intelligence and a machine gun equipped with a “satellite-controlled smart system”, Iranian news agencies quoted a senior Iranian commander as saying.

Ali Fadavi, the deputy commander of Iran’s Revolutionary Guards Corps, told Iranian news agencies that Mohsen Fakhrizadeh was driving when a weapon opened fire on his car on a highway near Tehran. The weapon “zoomed in on Fakhrizadeh” using an “advanced camera”, Fadavi said. “No terrorists were present on the ground.”

...

Fadavi said the gun used to kill Fakhrizadeh had been placed on a pickup truck and controlled by a satellite, and had fired 13 shots. “During the operation artificial intelligence and face recognition were used,” Fadavi said. “His wife, sitting 25cm away from him in the same car, was not injured.”

"

Scary.

[Yet Another Anonymous coward]

Re: AI and satellite controlled gun

Also worth noting that this is hardly a unbiased report.

[Fruit and Nutcase]

Re: AI and satellite controlled gun

With all due respect, and no disrespect intended to the subjects, but given the country, what attire was the lady wearing? Depending on that, it may be an easier task to differentiate between the 2 occupants. However, a moving vehicle, and optics through the windscreen would make things difficult. If computer-aided, it would probably be easier to get the system to target the occupant on the driver's seat/aim at the driver. And given that the device was alleged to have been mounted on the back of a pickup truck, then, there would have been human actors able to verify the identity visually and take the final decision to go ahead with the plan. In terms of technology, nothing too advanced or onerous (motion tracking)

[Bitsminer]

Re: AI and satellite controlled gun

"advanced camera", "terrorist", "machine gun", "pickup truck", "satellite", "artificial intelligence", "face recognition"

They forgot "lasers" and "agile development".

[John Brown (no body)]

Re: AI and satellite controlled gun

Or the big, heavy blockchain used to close off the side street, forcing the car into the "killing zone"

[Aussie Doc]

Re: AI and satellite controlled gun

Not to make light of the situation but this is the same Iran that showed some dude with a 'gadget' that claimed to detect COVID-19 which looked remarkably similar to the fake bomb detector gun.

Sorry about the daily mail ref:

https://www.dailymail.co.uk/news/article-8230139/Iran-parades-coronavirus-radar-looks-like-fake-bomb-detector.html

[Yet Another Anonymous coward]

Also if your top nuclear scientist was shot in your own capital city by the enemy and you are head of security - it's probably best to claim he was killed by a secret invisible robot satellite magic death ray