Cops responding to ShotSpotter's AI alerts rarely find evidence of gun crime, says Chicago watchdog

Police responding to ShotSpotter's AI-generated alerts of gunfire find evidence of actual gun-related crime only about one time in ten, a Chicago public watchdog has found. The California biz uses machine-learning algorithms to determine whether loud bangs caught by microphones deployed across more than 100 US cities are …

[Anonymous Coward]

And we're surprised?

I'm not.

And as for the response from Shotspotter:

“The OIG report does not negatively reflect on ShotSpotter’s accuracy which has been independently audited at 97 percent based on feedback from more than 120 customers."

What that tells me is that 120 customers don't want their purchase questioned.

Not present in the survey, I assume, is the original customer, the Department of Defense, which found the software worthless and terminated the contract.

[Geez Money]

Re: And we're surprised?

There's also the constitutional element. In Canada, where there is something very similar to the 4th amendment, the police rejected ShotSpotter as being a clear example of unlawful search. I wonder if it would survive a constitutional challenge in the US?

[The Man Who Fell To Earth]

Re: And we're surprised?

The fact that the system initially reported the location as being a mile away from the "corrected" location says it all about "accuracy" in this case. The only objective data they had to base the "correction" on was the presence of a dead body a mile away that was dead due to a gun shot, and no closer "event" near the right time in the ShotSpotter system. That is the very definition of fudging the data.

[James 51]

Feels like a hammer meet screw sort of situation.

Would be interesting to know if that 97% accuracy includes staff (and not the software) 'screening' recordings after talking with the police and overriding or reclassifying the results.

[Geez Money]

"We're 97% accurate when we're allowed to change our results to the correct ones 97% of the time"

[Cuddles]

Where "correct" translates as "the answer the customer requests".

[Psmo]

"We have an accuracy of 97% not counting the times the prosecution has wirhdrawn the results as obviously wrong"

[Anonymous Coward]

More to the point

What they mean by "accuracy" needs to be defined. The Chicago data, where they did define what they mean by accuracy, would say it's very low.

[Tromos]

Compensation

I'm hoping the guy that lost almost a year of his life sues them and takes a goodly chunk out of that 33 megabucks.

[UCAP]

Re: Compensation

Actually I would have thought that he would have a case if he complained about "attempted perversion of justice" or "falsification of evidence". By rights, whoever massaged the SpotShotter data to try to stitch him up should be looking at being sent down for a serious length of time.

[John Robson]

Re: Compensation

I would rather hope that he doesn't have to sue...

Not just any dismissal this, it's a case hinging on only one piece of evidence, and that piece of evidence was known to have been "manually edited".

[Anonymous Coward]

Irreconcilable differences

The manufactures claim of 97% accuracy is clearly facile and inaccurate. It is also glaringly self serving.

The proof of this is the uninterrupted string of negative results in the real world. Much like overhyped facial recognition systems and pre-crime behavioral analysis snake oil, the Shotspotter can't do what it claims. All it does is semi successfully is detect loud noises. Worse, like many other recent cases, there is a pattern of them fitting the data to the intended outcome. This is similar to problems with other forensic evidence(like bite mark "experts") whose methods are both flawed and biased. The police are buying these tools because they can use them as an excuse to continue patterns of abuse and policies that have been increasingly scrutinized and restricted.

The fact the Shotspotter is a black box that is shielded from further scrutiny makes it all the more attractive.

A system with a >90% failure rate isn't trustworthy, and there is a clear pattern of bias in the companies results. They favor results that match what the company thinks that law enforcement wants to hear. Like voice stress analyzers and so many other bogus devices law enforcement waste our money on, this device is a scam, pure and simple.

Don't expect this problem to fix itself either, push your representatives to bar the company from use by state and federal agencies. This company ruins innocent peoples lives as a service, and needs to be shut down.

[Geez Money]

Re: Irreconcilable differences

The problem will fix itself when criminals notice that playing gunshot sounds into these microphones is an effective denial of service attack against the police.

[John Robson]

Re: Irreconcilable differences

It's not a *pure* scam - it does detect gunshots... it's just that it seemingly can't distinguish between those and fireworks, cars backfiring, possibly even someone clapping near a mic.

There will be a huge market in the criminal fraternity for starting pistols and blanks - and the same should be true for the police. Have a series of officers do the same, and some fire fireworks etc., and see how quickly and accurately the reports flow in.

97% accuracy may be true, it's just that the accuracy referred to is the sensitivity, not the specificity. And of course that number would rely on 100% of shots being recorded to know about the 3% they missed - which clearly isn't the case, since otherwise the system wouldn't be needed at all.

So yeah, not a *pure* scam, just a scam.

[Cuddles]

Re: Irreconcilable differences

"It's not a *pure* scam - it does detect gunshots"

Is there actually any evidence for this? It detects some gunshots, sure. But all the analysis seems to focus on the false positives. What about the false negatives? It regularly detects gunshots that don't exist, how often does it fail to detect gunshots that actually happened?

[John Robson]

Re: Irreconcilable differences

“ 97% accuracy may be true, it's just that the accuracy referred to is the sensitivity, not the specificity. And of course that number would rely on 100% of shots being recorded to know about the 3% they missed - which clearly isn't the case, since otherwise the system wouldn't be needed at all.”

I think it’s highly doubtful that it could pick up fireworks and *not* gunshots.

How many are picked up is hard to determine without knowing how many there actually are - and if we knew that then we wouldn’t need the system.

Hence a real world testing regime would be rather useful.

[Cuddles]

Re: Irreconcilable differences

"I think it’s highly doubtful that it could pick up fireworks and *not* gunshots."

Sure, but we don't know how many fireworks it picks up either.

"How many are picked up is hard to determine without knowing how many there actually are - and if we knew that then we wouldn’t need the system."

I'm not sure I understand this point. The system is supposed to identify when and where a gunshot occurs, to allow police to respond as soon as possible without waiting for someone to call. Whether you know how many occur in total is completely irrelevant. Even if we had perfect knowledge of how many gunshots occur per week, that wouldn't help anyone actually identify them in real time. So the two questions are almost entirely orthogonal - we can count the total number of shots fired independently of locating them in real time. We can then compare that total with the number actually identified by this system. It's entirely possible that this data does not currently exist, but all that means is that no-one has bothered to collect it, not that this system is required in order to do so.

[Electronics'R'Us]

Re: Irreconcilable differences

The fact the Shotspotter is a black box that is shielded from further scrutiny makes it all the more attractive.

Some years ago, DUI defendants made motions for breathalyser source code to be made available so they could (from the 6th amendment) 'confront their accuser'.

I am not sure how successful that was in the long term (I do remember a district court issuing an order for disclosure).

Given that in the case here, the only evidence was from this black box (suitably tampered with later), perhaps being faced with that from anyone accused by this 'black box' might get a response from the DA's office.

[katrinab]

Re: Irreconcilable differences

In the UK, you are entitled to ask for a blood sample to be tested independently.

The breathalyser is sufficient for the police to have reasonable suspicion to arrest you, but it is the blood sample that is the beyond reasonable doubt evidence for conviction.

[katrinab]

Re: Irreconcilable differences

They are not necessarily lying when they say it is 97% accurate.

The vast majority of noises this thing will pick up are not gunshots.

Lets take a completely made up number for illustration purposes and run with it.

Suppose 1 in every 1,000,000 noises in Chicago is a gun shot, and the rest is engine noises, people talking, closing doors, power tools, all the other noises you hear in a city.

For a sample of 100,000,000 noises, 100 would be gun shots, 99,000,000 would be other things.

A 97% accurate system would pick up 97 of the 100 gun shots, but also mis-identify 2,970,000 of the other noises as gunshots. That would mean that of the 2,970,100 noises it identifies as gun shots, 100, or 0.0034% are actual gunshots.

[Geez Money]

This was discussed in the shotspotter thread a few days ago, but the company guarantees 80% or your money back. I would really like to see Chicago call them on it. I'm sure they'll try to slime their way out of it and claim that it was 80% accurate but the cops were just bad at getting there in time or some similar nonsense. Doubt it'll work though.

[Chris G]

I doubt the CPD would call them on it.

Let's face it, any time the cops get a shots fired alert, it gives them the right to go into the alleged target area in response mode.

They will also have probable cause to do whatever 'fishing' they need to do to investigate the reported shots.

I wonder how often shots are reported that are in proximity to persons of interest?

[jake]

"I wonder how often shots are reported that are in proximity to persons of interest?"

Well, seeing as shooting a firearm within the city limits of Chicago is illegal, I would think that pulling the trigger automagically makes you a person of interest. And on top of that, strangely enough normal people don't wander around Chicago shooting guns. In fact, I'd venture to say that everyone[0] firing a gun within city limits is a criminal before, during and after pulling the trigger. Most cops in the line of duty and people at registered/licensed ranges excluded, of course.

Don't get me wrong; the system is still snake-oil IMO.

[0] With exceptions being so rare as to be statistically insignificant.

[Chris G]

@Jake

I was wondering how often shots were reported by Shotspotter? Not necessarily the same thing as shots fired.

I have come across cops in the UK turning up at persons of interest's abodes, claiming there was a report of a disturbance/domestic/break in etc when everything was quiet.

[Anonymous Coward]

Discharging a firearm inside the CoC(k) is not on its face illegal, in and of itself. The individual circumstances surrounding a discharge are important.

But yes, discharging a firearm shall bring (un)wanted attention towards the reports. If you're a criminal, the attention is unwanted. If you're a victim, the attention is greatly sought.

As a CCL holder in Illinois and Illinois being a "Castle Doctrine" state, if I fire my carry weapon in legitimate defense (there are a lot of instances where deadly force is NOT allowed in Illinois), I am not a criminal.

If someone breaks into my residence and I discharge my weapon towards the intruder(s), I am not a criminal. (However, my ears would probably disagree.)

However, in no scenario shall a civilian inside the CoC(k) be allowed to carry or possess an assault rifle, so I guess you could be correct on that technicality if that is what was fired. Along with tasers and lasers.

[Trollslayer]

And how many are people of colour.

[jake]

PDNFTT. Ta.

[Geez Money]

That's a pretty good point to be fair. Nothing is going to make police happier than something which provides the opportunity to go play soldier. I do, however, suspect that when all of those cases get thrown for not having had any actual probable cause it might change their opinion.

[DS999]

Only 10% effective?

I'm sure they do better just waiting for someone to call the cops saying "I think a heard a gunshot". Even if half the time it doesn't get reported because people are numb to it in certain neighborhoods, sometimes turns out to be fireworks, or they can't pinpoint the direction they still are going to beat 10%!

[EarthDog]

Re: Only 10% effective?

Or doesn't get reported because they don't trust the cops.

[Joe W]

Reading comprehension?

"“The OIG report does not negatively reflect on ShotSpotter’s accuracy which has been independently audited at 97 percent based on feedback from more than 120 customers. Nor does the OIG propose that ShotSpotter alerts are not indicative of actual gunfire whether or not physical evidence is recovered.”

1. it does refelct negatively on the accuracy if less than 10% of the marked events are actually events

2. it does thusly propose the software is inaccurate

If I was to roll out a warning system in production that poor I'd have my head handed to me.

And I'd totally like to quote the signoff of a Chicago guy with a youtube channel on botany, but I hestitate to use profanity...

[Kevin Johnston]

Re: Reading comprehension?

It may be just that the microphone was not calibrated correctly or there may have been some manual editing but I am sure that quote ended with the word 'Yet'

[Pascal Monett]

"as it no longer had sufficient evidence"

The case never had sufficient evidence to start with.

Sufficient evidence would have been finding the gun.

It's rather obvious that, if ShotSpotter had correctly identified a gunshot in the immediate vicinity of the car, it was because of the shooting. That does not make the driver guilty of the shooting.

He never should been jailed in the first place.

[Anonymous Coward]

Re: "as it no longer had sufficient evidence"

Correct. You would have thought no gun, no gun shot residue on the accused and no gunshot residue found in the accused vehicle would have been enough to get him out on Chicago's "Affordable Bail" program.

Hell, real killers are getting released on $10-50K bonds. 10% deposit required for release, so just $1-5K to free someone accused of murder.

Right now Chicago courts have what, 40-something people YTD who have been charged with a violent crime, released on affordable bail, only to end up arrested of another violent crime while on affordable bail for the first crime.

[Alpharious]

Re: "as it no longer had sufficient evidence"

This is chicago. It's not about evidence or justice, it's about gaslighting the public. Homan square was a direct violation of the constitution and it was memory holed after 2016. They didn't care if shotspotter was accurate, they only cared if it held up in court. Now we have to review other cases in which that was evidence and determine if justice was done.

but that won't happen. people will forget about this in a week. Forget it Jake, this is chirak.

[jake]

Re: "as it no longer had sufficient evidence"

"Forget it Jake,"

I'm jake, not Jake (Jake apparently doesn't post here anymore). But assuming you meant me, what would you like me to forget, and why?

"this is chirak."

No. It is not.

[Gotno iShit Wantno iShit]

The watchdog said 50,176 alerts generated by ShotSpotter in Chicago between January 2020 and May this year probably were the result of gunfire

How many of these shots were fired by the police?

[Anonymous Coward]

I chuckled, so up arrow for you.

For data, looking at the COPA website, I see 37 Firearm Discharge logs created for 2020 and another 19 through the end of May 2021.

It's been a while, but I'm pretty sure that COPA (and their predecessor) investigate _all_ firearm discharges.

However, each "log" may contain multiple TRR (Tactical Response Report, which is where firearm discharges are documented), one for each officer involved in a single case. So, for a large response, there could be five or more officers who discharged their service weapon attached to a single log.

I am really curious how large of an increase in ShotSpotter reports around 4JUL due to actual fireworks. Of course, shootings also rise considerably during the 4th in Chicago as well -

2021 - 23 killed, 90 wounded [0]

2020 - 21 killed, 71 wounded [0]

Also, I wonder what ShotSpotter considers a report? Is there one generated for every single sound that it registers as a "probable gun" or does it group multiple noises in close proximity (both time and location) as a single report?

Of course, being statistics, some phrase about lies and damn lies come to mind.

[0] Stats provided by HeyJackass! :)

[Anonymous Coward]

Cross referencing opportunity

And how many of the recorded CPD firearm discharges did ShotSpotter report? How accurately? How many did it fail to spot?

The data's there to look at the true-positive/false-negative error rate, although only in a fairly narrow range of situations.

[teebie]

"ShotSpotter has detected hundreds of shootings that would have otherwise gone unreported."

Did the spokesman continue with "...tens of which weren't imaginary"?

[Electronics'R'Us]

Slight correction

Using ShotSpotter, CPD receives real-time alerts of detected gunfire a loud noise that we will use as an excuse to arrest people enabling patrol officers to arrive at a precise location of a shootingquota filling event quickly.

[Wade Burchette]

How is that working out?

"In order to reduce gun violence, knowing where it occurs is crucial." Yeah, and how is that working out for Chicago? It has one of the highest incidence of gun violence in the entire United States, if not the highest. Remember that Chicago was also home to Al Capone who said "You can get more with a kind word and a gun than with just a kind word". Or something like that.

A system so effective that it puts innocent people in jail while doing nothing to stop real gun violence. Why is Chicago paying for Shot Spotter again?

[John Brown (no body)]

Re: How is that working out?

I wonder why RoboCop was set in a future Chicago? Is Ed-209 under development there? Clearly CPD will be all over anything like that when it becomes available, despite the "documentary" evidence already available :-)

[Nightops25]

Re: How is that working out?

RoboCop was in Detroit, a similar but slightly different hellhole of a city.

[John Brown (no body)]

Re: How is that working out?

Ah, well, at least I got the right country :-)

[Someone Else]

Re: How is that working out?

Why is Chicago paying for Shot Spotter again?

Something about brown envelopes....

[KBeee]

There must have been a conversation a bit like this -

CPD "We arrested a guy suspected of shooting a passenger in his car who was seen on CCTV at XYZ Street at 11pm, but we didn't get a gunshot alert"

Shot Spotter "We only have data for a firework going off about a mile away from that location at about that time"

CPD "Are you sure?"

Shot Spotter "Wait a minute, I'll have the data re-evaluated"

Shot Spotter "Oh wait! After re-evaluating the data it turns out that it wasn't a firework a mile away, it was a gunshot at where-did-you-say? XYZ Street?"

[Anonymous Coward]

The basic problem is that gunshots don't leave much evidence and those doing shooting don't stick around. Even 30 seconds means the shooters can be a half-mile away before the police arrive. My guess is the cops take several minutes to get there.

The sounds a gun makes, unlike a bite mark, is very distinctive and not easily duplicated, so I have confidence when they say it detects a shot that there was a shot. If it was fireworks, those leave a significant amount of haze, unlike smokeless powder used in guns.

What is not in their capability is to make the shooter stay at the scene for apprehension and it is this operational issue that prevents its use as a certainty in law enforcement.

The company doesn't ruin anything. Law enforcement lying about the results and prosecutors accepting police fabrication of evidence does.

[Geez Money]

Bullet holes, casings and unburnt powder should be plenty of evidence to know that a firearm was discharged.

[MachDiamond]

"Bullet holes, casings and unburnt powder should be plenty of evidence to know that a firearm was discharged."

If those things can be found. The system isn't going to be mm accurate and the police aren't going to want to spend a hour or two looking around to find a casing or determine if a hole in something was caused by a bullet and a recent one at that.

The number of cars backfiring these days is not that great. With modern fuel injected cars, it's almost unheard of (pun intended). Fireworks can be an issue, but they don't have the same characteristics of a typical gunshot.

The system is just a tool and has to be used correctly to be effective. It might get used as a diversion by criminals trying to cover activities on the other side of town. Send one person to fire a few rounds into the air to get all of the cops to rush over there and then hit the bank/jewelry/liquor store. It could buy them 10 minutes or more. At the same time, if police know that one gang is wanting to extract revenge against another gang, getting reports of gun shots in the right area will tell them the perps are mixing it up.

[Geez Money]

There's a point where a tool is counterproductive even if it helps sometimes, the proverbial broken clock for instance. If it's diverting money and police time to the wrong place often enough it's a net negative. If it can't be "used correctly" majority of the time or the police spend more time working for it than it does for them, then it should be ejected.

[a_yank_lurker]

In a city, the acoustic signature and direction could easily be distorted to give a fair number of false positives and negatives. I would trust someone calling in saying they saw the shooting or they have a gunshot victim than the acoustic signature from a loud noise.

My guess, whatever testing was done was not done in an urban area. It works great in a field but not so much in a city.

[Falmari]

@AC For argument’s sake let’s say the 90.9% where no evidence was found were indeed gunshots. The end result, 90.9% of the time, is a waste of police time and resources. Because no evidence means no further investigation.

Now 9.1%, 4,556 did indicate that "evidence of a gun-related criminal offense was found". But how many call outs found "evidence of a gun-related criminal offense” in the years before ShotSpotter was deployed. How many more has ShotSpotter found to justify $11m per year and all that waste of police time.

So, I question if $11m per year paid to ShotSpotter from CPD’s budget would not be better spent elsewhere within the CPD.

[martinusher]

First generation tool is effectively a prototype out for beta testing

The basic idea behind Shot Spotter is sound but it seems to have made the leap from laboratory project to wide deployment without going through enough testing to verify that its delivering the information that it promises.This seems to be a common problem with every first generation product, products that often fail because they're essentially unworkable but the idea is so tied up in overpriced IP that it becomes uneconomic to develop it further.

So now police departments are stuck with marginally functioning and probably rather expensive (leased rather than bought) systems with the people responsible for their procurement ever more defensive as real-world experience shows that the original promise doesn't match the practical reality. Everyone's going to get very defensive before they finally admit they have a problem (its the same issue with police violence -- everyone knows it happens but without 100% cast iron evidence allegations are just dismissed out of hand -- and even with the evidence its automatically contentious).

Incidentally, the statistics for "gun related criminal activity" will gloss over the fact that in most urban jurisdictions just firing a gun is a criminal offense. Some cultural groups like to fire guns into the air to celebrate stuff which is discouraged because "what goes up comes down". So loosing off a clip has now become macho....all grist to the Shot Spotter mill.

[HenryBowman]

Critics have been reporting the same deficiencies in ShotSpotter for at least five years.

https://www.forbes.com/sites/mattdrange/2016/11/17/shotspotter-alerts-police-to-lots-of-gunfire-but-produces-few-tangible-results/

But like red light cameras, the municipalities who have purchased them have an ulterior motive for keeping them deployed.