An open letter to crimrxiv

At this point, virtually anyone working in the research field can merely glance at Flock’s original white paper and know that there’s not a grain of salt large enough to take it with. For starters, it was written by Flock employees who hired 2 academic researchers for consultation. You’ll notice that ResearchGate lists those researchers as “authors”. Those researchers have even come forward explaining the problems with the paper and their involvement.

Giving Flock credit for doing the right thing is something I hadn’t imagined I’d ever do, but in the last few months as pressure continues to grow from all sides, their damage control has sometimes included taking accountability and quietly remediating some of the mistakes they’ve made while rapidly scaling.
In this case, just a few days ago, they removed links to the aforementioned white paper.
That’s the type of thing only someone as pedantic as me would notice. And I did. Kudos, Flock. 🫡
Unfortunately, I’m going to have to take that kudos point back, because Flock recently published this blog post.

As law enforcement agencies fight tooth and nail to keep their dystopian toys, a lot of special interest groups have been summoned to “produce research”.
I’m going to examine this new study that Flock is promoting not as a predictable role in opposition to Flock, but for the purpose of improving the study before publication and actually understanding these problems and solutions.

But first, for the sake of transparency, I need to explain my position:

I have been extremely critical of Flock and other LEA surveillance vendors. I am, without a doubt, biased against the vast majority of the government’s usage of surveillance on innocent civilians. Through the lens of research, this bias is a problem that I strongly recognize, self-critique, and consistently try to mitigate.

I’ve spent nearly 2 years researching the efficacy of ALPRs while turning down every grant and endorsement I’ve been offered,. If my research is biased, all of that work and out-of-pocket expense will be next to useless. I’ve been asked to expedite, prepublish, and share this research with city councils, attorneys, politicians, and the United States Congress. The answer has been and still is “no”.
This is not because I have an upstanding threshold of research or infallible ethics, but because I want to know how effective ALPRs are more than I want to win an argument. The truth is, useful research is seldomly sexy.

I’m not going to attack the character of other researchers in this post, but it should be acknowledged that there are extensive bias problems that are not disclosed. Moving on:

I present you: Automated License Plate Readers, Vehicle Theft, and Clearance (Working Paper)
It’s rough, unaudited, and appears to not be anywhere near the prepublish stage. It arrives to a conclusion while lacking any type of disclosure statement. It is, technically, as academically valuable as this blog post.
To the credit of Scott and Ian, that’s not a problem, like, at all. They’re merely sharing their progress. I would maybe roll my eyes at including a “conclusion” in this progress, but they could’ve also been encouraged to by the University.
What is a problem is that it is being cited and referenced ad-nauseum as a research paper.

The Manhattan Institute wrote an extensive piece without bothering to mention the paper’s unpublished status, even citing conclusions that the paper itself would not. For the sake of preserving my own faith in society, I need to assume that any researcher, legislator, or person interested in objectivity would read the words “Manhattan Institute” and move onward to find a source that isn’t plagued with discreditation. So let’s return to the working paper.

Here are my unsolicited requests to improve the usefulness of Automated License Plate Readers, Vehicle Theft, and Clearance.

Confront Confounded Causality
The study claims that ALPR networks increased solved motor vehicle theft cases (arrest clearances) by 15.9%.
Okay. cool, but the authors' own data shows that this upward trend in solved cases began three months before the cameras were actually turned on.
At a glance, this is lacking precision and should be considered. But when you dig deeper into the context, it suggests that other concurrent factors such as LEAs launching anti-theft task forces, increasing patrol hours, or retraining officers in anticipation of the cameras may have been the actual drivers of the success. This needs to be carefully collected, analyzed, and verified before a conclusion.

To add to this thought, this is analyzing arrest clearances. That’s not the honeypot that someone looking for pro-ALPR data might think it is.
Tying ALPRs to vehicle theft recovery is, in my own research, actually pretty legit. It’s also what I personally find the most powerful, as most victims of vehicle theft just want their car back and are less interested in the case clearance.
However, ALPRs aren’t all that useful for clearances as the vast majority of recovered vehicles are abandoned. The actual stat here varies by region, but it’s generally safe to say that 8 out of 10 recovered vehicles do not have a person inside the vehicle to arrest. Since the crime is infrequently cleared and we see a spike in clearances, this further suggests confounding causality.

I Smell Statistical Illusions
The paper states that the cameras reduced the time it takes to recover a stolen vehicle by 0.28 days. But this metric only applies to the subset of cars that were actually recorded as recovered (a "conditional" metric). When Scott and Ian looked at all stolen vehicles combined, there was zero statistically significant improvement in recovery times. The cameras only sped up the recovery of vehicles driven carelessly past a lens, remaining completely blind to the vast majority of vehicles hidden in garages, chopped for parts, or abandoned off main roads.


The Disclosure Will Need Exhaustive Auditing
To determine when and where cameras were active, the researchers relied entirely on private operational logs supplied by Flock Safety.
Because this company has an obvious financial incentive to prove its product works and has previously repeatedly manipulated statistics to do so, this reliance introduces selection bias. Independent researchers cannot audit this data to see if Flock omitted failed deployments, ignored camera downtimes, or excluded canceled contracts from the dataset. To me, based on the aforementioned track record, data supplied from Flock is not reliable until proven otherwise.

NIBRS Data Requires Macro-Adjustment
The study measures crime using the National Incident-Based Reporting System (NIBRS) during a highly volatile transition period. NIBRS struggles to track cross-jurisdictional crime. If a car is stolen in a city with Flock cameras but recovered across state lines by an agency using a different system, that recovery may never be recorded. This creates systemic blind spots that limits the scope of the study.


False-Positives Matter!
The study treats the camera alerts as absolute truths. It ignores independent audits showing that Flock cameras misread license plates anywhere between 10% and 71% of the time due to weather, dirt, and poor lighting. This omission is critical and jeopardizes statistical error margins used to produce a conclusion.

Lack of Epistemological Framing
The lead author served for 20 years as a police officer and Deputy Chief. While this provides operational expertise, it shapes the study's framing to focus entirely on "arrests made" while completely ignoring the collateral damage of the technology. Since clearance is already a problematic statistic when it comes to ALPRs and vehicle theft, this isn’t as much of a dispute of the conclusion as it is encouragement to examine the impact and efficacy from the context of a civilian.

I realize that it’s impossible for me to write something like this without it being digested with a tone of snark, but I invite Scott and Ian to join me in saying “no” when asked to publish incomplete research. If the goal is truly to make the world a safer place, then we need researchers to help navigate policy in a way that is in parity with reality, not a company’s or legislator’s timeframe. All research, especially in areas of sociology, should never be framed in a way where “bad results” could be yielded. Proper research always yields good results, even if those results simply help design future methodologies.