By Erin Sherbert
By Howard Cole
By Erin Sherbert
By Erin Sherbert
By Leif Haven
By Erin Sherbert
By Chris Roberts
By Kate Conger
But the Seattle Police Department's chief of information technology, Mark Knutson, appears to have recognized the impropriety of making these public relations agreements. He wrote to PredPol's Coonerty in November 2012, saying "I don't think we can make this sort of thing a contractual commitment." But while Knutson refused to sign an agreement obligating Seattle employees to promote PredPol, nothing precluded voluntarily participating in press events.
A flashy press conference featuring the mayor and chief of police was the result; local and national media covered the event and wrote favorable articles, including a feature on NPR's All Things Considered.
The thrust of all this media hype has been that PredPol's software works, and that it has demonstrably reduced crime in cities where it's used, and that cops have even apprehended burglars caught in the act thanks to PredPol's little red boxes.
But the case of Norcross, Ga., casts doubt on these claims. Norcross purchased PredPol's software last summer for $28,500, agreeing to "participate in media outreach with PredPol, including issuance of a press release or holding of a press conference to announce the deployment of PredPol," and even requiring Norcross to "reference the PredPol brand name, wherever possible." On Aug. 23, PredPol issued a press release claiming that, thanks to its software, Norcross police "made multiple arrests on the first day of usage — including catching a burglar in the act."
PredPol's Brantingham called it a "big first day." Press reports multiplied, using quotes attributed to Norcross' police.
SF Weekly contacted Norcross requesting any sort of objective analysis of PredPol's performance. Norcross' Chief of Police Warren Summers responded that there have been no reviews or analyses of PredPol's software conducted, adding that "the Norcross Police Department has not utilized PredPol for a sufficient period of time to fully analyze its effectiveness."
In fact, every city that SF Weekly contacted seeking independent analyses or reviews of PredPol's software has told us that no such thing exists. In response to a public records request, Alhambra sent a "retrospective analysis" showing an increased accuracy in predicting where crimes were more likely to happen using PredPol over another common method called hotspotting, or mapping reported crimes to identify locations where offenses have frequently occurred. The analysis concluded that "at the level of deployment in Alhambra, PredPol predicts 262 percent more crime correctly than hotspotting." This analysis, however, was carried out by PredPol.
Similarly, when SF Weekly requested an independent third-party analysis of PredPol's performance in Salinas, the city replied that no such verification has ever been carried out. Nor has an independent analysis been conducted in PredPol's hometown of Santa Cruz. Instead, Santa Cruz provided a "radar" report assembled by PredPol's Mohler, which according to the Santa Cruz police "can go back a maximum of 3 months to compare the accuracy/effectiveness of PredPol." In addition, the Santa Cruz police provided academic journal articles authored by PredPol's Mohler and Brantingham, both of whom have a financial stake in proving the method works.
Philip Stark, chair of the statistics department at UC Berkeley, reviewed these documents and evaluated the company's claims of reducing crime. "I'm less than convinced," he says. Stark has studied demographics, climate models, and even methods of earthquake prediction. He's also been an expert witness in lawsuits involving truth in advertising, behavioral targeting, credit risk models, and oil exploration. Stark was skeptical.
"Does using it lead to a decrease in the crime rate?" he says. "You would need to do a comparison of similar-sized cities, with similar conditions, similar trends in their crime rates, with one group of cities using predictive policing, and the others not. Then you'd compare them to each other."
"A comparison of the same jurisdiction to itself means nothing," he continues. "Crime fluctuates normally from year to year in the same city." This is, however, exactly what PredPol and the LAPD have done in claiming that predictive policing reduced crime in L.A. PredPol made the same claims in Santa Cruz, and recently in Richmond, comparing year-over-year crime rates within a single city.
"By just doing it once, last year to this year, that's like a coin toss," says Stark.
"I know the guys behind PredPol. They're pretty smart," says Jerry Ratcliffe, the chair of Temple University's department of criminology. Ratcliffe has collaborated on multiple crime prediction projects, including development of software similar to PredPol's for another company in Pennsylvania. He says, however, that predictive policing is "relatively new," and as a technology it's "not proven by a stretch."
According to Ratcliffe, predictive policing methods simply haven't been subjected to rigorous independent testing that would allow a vendor to claim its product reduced crime and caught offenders. "Testing these systems requires experimental conditions which are rarely conducted in policing and crime prevention, unfortunately." And even if PredPol or another predictive policing algorithm works, that's not enough. "A computer will never prevent crime," says Ratcliffe. "Computer outputs need to tell police where crime is likely to happen, but then police need to come up with a policy response."
Ed Schmidt, a criminologist and veteran police officer, believes in the concept of predictive policing, but he has serious reservations about PredPol's supposed effectiveness. Schmidt just completed a review of predictive policing efforts across 156 cities, and says there is little actual data that predictive policing works. Even if it does work, there's no guarantee that using it will actually reduce the overall rate of crime in a city.
"Patrol officers are assigned to sit in computer-generated boxes produced by predictive policing software..."
This is very misleading. Refer to my explanation above about how the Koper Curve Principal is applied.
"In response to a public records request for contracts between L.A. and PredPol, the LAPD says no such agreements exist."
Perhaps the LAPD is being disingenuous if not outright just lying about this. In the previously mentioned video (http://vimeo.com/50315082) Mohler describes how the 6 month study couldn't have gone on longer because it required the LAPD crime analysts to generate a hot spot map on a daily basis rather than the usual weekly report. This was, apparently, much more time consuming that it took PredPol's software to generate daily prediction boxes based on the same data.
"...'In L.A. I heard that many officers were only patrolling the red boxes, not other areas,' says Merritt."
This doesn't make sense. Using the Koper Curve a single patrol can cover 8 different prediction boxes within a shift. The Foothill Division was generating only 20 prediction boxes total. 3 separate patrols that were dedicated to only these red boxes, could handle all of them. Even if the patrols spent 16 minutes per box that would mean they could each cover 6 per shift. Doesn't a division the size of S.F. have more than 3 or 4 patrols out per shift? What that may mean is that the LAPD was not implementing predictive policing correctly.
"Crockford believes that relying on for-profit companies to deliver effective crime-fighting solutions poses serious risks. 'There's a danger in overlap of the private sector and public sector. Policing shouldn't be influenced by corporate interests that profit from Big Data and that have an obvious interest in promoting these new technologies.'"
Crockford brings up an important consideration, but it is a general consideration and I don't believe there should be any concern here. Predictive policing doesn't require a police department to, somehow, put all its eggs in one basket. It is only a tool that will influence the placement of patrol time that already must take a back seat to responding to calls. What does concern me is the price that PredPol is charging. Not only is it high but it is recurring year to year because they are using a Software As a Service (SAAS) business model. There is, as far as I can see, no intellectual property in question. The equations the algorithm is based on are public and easily implemented in software. The tie in to GIS to generate maps is now commonplace. This field is ripe for competition and that should drive the price way way down. There is no valid reason, other than profit, this has to be a SAAS application. The software can be sold as a package and run on the police department's own computers. There needn't be an ongoing contractual relationship with a private company. A final point about Crockford's comment. Predictive Policing does not use or require "Big Data". The number of data points needed per type of crime is on the order of 1200-2000. It seems that the phrase "Big Data" is in vogue in the media and is used outside of situations where it is applicable. This is one of them.
"In fact, every city that SF Weekly contacted seeking independent analyses or reviews of PredPol's software has told us that no such thing exists."
All that is needed is to make available the crime data for a city. An independent researcher can write software themselves which implements a semi-parametric self-exciting point process to see if PredPol's model has any accuracy. It would be nice if the city or PredPol provided the generated daily prediction boxes for a period of time. A metric for accuracy relative to a crime analyst's choices or a random a selection of prediction boxes is easy to get. What will take time is showing the efficacy of the program in relation to crime reduction. That will take many years and there is no way around that. A city adopting predictive policing is experimenting but they do have data relating to using hot spots in general to rely on. If there is an immediate increase in accuracy it is reasonable to believe there will be a reduction in crime at least as good as simpler hot spotting programs. By the way, independent analysis is not so important as having the raw data and algorithm available for independent review.
"In addition, the Santa Cruz police provided academic journal articles authored by PredPol's Mohler and Brantingham, both of whom have a financial stake in proving the method works."
It doesn't matter that Mohler and Brantingham are potentially biased. As long as the journal articles are peer reviewed that will reveal proper criticisms of the research or model. Those articles are, in fact, peer reviewed. I do take Philip Stark's criticisms seriously. However, you just summarized his criticism of earthquake prediction and application of a swarm model to crime. There is not much more detail than "they don't work".
"If PredPol's links to earthquake prediction are questionable, its connection to militarized studies of insurgents and civilian deaths is potentially even more troubling."
I don't see this as troubling at all. The researchers are applying a model created for earthquake swarms to a couple of different areas, including a military application. This is not a case of city police adopting military tactics. The fact that the military application was worked on first is irrelevant. As far as stripping away constitutional rights, it would be completely invalid to use presence in a prediction box as reasonable suspicion. It doesn't matter how accurate the prediction for locality of crime is, that cannot, and must not, be translated into suspicion about a particular person.
"PredPol, short for "predictive policing," is riding this wave of techno-mania and capitalizing on the belief, especially here in San Francisco, that there's a killer app for everything, including crime-fighting."
I don't think the "wave of techo-mania" has anything to do with predictive policing. Three things have converged that allow PredPol to potentially flourish as a company.
1). Hot Spot policing, another term for predictive policing has become increasingly adopted across the nation in the last few years. This has been more of a formalization of a veteran cop's intuition rather than glomming on to some new technology out of the blue. The history of predictive policing described in this very article also contradicts the idea that there is a wave of techno-mania involved.
2). Law enforcement grants coming from federal or state government agencies have become more plentiful since 9/11.
3). Recent research into the application of Bayesian Inference on earthquake swarms has led to the application of the same model(s) to predictive policing.
Additionally, many police departments have suffered budget cuts in the last few years and are desperately looking to make limited resources more effective.
"Merritt pressed Fowler about whether the program could handle violent crime. "Homicide is a priority in the department — and if it is not there it would just beg the question why not," she wrote."
Merritt doesn't understand what "beg the question" means. Hers is a common error where not addressing an issue at all is begging the question. Homicide isn't included because it is much less common than crimes such as burglary. You need a minimum number of data points before the accuracy of the model is useful. Including a longer history of crime has its own limitations because over many years neighborhoods change for other reasons that ultimately affect crime. It is also the case that homicide can be a crime of passion which doesn't lend itself as well to predicting localized repetition.
PredPol now claims to be including gun violence in its prediction model. From their website:
"Gun violence predictions are as accurate as PredPol’s proven predictions for crimes like burglaries and auto thefts — anticipating twice as many crimes as traditional hotspot mapping."
Be skeptical of this. They are just saying their software's prediction is more accurate than what a crime analyst would come up with using the same data. If the prediction of the crime analyst is lousy than their prediction is a bit less lousy. This does not state that gun violence prediction is as accurate as the prediction for burglaries, it states that the ratio of accurate instances of gun violence in hot spots to those of a crime analyst is the same ratio one sees in the comparison of the two for other crimes. The accuracy is not the same, the ratio is.
"...showcased the company and its software as if it were straight out of Philip K. Dick's short story, "The Minority Report."
I am sure the media cannot resist making this comparison bringing in the SciFi aspect. This very article has now made itself more interesting by referencing "The Minority Report". I dearly love P. K. Dick's stories, but predictive policing has nothing in common with, hot tub enslaved, precogs pinpointing potential criminals and their future crimes. The authors here, to their credit, understand this but what they fail to point out is the main reason for this skewed, scifi, understanding of predictive policing is that law enforcement hides, or deemphasizes, the use of the Koper Curve Principle in implementing predictive policing or hot spotting in general. The Koper Curve Principle states that patrols 12-16 minutes in length every 2 hours within a hot spot will reduce crime within that hot spot. You can see why law enforcement doesn't want to talk about this aspect much. They are afraid criminals are smart enough to wait around for 16 minutes before starting or continuing their criminal activities. The upshot is that additional patrols are not a stakeout within a hot spot or prediction box. Each one will consume only 10%-12% of an officers time. The number of hot spots generated by the software needs to match the currently available resources of the police department. The officer is not expected to catch a criminal in the act. The reduction in crime is due to a perceived increase in police presence deterring potential criminals from executing their chosen crimes.
This article covers some important points and I appreciate the investigative research into the aggressive marketing done by PredPol. However, it doesn't give a good overall picture of predictive policing or PredPol's technology, in particular.
"An algorithm, the exact nature of which is a proprietary secret closely guarded by PredPol..."
Although the SAAS nature of the business model engenders a protective envelope around the algorithm, it is not, in fact, secret. You can view the following video (http://vimeo.com/50315082) which is a lecture by PredPol Chief Scientist, George Mohler. He discusses the mathematics/statistics behind the algorithm and, at one point, invites the audience not to take his word for it's accuracy because he is employed by PredPol, but to take the equations discussed and plug in crime data (e.g. Chicago's open source crime data) to see if the model has any accuracy. Inadvertently, he is pointing out that implementing these published algorithms is not that difficult. This undoubtedly makes the marketing folk at PredPol rather nervous and perhaps the video will be taken down soon. Nonetheless, there are scientific papers published covering those equations and they cannot be removed.
Some crime sprees occur when hard working people cannot find adequate work, housing and food despite their best efforts. And we pass judgment on them as lazy and irresponsible, when they are trying to feed their families and survive. America has turned its back on God and the poor, while worshiping at the altar of greed, guns and ruthless corporate bullies.
“Predictive policing” is equivalent to alchemy or transforming pseudoscience into science by collecting stochastic events into a computer database application and allowing it to populate a particular set of random outcomes into a non-particular set of variable outcomes. It's snake oil for sale.
PREDICTIVE CODING / MATRIX RANKING the wave of the future. The SF Weekly's capture in cards is prescient.
Data Surfer will help decipher the data, Quintal, for 100, will arrange it into an intuitive pattern.
San Francisco is run by such an amazing group of featherbedding, overpaid incompetents. How about the overtime Sheriff's sarges asleep on the job? That's going to cost the City big time.
I read the story in your print edition this morning and I can't believe the utterly moronic comment from the ACLU person at the end.
Why do people commit robberies ? Because they are criminals. We don't need a lengthy study on this.
Private sector is enough to take of policing needs and everything else in our city, the state, the country and the world.
@red.marcy.rand Thank you for providing an accurate example of "begging the question". I can now send a tweet off to Susan Merritt pointing to your example and correcting her mistaken use of "beg the question" quoted in this article.