Predictive Policing with Big Data

Police Departments nationwide have been using data and statistics to drive policing since the 90s in an approach founded by the NYPD named CompStat was credited with dramatic reductions in crime and increases in efficiency. CompStat, a process and philosophy rather than a single technology or software, uses databases and GIS to record and track criminal and police activity and identify areas that are lagging or need more attention. While it provides much more information than “primal policing”, CompStat has advanced little beyond simple spreadsheets and mapping software. Inspired by recent innovations in Big Data and Apache Hadoop and businesses like Walmart or Amazon using analytics to determine future demand, departments across the country and worldwide are looking to take this approach to the next level and go from tracking crime to predicting it.

The first department to adopt this strategy was Santa Cruz through their city-wide 6 month Predictive Policing Experiment, named one to Time Magazine’s 50 best innovations of 2011. The large scope of the experiment, the statistically average crime rate and the challenges faced by the department make is a good example for other cities. Like most police departments right now, SCPD has a declining budget and shrinking police force. On top of that, in the first 6 months of 2011 it saw an unprecedented crime wave. Driven to do more with less, the department signed on to work with researchers at UCLA to test a new method of modelling crime.

UCLA mathematician George Mohler noticed that, over time, crime maps resemble other natural phenomenon and modified algorithms used to predict aftershocks to instead predict future property crimes from past data. Using seismologists’  models for crime isn’t as crazy as it sounds, since they’ve already been adopted in epidemiology and finance. Mohler’s approach is supported by popular modern theories on crime, the rational choice model, which states that criminals, like consumers, pick their targets rationally based on value, cost, effort, and risk, and the Broken Window theory, popularized by the same NYPD Commisioner who implemented CompStat, Bill Bratton. Though the Broken Window theory is typically applied to vandalism, the essence is that petty crime leads to major crime and that crime is self-reinforcing by setting norms and making areas seem poorly controlled. Past crimes can be predictive of future crimes because they indicate that an environment is target-rich, convenient to access for a criminal, vulnerable, or simply seems like a good place to strike due to a pattern of crime and poor control.

Mohler’s algorithm is different from the CompStat approach, which simply identifies “hot spots” where crime is clustered. To predict the most likely type, location,  and time of future crimes, Mohler must compare each past crime to the others and generate a massive amount of metadata. For the Santa Cruz Experiment, he went back to 2006, looking at roughly 5,000 crimes requiring 5,000! or 5,000 x 4,999 x 4,998… comparisons. When he compared his method to traditional CompStat maps for the LAPD’s archives, he found that it predicted 20 to 95 percent more crimes.

The experiment was recently concluded, and the department believes that its predictive policing program was a success. Despite having fewer officers on the force, SCPD reversed the crime wave and lowered crime by 11% from the first half of the year to 4% below historical averages for those months. Still, from that information alone, it’s difficult to tell how effective Mohler’s strategy was, and we will have a better indication when the LAPD concludes a similar but even larger study in May, that includes a control group.

Elsewhere, other departments in the United States and abroad are adopting a Big Data approach to policing as well. Sponsored by the Bureau of Justice Assistance , the Smart Policing Initiative is exploring predictive policing in over a dozen departments and agencies nationwide, including Boston PD, Las Vegas PD, and the Michigan State Police. Some have already yielded results, such as in Richmond, where software using a combination of business intelligence, predictive analysis, data mining, and GIS has contributed to a drastic drop in crime.  Predictive policing is also being tried in the UK where, in the single ward of the Greater Manchester area studied, burglary decreased by 26% versus 9% city-wide, prompting follow-up studies in Birmingham.

While predictive policing is showing promise and, in limited trials, results, the practice is still in its infancy with plenty of room to grow. Much more metadata can be generated and factors included into the predictive algorithms. For example, Santa Cruz could only predict property crime, as violent crime depends less on targets and opportunities and more on events and interpersonal interactions.  In business and counter-terrorism, however, tools like social network analysis and social media monitoring have been used successfully to get a better feel for social dynamics.  As predictive policing gets more attention and is adopted more widely, we can expect to see these and other Big Data solutions applied to law enforcement.

CTOvision Pro Special Technology Assessments

We produce special technology reviews continuously updated for CTOvision Pro members. Categories we cover include:

  • Analytical Tools - With a special focus on technologies that can make dramatic positive improvements for enterprise analysts.
  • Big Data - We cover the technologies that help organizations deal with massive quantities of data.
  • Cloud Computing - We curate information on the technologies enabling enterprise use of the cloud.
  • Communications - Advances in communications are revolutionizing how data gets moved.
  • GreenIT - A great and virtuous reason to modernize!
  • Infrastructure  - Modernizing Infrastructure can have dramatic benefits on functionality while reducing operating costs.
  • Mobile - This revolution is empowering the workforce in ways few of us ever dreamed of.
  • Security  -  There are real needs for enhancements to security systems.
  • Visualization  - Connecting computers with humans.
  • Hot Technologies - Firms we believe warrant special attention.




  1. [...] The numbers work for them: officers have reversed the crime wave and lowered crime by 11% from the first half of the year to 4% below historical averages for those months. The full article is here: Predictive Policing with Big Data [...]

  2. [...] The Domain Awareness System gives the NYPD a number of new, powerful capabilities. For example, if a suspicious package is reported, the department can find video footage from relevant cameras then rewind to see who left it there. More information can also now be queried and correlated across time and space to reveal patterns and trends. Investigators can also use license plate readers to track down a suspect’s vehicle, discover where it has been, and monitor its movements. While these and countless other powers will make investigations, criminal analysis, and resource planning easier, civil libertarians are already uneasy with surveillance reminiscent of  ”Big Brother.”  The NYPD assures privacy groups that facial recognition will not be used, that only information currently being collected separately will be integrated into the Domain Awareness System, and that only public areas will be monitored. The Domain Awareness System, unlike similar Big Data solutions in cities like Los Angeles, also does not appear to have a predictive component. [...]

  3. [...] Big data puts officers in between likely targets and bad guys by enabling predictive policing. Police can use public data to predict the location and time that crimes are likely to occur, so [...]

  4. [...] Predictive policing – find hyper-local areas where crime is likely to occur [...]

  5. [...] is a great example of how Big Data can be leveraged for public safety. As with predictive policing, StreetCred helps officers do more and better with less time and money, which is critical now with [...]