Chris Griffin

Is the Prosecutorial Black Box Finally Opening?

March 12, 2018 by Chris Griffin

It’s not every day the headlines announce government officials releasing mounds of data about their own operations. Actually, the frequency verges on the unprecedented. Cook County State Attorney Kim Foxx’s decision to release information on years’ worth of her office’s decision-making is, in that light, astounding.

Legal scholars have lamented the existence of so-called “black boxes” within the criminal justice system. They arise whenever various dimensions of legal decision-making are unobservable to outsiders, whenever the reason for a decision is left unexplained. Whether it’s the jury or the prosecutor, the public rarely (if ever) has access to systematic information on their motivations and priorities. We can observe the outputs—who is charged, who receives a plea deal. But we only observe those outputs when the court official proceeds with the case. Without access to internal data, we cannot parse the determinants of, for example, different declination rates by race and crime type. For elected prosecutors, this information gap is thrown into sharp relief. The voting public is left to rely on their gut instincts and high-profile convictions when assessing an office’s track record in promoting public safety.

Data transparency is also relevant to the Access to Justice Lab’s work. Our expanding studies of pretrial release decision-making, although primarily focused on judicial discretion, can benefit from prosecutors’ data. Here’s how. The PSA’s developers define new criminal activity—a key outcome variable in our analysis—as any arrest for new charges. We can use the court’s database to locate all arrests that lead to filed charges. But what of the arrests that do not? We could merge the court data with jail booking information, but that comes at the cost of having to un-duplicate the dataset when a jail booking also leads to formal charges. The superior source would be the prosecutor’s own database; it contains information on all of the in-custody arrests and non-custodial citations regardless of eventual charge activity. It’s effectively a one-stop shop for identifying any new arrest following pretrial release.

Time will tell if Ms. Foxx’s bold action motivates other jurisdictions to follow her lead and facilitate evaluation of this relatively opaque player in the criminal justice system.

Previewing and Reviewing Pretrial Risk Assessment RCTs

January 18, 2018 by Chris Griffin

On Tuesday, Jan. 16 pretrial staff in Polk County, Iowa entered their offices with a slightly different charge. They had been accustomed to perusing a list of arrestees scheduled for first appearance and searching for individuals who qualified for an interview and pre-disposition release. That morning, some staff members continued this time- and resource-intensive practice. Others reviewed administrative records and entered nine risk factors into a new software system that calculates (hopefully familiar to readers of this blog) PSA risk scores. Polk County is the first jurisdiction in Iowa to implement the PSA. Three more counties will join them in the coming months as pilot sites, and eventually the entire state will adopt it.

As the A2J Lab looks ahead to launching its second RCT evaluation of the PSA, we came across a study of its progenitor, the Virginia Pretrial Risk Assessment Instrument (“VPRAI”). When the VPRAI arrived in courtrooms around the state, there was no way to convert risk predictions into actionable release recommendations. (That fact stands in stark contrast to the Decision-Making Framework accompanying the PSA.) The solution was the Praxis, “a decision grid that uses the VPRAI risk level and the charge category to determine the appropriate release type and level of supervision.” Virginia pretrial staff also embraced the so-called Strategies for Effective Pretrial Supervision (“STEPS”) program to “shift the focus . . . from conditions compliance to criminogenic needs and eliciting prosocial behavior.” The combination of these innovations, it seemed, would improve Virginia’s ability to pinpoint risk and reduce failure rates during the pre-disposition period.

Marie VanNostrand of Luminosity and two co-authors were interested in understanding, first, the VPRAI’s predictive value. Second, they assessed the benefits of the Praxis and STEPS program through a randomized study design. Unlike the A2J Lab’s field experiments, which usually take individuals as the units of randomization, the Virginia study randomized entire pretrial services offices to one of four conditions: (1) VPRAI only; (2) VPRAI + Praxis; (3) VPRAI + STEPS; and (4) VPRAI + Praxis + STEPS. The authors then used this exogenous (nerd speak for “completely external”) source of variation to analyze staff, judicial, and defendant responses.

The results were quite favorable for the introduction of the Praxis as well as for the VPRAI itself. One estimate suggested that higher VPRAI risk scores correlate strongly with higher actual risk. About two-thirds of the time, if one were to pick two defendants at random–one who failed and one who didn’t–the one who failed would have a higher VPRAI score. Pretrial services staff who had access to the Praxis also responded to its recommendations. Their concurrence (agreement) rate was 80%, and they were over twice as likely to recommend release relative to staff who did not have the decision grid. Next, the availability of the Praxis (versus not having it) was associated with a doubling of the likelihood that judges would release defendants before disposition.

What about defendant outcomes? The authors found that the availability of the Praxis was associated with a lower likelihood of failing to appear or being arrested for a new crime. STEPS alone had no discernible effect.

The VPRAI study suggests a few lessons for our ongoing pretrial risk assessment work, including in Iowa. First, we continue to emphasize that the tool under investigation, the PSA, is far from a cold, lawless automaton, as many commentators seem to worry. Yes, algorithms produce scores, and decision matrices generate recommendations. But human beings must still consider that evidence alongside their own human judgment. One hope is that such evidence will enhance the quality of judges’ decision-making. For now, we just don’t know; that’s the reason for our PSA RCTs. Relatedly, we think that final verdicts on actuarial risk assessments should await reports like the VPRAI study and the A2J Lab’s growing portfolio of evaluations. There will always be local policy issues deserving of debate and attention. However, we need strong evidence for or against these tools’ value before praising or condemning them wholesale. Finally we should, as always, evaluate this brave new world reliably. That means deploying, where possible, principles of experimental design. RCTs, simply put, represent our best shot at understanding causal relationships.

Stay tuned for more updates from Iowa and beyond!

Guest Post: Acesso à Justiça: O Grande Desafio / Access to Justice: The Great Challenge

June 8, 2017 by Chris Griffin

The A2J Lab receives many comments, inquiries, and questions via the website. Most come from U.S. attorneys and researchers. From time to time, though, we are lucky to hear from others around the world committed to making their court systems more open and their legal procedures more transparent. One such kindred spirit is Jéssica Raiane, an attorney living in Goianésia, Goias, Brazil. She kindly has shared her thoughts on the challenges facing A2J proponents in her home country. The original Portuguese post is followed by an English translation.

A2J Lab “Behind the Experiment”: Dane County Part III

May 2, 2017 by Chris Griffin

Today we present a final look at key field partners who have helped make the Dane County PSA RCT one of the A2J Lab’s signature series. Check out Part I and Part II as well!

In this installment, we are fortunate to share reflections from three officers of the court: Judge Juan Colás, Commissioner Jason Hanson, and Dane County District Attorney Ismael Ozanne.

A2J Lab “Behind the Experiment”: Dane County Part II

April 20, 2017 by Chris Griffin

Today we bring you another look at key field partners who have helped make the Dane County PSA RCT one of the A2J Lab’s signature series. Part I covered the work of the County’s two Assessors, who perform the essential work of populating the PSA’s risk factors and generating its recommendations day in and day out.