A2J Lab presents an interim report on the Public Safety Assessment–Decision Making Framework System RCT in Dane County, Wisconsin

Today, the A2J Lab presented the findings from the interim report of the randomized evaluation of the Public Safety Assessment–Decision Making Framework System (PSA-DMF), a pretrial risk assessment tool and related decision-making framework, in Dane County, Wisconsin. The full interim report is now available on the A2J Lab website.

The presentation at the Dane County Criminal Justice Council was the first result of a years-long study, which is not yet complete. In the late spring of 2017, Dane County started providing the PSA-DMF System information to judicial officers deciding how much and what kind of bail and supervision to assign to individuals who have been arrested; such decisions affect whether the individual will be released or remain in jail until trial. Working with the A2J Lab and Arnold Ventures, Dane County’s randomized field experiment began a month later.

The PSA-DMF System is one tool in the toolbox that the judicial officer can draw upon in the exercise of their professional discretion. Scientists supported by Arnold Ventures produced the PSA by reviewing past data on criminal history, demographics, new crimes, rates at which participants failed to appear at their court hearing, and other potential risk factors. The PSA scores are applied to the Decision Making Framework, which brings together the information from the PSA with a community’s local policies and values, its laws, and its resources to provide a recommendation regarding pretrial release and, if release is obtained, a supervision level. Judicial officers may use the PSA-DMF System report when deciding whether to release an individual before trial, and this decision rests always with the judicial officer.

The experiment used something like a coin flip to divide cases into two groups. In the treated group, the judicial officer who was deciding how much and what kind of bail and pretrial supervision to assign received a paper printout with the PSA-DMF System information on it. In the control group, the judicial officer did not receive the paper printout. In other words, the control group received standard practice, and the treated group received the new system in which the judicial officer got the PSA-DMF System printout.

The report released this week analyzes data from one year of follow-up for cases that were included in the study between the start of the study and the middle of 2018. Dane County randomized cases from the middle of 2017 until the end of 2019. In this interim report, the A2J Lab analyzed data the County provided to compare the treated and control groups on (among other things) measurements of racial fairness, number of days incarcerated (if any), rates at which participants failed to appear at their court hearing, new criminal activity, and new violent criminal activity. Criminal justice officials in Dane County worked to provide information to the A2J Lab to facilitate the A2J Lab’s independent evaluation in a spirit of learning and a desire to improve.

Quantitatively, the A2J Lab does not yet have enough cases, or enough of a follow-up period on those cases, to make firm conclusions about whether it is better or worse to make the PSA-DMF System printout available to the judicial officer before assignment of bail (if any) and release conditions. The A2J Lab’s studies, like all randomized control trials, require a number of participants sufficient to detect policy-relevant differences between treatment groups (here, the PSA-DMF System report versus business as usual). The number of participants must be sufficient to analyze for statistical significance. Because of the two-year follow-up period, the County will provide the A2J Lab full information on all arrestees in the study sometime in early 2022. If the data are provided then, the A2J Lab’s final report will be available in the summer of 2022.

The limited data available thus far, not enough to draw firm conclusions, suggested several findings:

  • There is some evidence that providing the PSA-DMF System printout to the judicial officer caused a change in the officer’s decisions.
  • Generally, when the printout indicated that an individual presented lower risk, the judicial officer was less likely to require cash bail or, if cash bail was required, the amount was lower than in comparable control group cases. The opposite was generally true in the treated group when the printout indicated that the individual presented higher risk, as compared to the control group. This change was statistically significant but mild, and we cannot yet tell whether the change is policy-relevant.
  • Treated group cases varied less in bail types and amounts than did control group cases. This change was strong and statistically significant.

As of the time of the interim report, there was no statistically significant difference between treated versus control group cases with respect to:

  • various measures of the racial fairness of the judicial officer’s decisions;
  • the number of days (if any) of pretrial incarceration;
  • the frequency with which arrestees failed to appear at court dates; or
  • the frequency with which arrestees were arrested for new crimes, including new violent crimes, during the pretrial period.

These finding might change, however, when the A2J Lab finishes analysis of the final dataset.

While it is too early to draw conclusions about whether the PSA-DMF System is positive, negative, or neutral for Dane County, this data provides a first look at the type of report that the final dataset will support.

Leave a Reply

Your email address will not be published. Required fields are marked *