November 12, 2020

Public Safety Assessment Validation Studies

Validating the Public Safety Assessment

The PSA was built using historical data from around the country. It is important to test whether the PSA is accurate in the current era and in different communities. The creation of the PSA-DMF System reports can be costly, and it is important that communities spend limited resources wisely, investing in tools that have been proven to be accurate. Additionally, the validation will help identify whether the PSA accurately categorizes risk across different races and genders.

The Studies

Unlike most of the A2J Lab’s studies, these studies are not RCTs. They are validation studies. A validation study uses statistical analysis to determine how well the PSA classifies what it was designed to classify, i.e., the risk that certain outcomes will occur. In this case, the outcomes of interest are failure to appear (“FTA”), new criminal activity (“NCA”), and new violent criminal activity (“NVCA”). Validation studies assess how accurately the PSA scores classified individuals (or cases) in past cases according to risk of FTA and N(V)CA. The closeness of the relationship between the classification and what subsequently happens determines validity.

The A2J Lab is working with four counties to perform validation studies based on their use of the PSA. The counties have provided (or are currently providing) the A2J Lab with data related to the PSA and to misbehavior rates. The A2J Lab is organizing and analyzing the data to determine the PSA’s validity for various groups of people.

Outcomes

The A2J Lab is studying validity for how well the PSA classifies risk of FTA, NCA, and NVCA.

What We’ve Learned

These two reports, focused on Harris County, Texas and McLean County, Illinois, both show moderate evidence of overall validity.

Harris County, Texas: There was moderate evidence that the PSA was overall valid in Harris. Some validation techniques (e.g., simple correlations, area under the curve, balanced accuracy) provided weak evidence of validity, others (e.g., simple plots, logistic regression) provided strong evidence of validity. No technique suggested invalidity. There was no substantial evidence to suggest that the PSA scales performed differently for different racial and gender groups.

McLean County, Illinois: There was moderate evidence for the overall validity of the PSA scales in McLean. Most validation techniques (simple plots, logistic regression, balanced accuracy measures, and area under the curve for NCA and FTA) provided either moderate or strong evidence for validity, while the remainder (correlations, area under the curve for NVCA) provided weak evidence of validity. No technique suggested invalidity. Analyses by gender and race were inconclusive.

The two additional reports will post when available.

The Research Team

Jim Greiner, Faculty Director, Access to Justice Lab; Professor of Law, Harvard Law School

Matthew Stubenberg, Associate Director of Legal Technology, Access to Justice Lab

Ryan Halen, Data Analyst, Access to Justice Lab