• Skip to primary navigation
  • Skip to main content
Access to Justice Lab logo

The Access to Justice Lab

at Harvard Law School

  • About
  • Projects
  • Publications
  • Resources
  • People
    • A2J Lab Staff
    • Advisory Board
    • Collaborators
  • Contact
  • Donate
  • Show Search
Hide Search

Blog

How to create safer policies for algorithmic assessments: A Study of Pre-Trial Risk Assessment Instruments

April 23, 2023 by Alice Hu

Overview

In recent years, algorithmic-driven policies and recommendations have become a ubiquitous and integral part of our society, from online shopping to job screening. Driven in part by this transformative change, the academic literature on optimal policy learning has flourished. The increasing availability of granular data about individuals at scale means that the use of these new methodologies will only continue to grow.

This investigatory study focuses on safe policy learning for pre-trial risk assessments. The study defines policies as “safe” if they do not lead to worse outcomes than the status quo on average. Pre-trial risk assessment instruments (RAIs) may help judges decide whether to release with different levels of conditions or detain an arrested individual before trial by providing simple scores classifying the risk of their failing to appear at court or committing a crime. Among other reasons, because arrestees are presumed innocent, it is important to avoid unnecessary incarceration. Pre-trial RAIs have become increasingly prominent in criminal proceedings, used in at least 23 states and affecting the lives of thousands every day. 

The study proposes that existing studies of RAIs have been asking the wrong question: they focus on how RAIs have impacted the rate of failure to appear or pre-trial crime, when they should investigate how RAIs improve or worsen judges’ decision-making. Unlike a pill, the intervention – the RAI in this case – is not given to the person whose behavior it is meant to change, but rather a third party, the judge. Thus, the effectiveness of the intervention must be measured by changes in judicial decisions. But to evaluate such changes in a randomized controlled trial would require randomizing RAI scores given to judges and measuring their responses, a prohibited act as it entails deliberately providing judges wrong information. 

How can we learn and create better policies on algorithmic assessment? Unfortunately, prior methods for policy learning are not applicable because they require existing policies to be “stochastic”  (meaning they vary randomly) rather than deterministic (meaning each set of inputs produces only one output value). This study develops a robust optimization approach that partially identifies the expected utility of a policy, and then finds an optimal policy by minimizing the “worst-case regret” (meaning the lowest possible utility). The resulting policy is conservative but has a statistical safety guarantee, allowing the policy-maker to limit the probability of producing a worse outcome than the existing policy. The researchers extend this approach to common and important settings where humans make decisions with the aid of algorithmic recommendations, such as judges using RAIs. We derive new classification and recommendation rules that retain the transparency and interpretability of the existing instrument while potentially leading to better overall outcomes at a lower cost.

Pre-trial Risk Assessment

The study’s methodology is motivated by a popular pre-trial risk assessment instrument called the Public Safety Assessment (PSA), used in Dane County, Wisconsin, along with many other places.

The PSA consists of classification scores based on the risk that each arrestee will engage in three types of risky behavior: (i) failing to appear in court (FTA), (ii) committing a new criminal activity (NCA), and (iii) committing a new violent criminal activity (NVCA). Judges abalance between these risks and the cost of incarceration when making their pre-trial release decisions.

The PSA consists of separate scores for FTA, NCA, and NVCA risks, based on 9 risk factors. Importantly, the only demographic factor used is the age of an arrestee, and other characteristics such as gender and race are not used. The other risk factors include the current offense and pending charges as well as criminal history, which is based on prior convictions and prior FTA. Each of these scores is constructed by taking a linear combination of underlying risk factors and thresholding the integer-weighted sum. For the sake of transparency, policy makers have made these weights and thresholds publicly available (see https://advancingpretrial.org/psa/factors).

Developing a Safe Policy 

The study’s primary goal is to construct new algorithmic scoring and recommendation rules that could lead to a higher overall expected utility than the status quo rules, while retaining a high level of transparency, interpretability, and robustness. While there are many factors besides the risk assessment instruments that affect the judge’s decision and the arrestee’s behavior, the study focuses on changing the existing algorithms rather than the other factors.

The researchers develop their optimal safe policy approach in two parts. To construct a safe policy in the population, i.e., with an infinite number of samples, they analyze the population optimization problem that constructs a safe policy. Then, they give concrete examples to build intuition before describing our methodology in greater generality. Finally, they develop several theoretical properties of our approach. 

Because we do not have access to an infinite amount of data, we cannot compute the population safe policy. Instead, the researchers show how to learn an empirical safe policy from observed data of finite sample size.

Applying the Safe Policy to Pre-trial Risk Assessment

The study applied this optimization methodology to the PSA. Given the highly technical nature of the findings, they will not be summarized in detail here. However, they are relevant for policy makers and technologists as they suggest an opportunity for improving the existing scoring rules.

Future Research

There are several avenues for future research. First, on implementation choices under the proposed approach: while we consider several representative cases, there are many other structural assumptions that would lead to different forms of extrapolation. 

Second, we can use similar statistical tools to create tests of safety for given policies. By creating a worst-case upper bound on the regret of a policy relative to the status quo, we can test for whether the proposed new policy is an improvement over the existing status quo.

Third, there are many ways in which optimal algorithmic recommendations may differ when considering long term societal outcomes rather than short term ones. For example, pre-trial detention driven by a risk assessment recommendation may alter the long term behavior and welfare of an arrestee. Understanding how to design algorithms when they affect long-term future outcomes is key to ensuring that recommendations do not take a myopic view. One potential way to incorporate long term outcomes may be with the use of surrogate measures. 

Finally, within the robust optimization framework, the notion of “safety” can be considerably expanded. In this paper, we consider policies to be safe if they do not lead to worse outcomes on average; however, this does not guarantee that outcomes are not worse for subgroups. 

Is obtaining expungement of criminal records simple?: An Empirical Study of Kansas

April 9, 2023 by Alice Hu

 

Overview

Over seventy-eight million individuals in the United States – one-third of the country’s adult population – have criminal arrest records. In Kansas, the incarceration rate increased 215% over the past three decades, with Black people overrepresented in both Kansas jails and prisons.

Having an arrest or conviction record may negatively impact one’s ability to participate in society, compromising educational, employment, and housing opportunities. In Kansas, certain convictions disqualify an individual from obtaining employment in the five years following the completion of their sentences in various industries, such as adult care homes, home health agencies, and hospitals.

To address the barriers that individuals with criminal records face, most states – including Kansas – have procedures for expungement, in which a charged or convicted individual petitions the government to clear their record. Some attorneys in Kansas have expressed that the process in Kansas for obtaining an expungement is simple and straightforward. Thus, they believe, even assuming criminal justice record clearing promotes desirable outcomes, Kansas need not pursue broadly applicable, automated record clearing consistent with the model in, for instance, Pennsylvania.  These sentiments have surfaced in discussions with legal services providers regarding whether self-help materials and procedural reforms for expungement might be useful in Kansas.

This investigatory study assesses the claim of the high accessibility of expungement in Kansas, as part of understanding the broader landscape of expungement. It found that obtaining expungement in Kansas is not simple or straightforward. Our findings are especially relevant for state legislatures, policymakers, and other stakeholders interested in record-clearing remedies.

Steps to Clear a Criminal Record in Kansas

In Kansas, arrest records, criminal convictions, and juvenile adjudications can all be expunged. As the process is similar for all types of expungements, this study focuses on the expungement of non-juvenile convictions. An individual obtains an expungement by petitioning the court of the county in which they were convicted. Some convictions are never eligible for expungement, including murder and child abuse. All expungement candidates must meet certain criteria and follow specific procedures, outlined below.

  • Mandatory waiting period. In Kansas, the waiting period for petitioning expungement of a misdemeanor or minor felony is three years after discharge from probation or parole. The waiting period is five years for major felony offenses. In addition, an individual must not have been convicted of any felony in the two years preceding petitioning for expungement.
  • Forms. A central component of obtaining an expungement is the completion of six forms: (1) Criminal Cover Sheet; (2) Petition for Expungement of Conviction, (3) Notice of Hearing, (4) Order of Expungement of Conviction Cover Sheet; (5) Order for Expungement of Conviction; and (6) Order Denying Expungement of Conviction. These forms are available online at the Kansas Judicial Council website or in-person at the reception desk of the District Attorney’s Office of the relevant jurisdiction. The petitioner must file a separate set of forms for each offense.
  • Submission and fees. After accurately completing the requisite forms, the petitioner must file with the court clerk their Petition for Expungement and Criminal Cover Sheet, discussed above. As of the time of this study, the petitioner must pay a docket fee of $195 via money order. If a petitioner cannot afford the $195 docket fee, they may submit a Poverty Affidavit with the required detailed information.
  • Hearing.  In some jurisdictions, courts may grant expungements without a hearing, particularly if no government agency objects, but likely only a lawyer would know that a hearing could be unnecessary. Under Kansas law as written, the petitioner must contact the clerk of the court to request an expungement hearing and attend the hearing. At the hearing, the court will inquire into the background of the petitioner and may have access to any reports or records on file with the prisoner review board. 
  • Decision. The presiding judge orders the expungement of a conviction if (1) the petitioner has not been convicted of a felony in the past two years and no proceeding involving any such crime is presently pending against the petitioner; (2) the circumstances and behavior of the petitioner warrant the expungement; and (3) the expungement is consistent with the public welfare. 

The clerk sends a certified copy of the order of expungement to the KBI. Once the certified order of expungement is received, it takes about two to four weeks for the KBI to process the expungement and update the individual’s criminal record history. The KBI will notify the FBI, the Kansas Secretary of Corrections, and any Kansas law enforcement agencies that were involved with the individual’s conviction. 

  • Impact. After obtaining an expungement in Kansas, a person “shall be treated as not having been arrested, convicted or diverted of the crime.” However, an expunged conviction may be considered for sentencing purposes if the individual is convicted of a subsequent crime. In addition, disclosure is required in applications for admission to the Kansas Bar, employment as a law enforcement officer, or a license to carry a concealed weapon.

Key Takeaways

Our findings are relevant for state legislatures and policy stakeholders interested in expungement. The process to obtain an expungement in Kansas is not straightforward or simple, as some attorneys in the state have claimed. As outlined above, the procedures consist of eleven steps, one with five subcomponents. If lowering the barriers to expungement is desirable under some circumstances, one key change may be reducing the number and complexity of steps.

Other improvements would also make the expungement process more accessible. With regard to forms, for example, the six forms could be rewritten in plain English language. Kanas could remove requirements to provide information that is not actually required, is difficult or simply impossible to find, and/or is already in the court’s possession. Kansas could also remove the requirement that a petitioner file a separate set of forms for each conviction. Finally, Kansas could remove requirements that exist only to reduce the court’s workload, shifting the burden from the petitioner to the court to complete the court’s administrative tasks. Short of automating this process, which may help remove barriers for both petitioners and the courts, any and all of these changes would be a step in the direction of accessibility in expungement.

What would lower the barriers to record clearing?: An Empirical Study of Two U.S. States

March 28, 2023 by Alice Hu

Overview

Seventy-four million people in the United States have a criminal record — mostly for arrests that did not lead to a conviction or for misdemeanor convictions. By age 23, half of Black and Latino men, more than one-third of white men, and one in eight women have been arrested. Those with criminal records face collateral consequences that may affect many aspects of their lives, from employment and housing to immigration. Unlike some countries, U.S. jurisdictions generally allow public access to criminal justice information, meaning both even private actors like employers and landlords can make decisions based on one’s records.

Recording clearing has become one way to attempt to lessen the burdens faced by those who have been arrested or convicted. “Expungement” is petition-based record clearing, while “sealing” refers to automatic record clearing without needing the individual with the record to initiate the clearing. However, record clearing (particularly expungement) is challenging in the U.S. due to factors such as racial marginalization and lack of access to justice.

While there are many arguments both supporting and opposing record clearing, this study focuses on an empirical question that has never been studied: If states desire to lower the barriers to record clearing, what would be the most effective legislative changes to do so?

We address this question by analyzing decades of criminal records in Pennsylvania and Kansas. We analyze each state separately, examining what could happen to record clearing in each state given certain amendments to the state’s record clearing laws. Our findings are particularly relevant for legislatures interested in implementing or expanding record clearing remedies. 

Pennsylvania Study

Pennsylvania offers record clearing mostly through the state’s Clean Slate Act, which passed in 2018 and took effect mid-2019. The Act expanded expungement eligibility as well as sealing for non-violent offenders, resulting in over thirty million sealed records in the year following its passage.

We obtained data from four Pennsylvania counties: Allegheny, Beaver, Butler and Lawrence. These counties were chosen because we were pursuing separate research into criminal justice record clearing in the same area. We scraped criminal records from the judicial website for the Court of Common Pleas, the trial court, for cases adjudicated from 1990 to early 2021. 

Results 

In Pennsylvania, we found some surprising results: 

  • Legally imposed financial obligations (LIFOs) are critical barriers. Our most important finding is that nominal fees, fines, and restitution are significant barriers to record clearing. There appeared to be few, if any, ability-to-pay exemptions from these barriers. To the extent that one believes that record clearing is a useful policy and that non-payment partly stems from inability to pay, our findings illuminate disturbing echoes of poverty-based punishment.
  • Look-back periods may be unnecessarily lengthy. Eligibility criteria for record clearing include lengthy look-back/waiting periods, despite some evidence suggesting recidivism rates become close to those with no criminal records after surprisingly short amounts of time free of criminal justice involvement. (This idea remains an open question, so we proceed with caution on this point.) The result is that individuals who may be statistically indistinguishable from those with no criminal records waited years to receive record clearing remedies. 
  • Difficulty in implementing sealing results in eligible records remaining visible. As of the time we scraped the data from publicly available judicial sources the Court had been sealing records for more than two years. Yet, a startling number of records resisted clearing, likely because the information needed to determine definitively their eligibility for clearing did not exist in practicable electronic form. The information missing here was of the most basic kind, such as the dispositions of criminal cases.
  • Difficulty in implementing automatic clearing results in individuals resorting to petitions. Accommodating nuanced legislation likely induced errors in implementing automatic record clearing, forcing a number of otherwise eligible individuals to resort to the petition-based process that the automatic processes were designed to supersede.

Kansas Study

Unlike Pennsylvania, Kansas had no sealing statute, so expungement was the only remedy for those seeking record clearing. Kansas law did allow for the expungement of most convictions. 

We obtained data from 29 localities in Kansas. During 2020 and 2021, we scraped criminal case records with filing dates from 1990 and forward from the Kansas statewide court look-up portal and from the Johnson County case search website. Counties outside of Johnson (the most populous country) allowed cases to be searched through a separate site but for a fee for each record, which proved costly. As a result, our effort produced records on 254,622 cases in Kansas, with Johnson County’s website contributing 94,074.

Results

Our analysis in Kansas suggests two results, the first of which is consistent with prior research: 

  • Lack of access to justice leads to a major second-chance gap. There are a large number of seemingly eligible individuals who have not yet requested petition-based record clearing. Such a high number of eligible but uncleared convictions likely stems from a distressingly familiar access to justice problem.
  • Data issues in Kansas’s system make accurate analysis impossible. Structural holes and wholesale irregularities in the data that were available make a complete analysis impossible. Unless there are sources of information easily available to the court but withheld from the public, it may be impossible to test hypothetical record clearing legislation without addressing these massive data issues. 

Conclusion

Our findings are particularly relevant for legislatures interested in implementing or expanding record clearing remedies. Most notably, our findings suggest that two of the most effective changes for lowering barriers would be removing eligibility criteria based on outstanding legally imposed financial obligations (LIFOs) and reducing the length of waiting or lookback periods. We have also demonstrated severe problems with criminal justice data, problems that do and will affect any effort to implement or expand record-clearing. Our findings also reinforce results that other researchers have reported: the second-chance gap is staggering. If record clearing is a wise policy, there is much work to be done. 

Does Online Dispute Resolution Increase Access to Justice?: Lessons for Courts

March 8, 2023 by Alice Hu

The Covid-19 pandemic forced courts to change overnight the way they heard cases, from requiring masks in courtrooms to holding hearings online or via the telephone. Online dispute resolution (ODR) systems became a popular method for courts to dispense justice. Proponents have long argued that ODR increases access to justice, mitigates procedural and substantive errors, and conserves court resources—particularly for self-represented litigants. Is this true? The A2J Lab conducted two randomized control trials (RCTs) to assess newly implemented ODR programs. This post discusses those RCTs and one useful and unexpected finding.

Image credit: Hallie Jay Pope (Graphic Advocacy Project, https://www.graphicadvocacy.org/)

What is ODR and Current Landscape

ODR has seen a rise in popularity even before the pandemic. In 2019, the ABA reported that sixty-six courts in twelve states maintained ODR platforms for settling civil and criminal disputes. Between 2018 and 2019, the number of court-annexed ODR platforms more than doubled. 

ODR platforms are often classified into two types: instrumental ODR systems offer a virtual platform for facilitating otherwise face-to-face dispute resolution, while principal ODR systems take a “proactive role facilitating the resolution of the dispute.” Courts may implement ODR as a mandatory pretrial process or as an optional alternative to trial. ODR can occur synchronously in real-time videoconferences or asynchronously in chat rooms. ODR systems may operate differently in disputes involving an individual against the state versus disputes between private individuals, and their usage may be free or cost money.

Two vendors, Matterhorn and Modria (the latter recently purchased by Tyler Technologies), currently dominate commercial ODR systems in the United States. 

Purported Benefits of ODR

Increasing Access to Justice

Advocates argue that ODR allows parties who lack the time and money to attend an in-person hearing a cheaper and more accessible online mediation process. If individuals can negotiate small claims or contest citations over an asynchronous, online platform, they may be less likely to default. 

However, barriers include a lack of willingness to use ODR as well as digital divides. Low-income and disadvantaged communities are more likely to lack the necessary broadband connection for video conferencing.

Mitigating Procedural and Substantive Mistakes

Proponents argue that ODR reduces the risk of procedural and substantive errors on the part of participants. Pro se (self-represented) litigants face substantial obstacles in court to follow the correct procedures and adequately present their case. ODR’s informality and physical separation might allow self-represented litigants to feel less nervous and mitigate biases from face-to-face hearings. 

However, some researchers argue that parties who suffer disadvantages in traditional hearings or alternative dispute resolutions may suffer similar disadvantages in ODR. Further, efficiency gains may hinder equity goals. For instance, more efficient resolution of eviction suits by landlords against tenants may exacerbate landlords’ comparative power over tenants. 

Conserving Court Resources

Advocates argue that ODR may reduce resource demands on the court. By using ODR, courts might avoid the costs of physical spaces and staff needed for face-to-face hearings. ODR may also reduce the administrative costs of procedural errors by self-represented litigants, such as failure to file documents on time or appear in court. 

There has been little testing of all the popularly cited benefits of ODR, and further research is needed.

Our Randomized Controlled Trials (RCTs)

There are currently no RCTs evaluating whether ODR serves the equity, access to justice, or efficiency goals raised by proponents. RCTs are considered the “gold standard” for empirical testing because randomization creates statistically equivalent groups but for the tested intervention, minimizing the possibility that observed differences are due to chance. Unlike studies that compare pre-intervention with post-intervention outcomes, RCTs do not create a risk that another variable influenced experiment outcomes.

Design

The most straightforward way to assess the said benefits of any ODR program is to randomize its usage. The randomization creates two identical groups with a single variable—one of which uses ODR and other of which does not; as a result, any observed differences in access to justice, budgetary, or other outcomes can be attributed to the ODR. Depending on the court’s usage of ODR, the study might randomize either access to the platform (if usage is optional) or orders compelling litigants to use it (if usage is mandatory). Either design, if implemented properly, would produce credible results.

However, the two courts (in different jurisdictions) in our studies declined to implement either design. Instead, in Court A, all individuals who received traffic citations received information about the ODR platform from the citation form. Users who registered for the ODR platform were randomized to either receive or not receive additional information about negotiating their charges and alternative options like payment plans. In Court B, all individuals who received traffic citations received a web address from the citation firm that directed them to either pay the citation and plead guilty, use the ODR platform, or appear in court. Eligible users were randomized to either receive or not receive a postcard that encouraged them to use the ODR platform.

Obstacles

With Court A, we faced four major barriers: 1) Court A abandoned plans to advertise the ODR pilot; 2) The court buried ODR information in small font, hidden within complex boilerplate language on the citation; 3) The court crafted narrow eligibility guidelines; some individuals who attempted to use the platform were found ineligible; 4) Court A’s citation form both provided information about the ODR platform and scheduled a hearing, causing potential confusion over whether the ODR platform was available to defendants. Court A also set an internal deadline for registration for the ODR platform which was not communicated to defendants.

With Court B, we faced three key obstacles: 1) Court B moved in-person hearings online at the start of the study at the start of the COVID-19 pandemic, potentially making the platform less attractive; 2) Court B paid for its ODR vendor to process 1,500 cases, after which the court passed the cost to users; and 3) The ODR vendor administered our survey incorrectly, missing some participants and allowing others to make multiple, inconsistent submissions.

Results

In Court A, only one individual registered to use the ODR platform over four months. Facing such low usage, we recommended terminating the study, and Court A accepted our recommendation.

In Court B, the encouragement deployed failed to encourage. The ODR usage rate in the group randomized to receive the postcard was 72.5%, as compared to 72.6% for the no-postcard group; the corresponding p-value was a .996, with a frequentist confidence interval for the treatment effect of (-.075, .075). (P-value quantifies the possibility that the observed differences are due to chance and not the intervention; a low p-value, such as below .05, indicates that the difference is not likely due to chance.) With no discernible difference in ODR usage rates, we could infer nothing about the effect of ODR.

An Unexpected Finding

While not part of our research design, we discovered a large and statistically significant reminder effect. We discovered that litigants to whom we sent the encouragement postcard were 23% more likely to have resolved their cases within the experiment timeframe, regardless of whether they used the ODR platform (p < .0005, 95% confidence interval (.16, .31)). In addition, litigants to whom we sent the postcard were 12% more likely to appear for their hearings (p > .002, 95% confidence interval (.05, .19)). The size of the reminder effect comfortably exceeded expectations for the effects of postcard reminders found in nudge literature.

We postulate that the reminder effect arose from the combination of the encouragement and the presence of the ODR platform, especially given that the ODR platform permitted asynchronous negotiation and resolution. The postcard may encourage participants to “take care of this quickly right now,” and the ODR platform provides them with a means to do so from their homes not otherwise available.

Suggestions for Courts Implementing ODR

We encourage courts to conduct RCTs in which users are randomly assigned to ODR platforms directly, rather than through the proxy of the encouragement postcard. The former enables direct observation of the effects of ODR. While many in the U.S. legal profession remain skeptical toward randomization, it is the gold standard for research. We hope that the results here encourage additional research into the efficacy of ODR.

We have three additional suggestions. First, courts should carefully consider and balance cost and utility before deploying ODR platforms. Many vendors charge on a per-case basis. Courts who craft broad eligibility guidelines may see higher costs if many litigants elect to use ODR. Courts may accept this cost if the ODR “buys” more access to justice for litigants who otherwise would not have responded.

Some have considered potential future “open source” solutions to reduce the costs of ODR systems, but open-source software contains hidden costs given developer unfamiliarity with the code, a lack of support staff, and issues with scaling. Thus, courts should not look to open-source solutions as an immediate solution to ODR platforms’ cost.

Second, courts desiring greater ODR adoption should craft broad, clear eligibility criteria. As shown by our studies, narrowly crafted eligibility requirements, failure to remind users of ODR, or unintentionally hiding registration information for ODR are all likely to limit the number of users. Our research suggests that reminding users and reducing registration friction increases participation. 

Lastly, courts should carefully evaluate compatibility between court use cases and vendor features before adopting any ODR system. Various technical issues have delayed different courts’ implementation of ODR systems and increased overhead costs, including issues in Application Programming Interface (API) and database integration. Responding to these delays, courts have recommended establishing a project management team to oversee the ODR deployment process.

Lessons in A2J Techniques from ODR Evaluations

June 22, 2022 by Renee Danser

The rise in court-hosted online dispute resolution (“ODR”) is noticeable. But does it work? It seems intuitive that if courts are widely deploying ODR, touted to improve access to justice by removing the inconvenience of—some argue any necessity to—having to visit a physical courthouse to participate in one’s case, we would find strong evidence to support such wide adoption.

That strong evidence base does not currently exist.[1] In fact, no credible evidence as to ODR’s effectiveness, one way or another, exists. We tried ourselves to investigate whether ODR works. Unfortunately, we still don’t know.

If you are reading this post, you likely know already that at the Access to Justice Lab at Harvard Law School, we focus on credible, meaning almost always randomized, evaluation to understand the direct causal impact of those interventions using the randomized control trial.[2] In this post, we summarize two evaluations completed in partnership with the Iowa Judicial Branch and the 11th Judicial Circuit of Florida, invaluable partners.

In the coming months, you can watch for a longer publication going into more detail. Here we will proceed as follows: first we will remind readers of why randomization is an important component to a credible evaluation design. Then we will summarize our two studies, which differ in goals and design, but neither of which tested whether the key question of whether ODR (versus no ODR) affects relevant outcomes. Finally, we will end with lessons learned and opportunities for future research/court partnerships.

The Importance of Randomization in Evaluation Designs
Why are we so committed to evaluations incorporating randomization? Randomized studies select groups statistically identical to one another except for that one is not exposed to an intervention or program (here, the availability of an ODR platform), allowing us to know with as much certainty as our current knowledge systems allow that the reason for any observed differences in outcomes is the intervention or program. By contrast, a commonly used methodology that compares outcomes prior to an intervention’s implementation to outcomes after the implementation could be rendered of little value by changes, fast or evolutionary, occurring at about the same time as the intervention. Such factors might include a change in presiding judge, a new crop of mediators or lawyers working on these cases, a change in mechanism to access the court such as by phone or synchronous or synchronous online interaction (uncoincidentally, similar to how ODR works), a change in filing fee amount, a change in way cases process through the court, or change in thinking among members of the bar regarding what is trial worthy and what is better to settle. The gold-standard RCT neutralizes these potentially influencing factors as much as we currently know how to neutralize them.

It was the Best of Times, it was the Worst of Time: A Tale of Two Studies
As suggested above, the threshold question of whether ODR works or not is not yet answered. Before we understand what components of ODR make it better or worse, we need to know if the concept overall works. We went into these evaluations hoping to investigate that important first inquiry. We came closer in Florida.

Florida
In Florida we attempted to use an encouragement design.[3] We sought to answer the question of whether providing encouragement to use an ODR platform to resolve traffic compliance matters results in more use of the platform and, if so, whether those that use the platform experience better outcomes as compared to those who do not. The hope was that people receiving encouragement would do the thing we encouraged them to do at much higher rates than those who did not receive the encouragement. If that hope had been realized, then by randomizing encouragement to use the intervention—giving encouragement to some and not others, randomly selected—we would have effectively been randomizing the intervention.[4]

Encouragement came in the form of a postcard. Individuals with eligible alleged traffic infractions were randomly assigned to receive this encouragement or not. Nothing else about their case changed: law enforcement still issued the same citations, cases were still scheduled as a matter of course with the court, and they proceeded if no other action was taken prior to that scheduled date (such as paying the ticket or using the ODR platform to show remediation of noncompliance). We ended with 289 study participants getting the encouragement and 274 not.

Encouragement designs work only if the encouragements . . . er . . . encourage. Our postcard didn’t. We weren’t terribly optimistic that it would. But we were unable to persuade stakeholders to adopt a stronger design.

In other words, our two groups used the ODR platform at nearly the same rate, meaning we are not able to untangle the effect of ODR itself from other possible outcomes.

What we did see in our data is a possible reminder effect[5] to a large degree. We saw those that received the encouragement postcard were more likely to appear at their subsequent court events at a rate of twelve percentage points. And, perhaps a product of that appearance, those receiving the postcard were more likely to resolve their case at a rate of twenty-five percentage points. Previous researchers have observed reminder effects, usually with postcards or text messages about hearings, but that those have tended to be in the five-eight percent range.[6] The fact that we’re observing larger effects leads us to hypothesize that the combination of a reminder and a convenient method of resolution may be larger than the reminder effect alone.

What we think we see in this data is a lot of people were able to access the ODR platform and did so notwithstanding encouragement. But, subsequently, those that received the encouragement paid more attention to their cases than those who did not. We cannot be definitive about this finding. This is not the outcome for which we were testing. But, it is a hypothesis that emerged and deserves more attention from future research.

Iowa
The Iowa study differed in goals and design. The Iowa Judicial Branch deployed the ODR platform, for the purposes of this evaluation, for a handful of pre-selected traffic infractions. The Iowa Judicial Branch agreed to randomize neither access to the platform nor encouragement to use it. Instead, the Branch agreed to randomize information about payment plans available via the court system as well as information about what prosecutors ordinarily negotiate. Randomization would occur for litigants who created accounts to use the platform.

In the case of information about payment plans, that information was not a secret but also was not openly available, either. Usually it required a litigant to affirmatively ask for a payment plan rather than being an option the litigant could select.

In the case of reasonable expectations, the idea was that giving the litigant some information about what a prosecutor might offer in a plea might help the litigant to decide to pursue negotiation or resolve the matter more quickly (presumable by paying or expediting a not guilty plea).

We were not particularly optimistic that randomization of information of this type would produce much of a treatment contrast. As it turned out, the issue was not the treatment contrast but the platform. During the several months of enrollment, no one used it. Actually, over the course of enrollment, only one participant successfully made it through the platform. Volume did tick up slightly after enrollment closed, but remained too low for an evaluation.

Lessons Learned and Opportunities
Some themes emerged from both evaluation attempts which we think are useful to courts as they move to ODR 2.0. We will use this section to highlight some. Not all applied in each jurisdiction; some applied in both.

Informing the User about the Option
It seemed that courts did not make substantial efforts to inform the user about the option to use ODR. Neither of our evaluation jurisdictions implemented a program mandating the use of ODR to resolve the selected use, which likely would de facto serve to inform the user about the platform. Instead, both made use of ODR opt-in. Litigants cannot opt into something unless they know it exists. In traffic matters, most courts attempt to alert litigants to the existence of ODR by including text about it on citation forms. In most jurisdiction that we have observed, in this study and others, citations are not a good forum for notifying anyone of anything important. They are packed full of dense text with numerous unintelligible statutory citations. The notification of the ODR option amounts to a URL that may not intuitively appear connected to the court. Notwithstanding the URL and the option to resolve one’s case without attending a court hearing, the citation has a court date at which, the citation says, the alleged offender is compelled to appear. It is easy to see how litigants may get confused and/or disregard the ODR option.

Expanding Eligibility Guidelines
In implementing ODR, some courts fear that a deluge of users will flood the platform, making the platform cumbersome for the court to manage. This fear can result in narrowly drawing eligibility guidelines. That could be a method to thin the herd of users. But attempts to clarify who can and cannot use an ODR platform may also confuse potential users, resulting in some eligible users concluding that they cannot use the platform and some ineligible users concluding the opposite. When determining eligibility for ODR, and perhaps any program, we should consider what will make logical sense to the user. Particularly given the Iowa experience, broad rather than narrow eligibility guidelines may help to increase usage.

Clear Timelines
Courts often develop processes around the idea that judicial time should not be wasted. Likely the process discussed above of providing in the citation both information about ODR while also scheduling the next hearing is to preserve judicial time and to keep cases moving. Internal processes around deadlines for users to complete ODR developed with likely the same goal in mind. However, these ODR-specific internal deadlines are not, as far as we were able to observe, communicated to the user, and they differed from the live-appearance deadlines. This results in individuals otherwise eligible to use the platform being unable to do so because an undisclosed deadline for platform use has passed. After the passing of this deadline, the only option is to appear in court to resolve the matter. ODR-specific deadlines should be communicated. Indeed, we wonder what harm there is in either letting someone resolve their case using the platform right up until the day of the hearing, or only scheduling a hearing after a disclosed deadline passes, with the latter option seeming to accommodate both the robust use of the platform and the preservation of judicial time.

Increasing Access to Justice with Simplification and Reminders
Not everything went as we had hoped in these two evaluations. We came away with some thoughts on improvement of court installed ODR platforms. We also came away with a hypothesis that could be a boon to the access to justice community. The idea that combining tools that are separately thought of as improving access to justice (here, reminders and a tool to ease case resolution) might work. The takeaway here is that simplified processes coupled with reminders may create a significant increase in usage of tools designed, we think, to improve access to justice. We are looking forward to testing, and observing others test, this hypothesis in the future.

Support for this project was provided in part by The Pew Charitable Trusts. The views expressed herein are those of the author(s) and do not necessarily reflect the views of The Pew Charitable Trusts.

[1] There are a handful of empirical studies that investigated user experiences in ODR processes. See, Martin Gramatikov & Laura Klaming, Getting Divorced Online: Procedural and Outcome Justice in Online Divorce Mediation, 14 J.L. & FAM. STUD. 97, 117 – 8 (2012) (“finding high levels of satisfaction with online divorce procedures and quality of outcomes of both male and female divorcees in the Netherlands, although the former focused more on monetary and time costs while the latter focused on negative emotions”); See, Katalien Bollen & Martin Euwema, The Role of Hierarchy in Face-to-Face and E-Supported Mediations: The Use of an Online Intake to Balance the Influence of Hierarchy, 6 Negotiation and Conflict Management Research 4:305 – 19, 313 (2013) (“finding that a hybrid process combining online intake with face-to-face mediation had an equalizing effect in hierarchical labor settings on parties’ fairness and satisfaction perceptions”); See, Marc Mason & Avrom H. Sherr, Evaluation of the Small Claims Online Dispute Resolution Pilot, Institute of Advanced Legal Studies, at 19 (Sept. 1, 2008), (finding a lower settlement rate than offline small claims mediations, as well as problems such as the online system timing out, the registration process, spam filtering, a lack of transparency, and digital access and competency, although the study was limited in scope and only had a sample size of 25 cases in the UK); See, Laura Klaming, Jelle van Veenen and Ronald Leenes, I Want the Opposite of What You Want: Reducing Fixed-pie Perceptions in Online Negotiations, 2009 J. DISP. RESOL. 139:85 – 94, 92 – 93 (finding that “providing negotiators with incentives independent from the resources that have to be divided, as well as providing them with information about the opponent’s preferences, led to more agreements”); See, Udechukwu Ojiako et al., An Examination of the ‘Rule of Law’ and ‘Justice’ Implications in Online Dispute Resolution in Construction Projects, 36 International Journal of Project Management 301, 305 , 308(2018) (“finding that the ODR process does not affect parties’ satisfaction with the “rule of law” or “justice” in small claims ODR in construction projects, while suggesting further research on the cultural contexts around these concepts”), Due to the limited scale on which ODR has been implemented in American courts, there are few independent efforts to quantify the outcomes of ODR initiatives in the public sector. “There are some self-reported pre-ODR and post-ODR datasets, mostly compiled by courts and private platforms and unconfirmed by independent research” (See, Joint Technology Committee Resource Bulletin, Case Studies in ODR for courts; A View from the Front Lines, 3 – 18 (2017); Amy Schmitz, Expanding Access to Remedies Through E-Court Initiatives, 67 Buff. L. Rev. 89, 158 (2019); Kevin Bowling, Jennell Challa, & Di Graski, Improving Child Support Enforcement Outcomes with Online Dispute Resolution, Trends in State Courts, 43 – 8, 46 (2019), and Avital Mentovich, J.J. Prescott, & Orna Rabinovich-Einy, Are Litigation Outcome Disparities Inevitable? Courts, Technology, and the Future of Impartiality, 73 Ala. L. Rev. (2020) at 893.

[2] See, Joshua D. Angrist, Instrumental Variables Methods in Experimental Criminological Research: What, Why and How, 2 Journal of Experimental Criminology 23 – 44, 24 (2006) (arguing that randomized studies are considered the gold standard for scientific evidence).

[3] Conner Mullally, Steve Boucher, & Michael Carter, Encouraging Development: Randomized Encouragement Designs in Agriculture, 95 American Journal of Agricultural Economics 5:1352 – 8, 2 (2013) (defining the encouragement design); Paul J. Ferraro, Counterfactual Thinking and Impact Evaluation in Environmental Policy, Environmental Program and Policy Evaluation: Addressing Methodological Challenges. New Directions for Evaluation, 75 – 84, 80 (2009) (suggesting the encouragement design may be appropriate when randomly restricting access to an intervention cannot be done). One assumes Ferraro means literally cannot, or, in other words, ethically impermissible to, rather than a preference not to, randomly restrict access. As an aside, Ferraro’s description of the need for rigorous evaluation in environmental policy is directly analogous to the need for same in the legal sphere. See generally, D. James Greiner & Andrea Matthews, Randomized Control Trials in the United States Legal Profession, Annu. Rev. Law Soc. Sci. 12:295 – 312 (2016).

[4] Id., Mullally et. al., (“Assignment to th ‘encouraged’ group is then used as an instrumental variable in order to estimate the impact of the treatment”).

[5] For more information about reminder effects, See, e.g., Brian H. Bornstein, et. al., Reducing Courts’ Failure to Appear Rate: A Procedural Justice Approach, (May 2011), https://www.ojp.gov/pdffiles1/nij/grants/234370.pdf (evaluating the effectiveness of different messaging approaches in mailed post cards); Timothy R. Schnacke, et. al., Increasing Court Appearance Rates and Other Benefits of Live-Caller Telephone Court-Date Reminders: The Jefferson County, Colorado, FTA Pilot Project and Resulting Court Date Notification Program, 393 Ct. Rev.: J. Am. Judges 86 (2012) (evaluating the effectiveness of providing information about the consequences of failing to appear by live-calling individuals); Christopher T. Lowenkamp, et. al., Assessing the Effects of Court Date Notifications within Pretrial Case Processing, 43(2) Am. J. Crim. Just. 167, 173 (2017) (evaluated the effectiveness of different messaging approaches and different methods of delivering notifications); Brice Cooke, et. al., Using Behavioral Science to Improve Criminal Justice Outcomes: Preventing Failures to Appear in Court, (January 2018), https://www.prisonpolicy.org/scans/Using_Behavioral_Science_to_Improve_Crimina_Justice_Outcomes_Cooke_et_al_2018.pdf (evaluated the effectiveness of the timing and messaging approaches in text notifications); Stephen H. Taplin, et. al., Testing Reminder and Motivational Telephone Calls to Increase Screening Mammography: A Randomized Study, 92(3) J. of the Nat’l Cancer Inst. 233 (2000) (finding reminders to be as efficacious as addressing barriers with phone call reminders performing better than postcards); Susan Maxwell, et. al., Effectiveness of Reminder Systems on Appointment Adherence Rates, 12(4) J. of Health Care for the Poor and Underserved 504, 508 (2001) (finding show rates for appointments for those who received no reminder to be 49.9% as compared to those that received a mailed reminder to be 52.1%); Mary Elaine Koren, et. al., Interventions to Improve Patient Appointments in an Ambulatory Care Facility, 15(4) J. Ambulatory Care Mgmt. 76 (1994) (finding insignificant difference between the type of reminder, when using phone or mailings, but did find some reminder to be more effective than no reminder).

[6] See, Id.

  • Go to page 1
  • Go to page 2
  • Go to page 3
  • Interim pages omitted …
  • Go to page 25
  • Go to Next Page »

Using empirical research to make the U.S. justice system work better for everyone.

The Access to Justice Lab

Copyright © 2023 · Monochrome Pro on Genesis Framework · WordPress · Log in

 

Loading Comments...