The Covid-19 pandemic forced courts to change overnight the way they heard cases, from requiring masks in courtrooms to holding hearings online or via the telephone. Online dispute resolution (ODR) systems became a popular method for courts to dispense justice. Proponents have long argued that ODR increases access to justice, mitigates procedural and substantive errors, and conserves court resources—particularly for self-represented litigants. Is this true? The A2J Lab conducted two randomized control trials (RCTs) to assess newly implemented ODR programs. This post discusses those RCTs and one useful and unexpected finding.
What is ODR and Current Landscape
ODR has seen a rise in popularity even before the pandemic. In 2019, the ABA reported that sixty-six courts in twelve states maintained ODR platforms for settling civil and criminal disputes. Between 2018 and 2019, the number of court-annexed ODR platforms more than doubled.
ODR platforms are often classified into two types: instrumental ODR systems offer a virtual platform for facilitating otherwise face-to-face dispute resolution, while principal ODR systems take a “proactive role facilitating the resolution of the dispute.” Courts may implement ODR as a mandatory pretrial process or as an optional alternative to trial. ODR can occur synchronously in real-time videoconferences or asynchronously in chat rooms. ODR systems may operate differently in disputes involving an individual against the state versus disputes between private individuals, and their usage may be free or cost money.
Two vendors, Matterhorn and Modria (the latter recently purchased by Tyler Technologies), currently dominate commercial ODR systems in the United States.
Purported Benefits of ODR
Increasing Access to Justice
Advocates argue that ODR allows parties who lack the time and money to attend an in-person hearing a cheaper and more accessible online mediation process. If individuals can negotiate small claims or contest citations over an asynchronous, online platform, they may be less likely to default.
However, barriers include a lack of willingness to use ODR as well as digital divides. Low-income and disadvantaged communities are more likely to lack the necessary broadband connection for video conferencing.
Mitigating Procedural and Substantive Mistakes
Proponents argue that ODR reduces the risk of procedural and substantive errors on the part of participants. Pro se (self-represented) litigants face substantial obstacles in court to follow the correct procedures and adequately present their case. ODR’s informality and physical separation might allow self-represented litigants to feel less nervous and mitigate biases from face-to-face hearings.
However, some researchers argue that parties who suffer disadvantages in traditional hearings or alternative dispute resolutions may suffer similar disadvantages in ODR. Further, efficiency gains may hinder equity goals. For instance, more efficient resolution of eviction suits by landlords against tenants may exacerbate landlords’ comparative power over tenants.
Conserving Court Resources
Advocates argue that ODR may reduce resource demands on the court. By using ODR, courts might avoid the costs of physical spaces and staff needed for face-to-face hearings. ODR may also reduce the administrative costs of procedural errors by self-represented litigants, such as failure to file documents on time or appear in court.
There has been little testing of all the popularly cited benefits of ODR, and further research is needed.
Our Randomized Controlled Trials (RCTs)
There are currently no RCTs evaluating whether ODR serves the equity, access to justice, or efficiency goals raised by proponents. RCTs are considered the “gold standard” for empirical testing because randomization creates statistically equivalent groups but for the tested intervention, minimizing the possibility that observed differences are due to chance. Unlike studies that compare pre-intervention with post-intervention outcomes, RCTs do not create a risk that another variable influenced experiment outcomes.
The most straightforward way to assess the said benefits of any ODR program is to randomize its usage. The randomization creates two identical groups with a single variable—one of which uses ODR and other of which does not; as a result, any observed differences in access to justice, budgetary, or other outcomes can be attributed to the ODR. Depending on the court’s usage of ODR, the study might randomize either access to the platform (if usage is optional) or orders compelling litigants to use it (if usage is mandatory). Either design, if implemented properly, would produce credible results.
However, the two courts (in different jurisdictions) in our studies declined to implement either design. Instead, in Court A, all individuals who received traffic citations received information about the ODR platform from the citation form. Users who registered for the ODR platform were randomized to either receive or not receive additional information about negotiating their charges and alternative options like payment plans. In Court B, all individuals who received traffic citations received a web address from the citation firm that directed them to either pay the citation and plead guilty, use the ODR platform, or appear in court. Eligible users were randomized to either receive or not receive a postcard that encouraged them to use the ODR platform.
With Court A, we faced four major barriers: 1) Court A abandoned plans to advertise the ODR pilot; 2) The court buried ODR information in small font, hidden within complex boilerplate language on the citation; 3) The court crafted narrow eligibility guidelines; some individuals who attempted to use the platform were found ineligible; 4) Court A’s citation form both provided information about the ODR platform and scheduled a hearing, causing potential confusion over whether the ODR platform was available to defendants. Court A also set an internal deadline for registration for the ODR platform which was not communicated to defendants.
With Court B, we faced three key obstacles: 1) Court B moved in-person hearings online at the start of the study at the start of the COVID-19 pandemic, potentially making the platform less attractive; 2) Court B paid for its ODR vendor to process 1,500 cases, after which the court passed the cost to users; and 3) The ODR vendor administered our survey incorrectly, missing some participants and allowing others to make multiple, inconsistent submissions.
In Court A, only one individual registered to use the ODR platform over four months. Facing such low usage, we recommended terminating the study, and Court A accepted our recommendation.
In Court B, the encouragement deployed failed to encourage. The ODR usage rate in the group randomized to receive the postcard was 72.5%, as compared to 72.6% for the no-postcard group; the corresponding p-value was a .996, with a frequentist confidence interval for the treatment effect of (-.075, .075). (P-value quantifies the possibility that the observed differences are due to chance and not the intervention; a low p-value, such as below .05, indicates that the difference is not likely due to chance.) With no discernible difference in ODR usage rates, we could infer nothing about the effect of ODR.
An Unexpected Finding
While not part of our research design, we discovered a large and statistically significant reminder effect. We discovered that litigants to whom we sent the encouragement postcard were 23% more likely to have resolved their cases within the experiment timeframe, regardless of whether they used the ODR platform (p < .0005, 95% confidence interval (.16, .31)). In addition, litigants to whom we sent the postcard were 12% more likely to appear for their hearings (p > .002, 95% confidence interval (.05, .19)). The size of the reminder effect comfortably exceeded expectations for the effects of postcard reminders found in nudge literature.
We postulate that the reminder effect arose from the combination of the encouragement and the presence of the ODR platform, especially given that the ODR platform permitted asynchronous negotiation and resolution. The postcard may encourage participants to “take care of this quickly right now,” and the ODR platform provides them with a means to do so from their homes not otherwise available.
Suggestions for Courts Implementing ODR
We encourage courts to conduct RCTs in which users are randomly assigned to ODR platforms directly, rather than through the proxy of the encouragement postcard. The former enables direct observation of the effects of ODR. While many in the U.S. legal profession remain skeptical toward randomization, it is the gold standard for research. We hope that the results here encourage additional research into the efficacy of ODR.
We have three additional suggestions. First, courts should carefully consider and balance cost and utility before deploying ODR platforms. Many vendors charge on a per-case basis. Courts who craft broad eligibility guidelines may see higher costs if many litigants elect to use ODR. Courts may accept this cost if the ODR “buys” more access to justice for litigants who otherwise would not have responded.
Some have considered potential future “open source” solutions to reduce the costs of ODR systems, but open-source software contains hidden costs given developer unfamiliarity with the code, a lack of support staff, and issues with scaling. Thus, courts should not look to open-source solutions as an immediate solution to ODR platforms’ cost.
Second, courts desiring greater ODR adoption should craft broad, clear eligibility criteria. As shown by our studies, narrowly crafted eligibility requirements, failure to remind users of ODR, or unintentionally hiding registration information for ODR are all likely to limit the number of users. Our research suggests that reminding users and reducing registration friction increases participation.
Lastly, courts should carefully evaluate compatibility between court use cases and vendor features before adopting any ODR system. Various technical issues have delayed different courts’ implementation of ODR systems and increased overhead costs, including issues in Application Programming Interface (API) and database integration. Responding to these delays, courts have recommended establishing a project management team to oversee the ODR deployment process.