By Michelle Blouin, Communications Associate, Access to Justice Lab, Harvard Law School
The Access to Justice Lab’s study, which we casually refer to as the “Online Courts” study, was designed to be the first randomized controlled trial to test the pre-pandemic status quo of in-person hearings versus the pandemic status quo of virtual hearings, all in the context of family law. Several years of data collection and observational research and some funding dollars later, the study produced no usable conclusions to share. So, what happens now? We play Tiffany’s 80s hit “Could’ve Been” on repeat as we write a blog about the lessons learned from a flawed study.
What Could’ve Been
We’ve conducted dozens of RCTs in the last 10 years. In the case of the Online Courts study, officially called “Evaluating the Impact of Virtual Proceedings in Family Law Matters,” this should have been a straightforward research project akin to others we’ve conducted.
It all started with the Covid-19 pandemic and how it changed the way we did things (Zoom parties, anyone?). All but the “most essential” personnel adapted to working virtually. Courts, in turn, adapted as a justice service provider rather than a place to serve justice. With court proceedings occurring virtually, access-to-justice debates centered on in-person constraints (transportation issues, scheduling conflicts, and proximity to courthouses) versus technology constraints (access to computers and high-speed Internet and understanding of software). Questions ensued over the efficiency and outcomes of either setup, but no evidence existed in favor of a particular medium. Emerging from the pandemic, judges and court administrators in the Third Judicial District in Utah (Salt Lake City) were split on the mode of hearing, specifically for self-represented litigants in family law cases; post-pandemic, two of the four family court commissioners continued to hear pro se litigant cases remotely (as was the norm during the pandemic), and the other two commissioners returned to in-person hearings for pro se litigants (as was the norm pre-pandemic). The disagreement—and interest in finding the better medium—prompted the formation of the A2J Lab’s Online Courts study.
The Initial Assumption
Family courts, more than any other judicial setting, see the highest number of self-represented litigants—about 80 percent. Based on the few written articles related to this field, A2J Lab researchers believed that both in-person and virtual proceedings had merits and faults, and they hypothesized that, with access-to-justice barriers already stacked against pro se litigants, the mode of hearing mattered less than the litigant having the choice on whether the hearing is conducted in person or virtually.
Study Specs
Both pre-pandemic and mid-pandemic, pro se litigants never had a choice in the medium of their hearings, and neither mode—in person nor virtual—was determined to be a “better” option anyway (“equipoise” for those interested in measurements of ethical standards). So, given that the post-pandemic hearings also did not provide a choice to litigants, the A2J Lab was able to conduct the research as a non-consent study, approved by the Institutional Review Board, which governs the ethics of human-related studies. Data retrieved from the study is publicly available and not sensitive in nature as well. So the court, with the Lab’s assistance, assigned self-represented litigants in divorce or custody matters randomly to either in-courtroom or WebEx/remote proceedings. The randomization accounted for a balance in age, race, and gender.
In comparing in-person and remote hearings, we collected data to measure the following outcomes based on the medium:
- Time to resolution—the time that the case was activated, requiring court action on a matter, upon entering the study to the time that the matter was resolved
- Durability of court order—the time between a disposition and a request to make a change on that disposition (measuring participant satisfaction with the order)
- Failure to appear—the measurement of parties appearing for their proceedings
- Court burden—measurement of how involved the court needs to be in a case (tends to be higher burden for more contentious cases)
Court data is notoriously difficult to work with, and this study data confirmed that notion. A2J Lab Data Analyst Patricia Gansert, in the accompanying podcast, noted court administrative obstacles stemming from typewriter data entry issues (resolved in Patricia’s data transfer); missing details on the presence of individuals in hearings (mainly resolved in Patricia’s data transfer); and categorization issues (resolved with help from court administrators). In spot checking the transcripts from 20-30 court hearings, the researchers then confirmed the accuracy of the data the lab coded based on the minutes.
All this is to say that the lab conducted its research thoroughly and was confident in the accuracy of the data.
What Went Wrong
In short, the players didn’t follow the plan.
The Court did not comply with the randomization protocol for the study. In many cases—too many to make the data truly scientific and useful—the Court assigned the hearing mode, in person or remote, on its own rather than according to the assigned randomization. The data could still be useful if we could ensure those decisions were truly random; but we don’t know the Court’s motivation in making the decision (e.g., was a decision for remote hearing due to a Commissioner’s knowledge of the distance a litigant lived from the court? Did a Commissioner decide on an in-person hearing based on the contentious nature of a case?).
“A lot of hearings were not in the intended medium,” said Gansert. “When we look at the case level, just under 50% of our cases that were randomized in person actually had all of their hearings in person; for virtual, about 30% had all of their hearings virtually. So you can see that the level of compliance was pretty low.”
If the decisions for hearing mediums pointed to a pattern, the study could potentially continue with the A2J Lab data analysts adjusting for the change in randomization assignment. That wasn’t the case. Without the controlled environment of the randomization study, the Lab could not point to a particular medium producing better outcomes.
Anecdotally, the Commissioners explained that they made some choices on hearing mode based on what they thought was best at the time. They also underestimated the frequency of their assignment deviation, believing that they faltered only occasionally. This digression from the study plan sparked conversations with the court partners about potential bias in deciding hearing modes and also underscored the necessity for compliance in research.
Fragile Findings
Without real evidence behind the data, we have just a broad overview of study results that show no real differences between in person and remote hearings when looking at time to resolution, durability of court order, failure to appear, and court burden rates. The researchers outlined these findings, or lack of findings, in a final report released this summer.
Though we may be throwing our hands in the air on this study, it could be that our original hypothesis may stand: litigants, perhaps, should have a say in the way they appear for court hearings.
“We do our best to try to draw everything we can from RCTs that crash and burn, and we put out this one as an instance to show just how fragile field operations are,” said HLS Professor Jim Greiner, faculty director of the A2J Lab. “When we complain all the time that field operations are expensive, time-consuming, long and fragile, we mean it.”
Alongside the quantitative study, the A2J Lab also worked with research partners on a qualitative study looking at litigants’ perceptions of fairness based on the mode of hearing; we’ll share these results in next week’s Proof Over Precedent.
If you’re interested in more on this topic, listen to our podcast episode.