Why RCTs? Recent study on stents is one example

This past week, we’ve been avidly watching reactions to a new study, published in The Lancet, about the efficacy of using stents to help patients with chest pain. The New York Times ran an article on the study; so did The Atlantic.

If you haven’t been following this (potential) bombshell of an RCT, the study found no value in using stents to combat heart pain. Why is this such big news? Partially because using stents for cardiac pain is big business. According to the study’s authors, more than 500,000 patients receive the procedure annually for chest discomfort.

It’s also big news because it goes against intuition, even the sort that medical laypeople possess. Without evidence to the contrary, it might seem logical that opening blocked arteries with a stent would reduce chest pain. No wonder doctors adopted the practice with vigor! Now there are data that don’t back up that perception. Even in medicine, a field long conditioned to accepting the validity of empirical research, studies will bump up against the fallacy of conventional wisdom.

That fact doesn’t surprise us at the A2J Lab. What did grab our attention is that the authors received permission to run the study at all. As we mentioned in a recent post, all RCTs in the U.S. need to receive institutional approval before human subjects can enroll in a study. Based on our experience, it would be fairly startling if this type of study, which flies so baldly in the face of “conventional wisdom,” were to receive approval in the United States. An ethical review committee could have responded that this evaluation would prevent some participants from receiving a “benefit,” namely the treatment they “need.” The deeper held the belief, the harder it is to accept or allow the introduction of contrary evidence. That’s why we need to test interventions rigorously, particularly when resources are scarce and lives are at stake.

One final note on the study’s design. Critiques from medical researchers have included that the study is flawed due to “Type II error.” In short, they contend that the sample size (in this case, about 200) is insufficient to rule out false negatives. The challenge of having sufficient sample size is an important component of any RCT. The Lab, for example, uses power analysis to maximize the chance that a study will have enough observations to detect an effect, should that effect really exist. But a study’s sample size isn’t the only factor that’s important in determining its validity; it’s also important to know how generalizable the results are, regardless of their statistical significance.

This is just one more example of why RCTs are important. Have you seen others recently? Share them with us in the comments or on social media.

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

RITES Goes Local

We’re excited to announce a new collaborative venture on the Rhode Island Triage and Eviction Study (“RITES”) team. Five outstanding students at the Roger Williams University School of Law have joined the project to conduct summary eviction court observations and interview unrepresented tenant-defendants who are experiencing the eviction process themselves. These raw data, so to speak, will give everyone working on the project a better understanding of the current system and inform ongoing study design. For example, information about what tenants would have liked to know before their hearings will enhance the self-help materials designed for the evaluation.

In addition to this unique learning experience, their participation strengthens the project’s connections within the Providence community. These dedicated students—under the supervision of Professor Jonathan Gutoff; Director of the Feinstein Center for Pro Bono & Experiential Education, Laurie Bannon; and Director of Pro Bono & Community PartnershipsEliza Vorenberg—will bring important new perspectives and added capacity to the growing RITES field organization.

This partnership is an exciting one for us at the Lab. One of our core missions is to work with the next generation of legal practitioners and scholars and cultivate dedication to making the law an evidence-based profession. Having those future leaders actively participate in all phases of a Lab study is the most effective way for us to do just that.

We also prioritize having multiple partners work together on RCTs to maximize their impact. Having RWUSOL join us and the Rhode Island Center for Justice as we study the latter’s triage process for assigning representation in eviction cases will only improve our work. We all will benefit from each other’s perspectives, ideas, and insights into the understudied process of distributing scarce legal resources.

Stay tuned for more on this venture!

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Guardianship Service of Process

The Problem

Service of process can be a very complicated step in obtaining legal guardianship. The phrase “service of process” alone is a confusing one.

Petitioners, most of whom are not lawyers, have to: (1) identify “interested parties,” many of whom are not obvious candidates; (2) determine the proper method of service; (3) effectuate service; and (4) return proof of service to the Probate and Family Court. Completing the process exactly as described is equally important. Service isn’t just a legal formality; it’s a crucial part of the petition. If interested parties, i.e., those who might want to contest the petition aren’t notified, due process concerns would arise. Failing to serve within the prescribed timeline will stall the petition.

People who have gone through the process have described the paperwork as overly complicated, repetitive, and time-consuming. Worse, a significant number of petitioners fail to reach a judge at all—not because of the substance or validity of their case, but because they have failed to overcome the procedural hurdles standing in the way of having a case heard on the merits.

Current Solutions

Courts and legal aid organizations provide individual assistance explaining court procedures. Courts have, for example, made attempts at drafting checklists or other instructions about the process. Many legal service providers develop their own self-help materials or employ different techniques to get litigants to remember at least some of these very complicated steps. Some tell litigants to come back once they receive a new piece of mail from the court, so that the next step can be explained to them in a way that is more concrete and obvious. Repeat visits, in-person explanations, and drafting instructions all take significant time and energy that attorneys could otherwise spend assisting more court users. Are these solutions having any effect on litigants’ ability to navigate the court procedure and get their first hearing in front of a judge?

The Study

In partnership with the Boston Court Service Center and the Volunteer Lawyers Project of the Boston Bar Association, the Lab’s Guardianship Service of Process study evaluates whether self-help materials can make a difference for court users navigating the complex web of court procedures to initiate a guardianship case.

The Guardianship Service of Process Study, which launched in early September 2017, tests Lab-designed self-help materials. Participants receive printed materials (developed in large part at our first hackathon) on a randomized basis for both adult or minor guardianship cases and in English or Spanish. In addition, minor guardianship petitioners randomized to receive the hard copy booklets will also gain access to an online tool developed by Bill Palin, the Access to Justice/Technology Fellow with Harvard Law School’s clinical programs. That site walks users through their case to provide personalized instructions, using new guided interview software similar to TurboTax. The RCT will compare rates of successful service, among other outcomes, between the treatment and control groups.

If self-help packets or a new tech tool can help people file for guardianship and then correctly complete service of process, then legal services providers know what types of resources to invest in and how best to allocate their limited resources. And if the self-help materials aren’t at all effective, perhaps we can learn something about the procedural hurdles and have a better understanding of how these hurdles themselves may need to change.

 

The Research Team

Jim Greiner, Faculty Director, The Access to Justice Lab; Professor of Law at Harvard Law School

Chris Griffin, Research Director, The Access to Justice Lab

Erika Rickard, Associate Director of Field Research, The Access to Justice Lab

With intervention design thanks to our affiliates,

Bill Palin, Developing Justice, Harvard Law School

Hallie Jay Pope, Graphic Advocacy Project

The Lab’s New Guardianship Service of Process Study Has Launched!

As of yesterday, September 5, a new A2J Lab study is in the field at the Edward W. Brooke Courthouse in downtown Boston!

CSC staff attorney Jorge Cólon couldn’t be more excited to have our hard copy self-help materials.

Together with our partners at the Boston Court Service Center (“CSC”) and the Volunteer Lawyers Project (“VLP”) of the Boston Bar Association, we are evaluating (via RCT, of course) whether self-help materials can make a difference for court users seeking guardianship over incapacitated adults or minors. CSC and VLP reported high rates of return visits from users they assisted with filling out petitions. Why? Those petitioners often got stuck trying to navigate the often-labyrinthine service of process requirements.

A previous blog entry revealed just how confusing the process of service can be. Petitioners, most of whom are not lawyers, have to:  (1) identify “interested parties,” many of whom are not obvious candidates; (2) determine the proper method of service; (3) effectuate service; and (4) return proof of service to the Probate and Family Court. Completing the process exactly as described is equally important. Unless parties complete all the steps in the correct order, the guardianship matter cannot proceed.

Cólon and his CSC colleague Carolin Hetzner are already distributing (randomly!) Blob’s latest adventures to guardianship petitioners.

Because the process is complicated and the constituencies served have limited access to legal resources (beyond the excellent help of the CSC and VLP), the Lab’s familiar promotion and development of self-help materials seemed like a natural response. The associated RCT will lead to randomized provision of printed materials (developed in large part at our first hackathon) for both adult or minor guardianship cases and in English or Spanish. Once again, Hallie Jay Pope, the intrepid leader of the Graphic Advocacy Project, designed vivid flowcharts, “happy maps,” and new manifestations of the Lab’s favorite humanoid, Blob, to enrich the paper product. In addition, minor guardianship petitioners randomized to receive the hard copy booklets will also gain access to an online tool developed by Bill Palin, the Access to Justice/Technology Fellow with Harvard Law’s clinical programs. That site walks users through their unique legal needs, much like the software pioneered by TurboTax and other online service providers. The RCT will compare rates of successful service, among other outcomes, between the treatment and control groups.

What happens now? The first steps are randomizing cases and letting users go forth with service of process; that’s the part that began yesterday. We’ll start collecting data later this month. Stay tuned here for continued updates on the project!

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Public Safety Assessment Featured on NPR’s Planet Money

Our first RCT studying the Public Safety Assessment (“PSA”), the pre-disposition release assessment tool developed by the Laura and John Arnold Foundation, is well underway in Dane County, WI.

Madison is far from the only place the PSA currently is in use; this recent piece from the Planet Money podcast highlights the implementation of the PSA in New Jersey.

The PSA is up and running in New Jersey, as well as the entire State of Kentucky and scattered counties from Santa Cruz, CA to Volusia, FL. Although the PSA has been scientifically validated, only the results from our Dane County RCT—and, hopefully, upcoming trials in other locations—will give us a good sense of how well the PSA really works for a given jurisdiction.

If you have some time on your commute (or whenever you listen to the news), take a listen. It’s about twenty minutes long, and it will give you a great sense of what we’re studying in the PSA trials. Prefer to read the story? The full transcript is also available online.

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Assessing Pretrial Risk Assessment

Of Scores and Simulations

Evidence-based practices in criminal law proceedings are rapidly gaining traction. Some high-level officials and scholars have voiced concerns about their use in sentencing, as the empirical turn in judicial decision-making continues to gain momentum. A significant portion of the A2J Lab’s mission is to take the practice community’s concerns seriously and then rigorously evaluate solutions to procedural problems.

Divorce

The Problem

For the past 30 years, an increasing number of people come to court without a lawyer. As more people come to courts to access their legal rights, they are met with fewer free or low-cost legal services to help. Across the country, legal practitioners, scholars, and appellate courts have begun to question whether court procedure is effectively preventing access to justice.

Current Solutions

Courts, legal services providers, and state and local Bars have responded to the flood of people without lawyers in numerous ways, including:

  • amending ethical rules to legitimate already-existing forms of lawyer representation
  • self-help centers
  • uniform court forms
  • self-help materials
  • technology
  • and non-lawyer representation.

In 2015, the Conferences of Chief Judges and State Court Administrators passed a joint resolution adopting “the aspirational goal of 100 percent access to effective assistance for essential civil legal needs.”

Some of the efforts to stem the tide of pro se, self-represented, or unrepresented litigants have focused on connecting people with lawyers: by increasing pro bono efforts and leveraging different technologies to connect people with free legal aid or low-cost representation. Recent efforts, however, primarily focus on alternatives to representation, and very few efforts address changes to court processes themselves.

The Study

Field operation

During the study, potential clients seeking a divorce underwent a 45-60-minute interview to determine eligibility and learn more about the details of their case.

After the interview, consenting study-eligible individuals were randomized to one of two groups:

  • Treated group: an effort by the service provider to find a pro bono attorney to represent her;
  • Control group: a referral to existing self-help resources and an offer to answer questions by telephone.

Results

We reviewed the court case files for all study participants, to review which cases successfully (a) filed for divorce in court, and (b) got divorced.

divorceinfographic

The Research Team

Jim Greiner, Faculty Director, The Access to Justice Lab; Professor of Law, Harvard Law School

Ellen Degnan, Law Student, University of Miami School of Law

Tom Ferriss, Quantitative Analyst, Google

Roseanna Sommers, JD/PhD candidate, Yale Law School and Yale University

 

More Information

About divorce proceedings and legal rights

 

Housing Court Study

D. James Greiner, Cassandra Wolos Pattanayak, and Jonathan Philip Hennessy, “How Effective are Limited Legal Assistance Programs? A Randomized Experiment in a Massachusetts Housing Court”  (2012).

We persuaded entities conducting a civil Gideon pilot program in summary eviction cases to allow us to randomize which potential clients would receive offers of traditional attorney-client relationships from oversubscribed legal aid staff attorneys and which would be referred to a lawyer for the day program. We examine outcomes related to whether matters not yet in litigation reached court, possession of the unit, monetary consequences of non-payment of rent cases, and court burden. We find no statistically significant evidence that the Provider’s offer of full, as opposed to limited, representation had a large (or any) effect on any outcome of substantive import. We explore several possible interpretations of our results, and we caution against both over-interpretation and under-interpretation.

District Court Study

D. James Greiner, Cassandra Wolos Pattanayak, and Jonathan Hennessy, “The Limits of Unbundled Legal Assistance: A Randomized Study in a Massachusetts District Court and Prospects for the Future,” 126 Harvard Law Review 901 (2012).

We persuaded entities conducting two civil Gideon pilot programs to randomize which potential clients would receive offers of traditional attorney-client relationships from professional service provider staff attorneys and which would receive only limited (“unbundled”) assistance. In both pilot programs potential clients were defendants in housing eviction proceedings, and both programs were oversubscribed. In this Article, we report the results of one of these two resulting randomized control trials, which we label the “District Court Study,” after the type of the court in which it took place. In the District Court Study, almost all study-eligible eviction defendants received limited assistance in the form of help in filling out answer and discovery request forms, and most also attended an instructional session on the summary eviction process. After receiving this limited assistance, each member of a randomly selected treated group received an offer of a traditional attorney-client relationship from one of the legal services provider’s staff attorneys; each member of the corresponding randomly selected control group received no such offer. We compare outcomes for the treated (offered traditional representation from a service provider staff attorney) group versus the control (no such offer) group on a variety of dimensions, focusing primarily on possession of the unit, financial consequences of the litigation, and measures of court burden.

At least for the clientele involved in this District Court Study, a clientele recruited and chosen by the service provider’s proactive, timely, specific, and selective outreach and intake system, an offer of full representation mattered. Approximately two thirds of defendants in the treated group, versus about one-third of defendants in the control group, retained possession of their units at the end of litigation. Using a highly conservative proxy for financial consequences, treated group defendants received payments or rent waivers worth a net of 9.4 months of rent per case, versus 1.9 months of rent per case in the control group. Both results were statistically significant. Meanwhile, although treated cases did take longer to reach judgment, the offer of representation caused no increase in court burden as measured by other, more salient metrics.

Read more