Beyond the extraordinary “Why RCTs?” examples
If you’ve been following our “Why RCTs?” series, you’ve read about some drastic examples across several fields of study- many in the medical field- where RCTs have turned common knowledge and practices upside down. These examples are meant to be shocking; they show us the very real and serious implications that can follow from not testing interventions at all or with non-randomized interventions. The accounts I’ve highlighted in the series illustrate several things:
- First, just because one has a gut feeling of what works, even when one is well-versed in the field—does not mean one is even close to knowing the existence and direction of causal effects;
- Second, randomized testing may at first seem frightening in high-risk situations; however such evaluation may be most needed and appropriate exactly for these types of situations; and
- Finally, it is important to note that, while the shocking examples we have highlighted may appear anomalous, reversals of medical practices due to rigorous evaluation are not that few and far between.
Throughout the “Why RCTs?” series, as well as in other blog posts, the A2J Lab has drawn many parallels between the medical profession and the legal profession. We’ve been drawn to the experiences of the medical profession since the field has been transformed, from essentially the dark ages of the 1920s to a flourishing science that embraces rigorous evaluation and growth.
The few remarkable examples of medical reversal that we have featured aren’t outliers in the medical field. An analysis of articles published over a ten-year period (2044 original articles) in the Mayo Clinic Proceedings, titled “A Decade of Medical Reversals: An Analysis of 146 Contradicted Medical Practices,” provides context for these stunning examples. The meta-analysis found that “the reversal of established medical practice is common and across all classes of medical practice.” And despite these common outcomes, the medical profession still embraces rigorous, randomized testing. Its adherents know the value that RCTs carry in advancing knowledge, particularly with respect to proposed interventions. The ten year analysis also found that most articles were randomized trials that examined new practices and found benefits for that practice (530, or 39.4%). The legal profession also has the potential constantly learn and develop high-quality service provision that is evidence-based and shown to work.
The meta-analysis points to numerous examples, including hormone replacement therapy, that highlight how new interventions (e.g., technological or medicinal improvements) actually proved to be not better, or worse, than previous trends or no study at all. Articles were classified in one of four ways:
- “[R]eplacement, when a new practice surpasses standard of care”;
- “[B]ack to the drawing board, when a new practice is no better than current practice”;
- “[R]eaffirmation, when an existing practice is found to be better than a lesser standard”; and
- “[R]eversal, when an existing practice is found to be no better than a lesser therapy.”
Not all of the over 2000 studies that were analyzed were randomized controlled trials, but most of the reversal articles were (111 of the 146 reversal studies). In a previous covering just one year of publications, the same authors found that 16 of 35 studies (46%) that tested a standard of care resulted in medical reversals. Around the same time that this Mayo Clinic study came out, the British Medical Journal (BMJ) conducted a similar project called Clinical Evidence, in which a review of 3000 medical practices found that, “slightly more than a third of [them] are effective or likely to be effective; 15% are harmful, unlikely to be beneficial, or a tradeoff between benefits and harms; and 50% are of unknown effectiveness.”
Think about this. Research physicians are constantly upending their conventional wisdom thanks to the power of RCTs. Lawyers have the same tools at their disposal, and the A2J Lab’s mission is geared toward implementing them in the service of narrowing the justice gap.
Continuous testing in a field is necessary lest it become calcified and detached from emerging evidence. We can learn a lot from the results of any RCT, not just the “shocking” ones. Such rigorous evaluation develops a deeper understanding of how to allocated resources more efficiently and how to reach people who need scarce services the most. Testing in the legal profession doesn’t just mean that some services will be found to be more effective or less effective than others. Rather rigorous evaluation can start to create an evidence-based culture of practice. It will enable legal services providers to continue doing their best work and helping the clients for whom they are champions.
The authors investigating medical reversals argue that “[i]dentifying medical practices that do not work is necessary. The continued use of such practices wastes resources, jeopardizes patient health, and undermines trust in medicine.” We, at the A2J Lab believe the same is more than true for legal practice. Facing hard truths about which approaches work and which are less effective can be a difficult pill to swallow. But the alternative—persisting with techniques untethered from solid evidence—should be just as unacceptable, if not more so.