RCTs and Other Evaluation Methods

With so many types of assessments, evaluation designs, and research methods out there, it can often be really confusing for non-profits to figure out which approach to take when assessing the effectiveness of their programs. The A2J Lab has certainly spent some time talking about the value of the randomized control trial method, but how this method fits in with the other methods out there might not always be super clear. So, we’ve put together a chart that briefly explains some of the most essential evaluation methods, describes when to use them, explains their strengths and limitations, and provides examples of some studies and reports for each method.

The chart can be viewed and downloaded here. We have also created versions of the chart that integrate a technology example scenario and a pro bono innovation example scenario.

A few brief points about how to think about the chart:

  1. Define your research question. Be really clear about what the original research question is that you’re asking and answering. The different methodologies explained in the chart answer different types of questions. Be sure to check out the first column to identify your research question and then determine which methodology will best answer that question.
  2. Know what you know and don’t know. Be really clear about what you can know and what you aren’t able to know when using a particular methodology. Refer to the columns labeled “usefulness” and “limitations” in the chart to understand what each approach will be able to tell you or not tell you. When you report on the results, it is important to not extend the results beyond what you actually are able to know based on the methodology.
  3. Consider multiple methods. Consider using more than one type of evaluation method to make sure you have all of your bases covered. In fact, when we conduct randomized control trials, we often support our studies with other methodologies as well.

Thanks to the Self Represented Litigation Network Research Working Group and to David Udell at the Nation Center for Access to Justice for providing feedback on this chart. If you have any questions about the chart or the various methods explained here, feel free to email April at afaithslaker[at]law.harvard.edu

Leave a Reply

Your email address will not be published. Required fields are marked *