Recent scholarship has offered varying interpretations of what the appropriate function of foundations should be within a democracy. One dominant perspective highlights foundations’ contributions as drivers of social innovation, arguing that their financial capacity and independence allow foundations to experiment with promising approaches that have the potential to benefit society. To make this argument convincingly, one tool at foundations’ disposal is to support rigorous evaluation of program effectiveness, thereby drawing on scientific norms to demonstrate the societal benefit of the innovations they have supported. In particular, foundations might be drawn to the use of randomized controlled trials (RCTs), which are experiments that compare the outcomes of a randomly selected “treatment group” of people who receive a program with a randomly selected “control group” of people who don’t. RCTs are widely regarded as the “gold standard” for evidence of program impact, and foundations that use them often do so to bolster the case that foundation investments are effective ones.

In our new book, Mismeasuring Impact: How Randomized Controlled Trials Threaten the Nonprofit Sector, we report on what three key groups of nonprofit sector professionals—nonprofit managers, program evaluators, and foundation program officers—think about the growing use of RCTs to evaluate nonprofit social programs.  In interviews with these key nonprofit stakeholders, we found a surprising degree of concern regarding this development and use this data to identify five problems with relying on RCTs for nonprofit evaluation. Foundation program officers were particularly worried about the consequences of widespread RCT use in the nonprofit sector. In what follows, we discuss how and why RCTs have grown in stature in the sector, and what the program officers we interviewed saw as some of the main challenges.

RCTs have a long history as a tool for evaluating social programs, both in the United States and internationally. Many people concerned with making sure social programs are improving people’s lives have embraced the idea of a “hierarchy of evidence.” This formulation suggests that there are better and worse types of evidence, and that the RCT naturally sits atop the hierarchy. But our reading of the history suggests that the RCT did not find its place at the top of the evidence hierarchy simply on its own merits. Rather, it took significant work for this outcome to be achieved—efforts we refer to as the “Gold Standard movement.”

Read the full article about the limits of randomized controlled trials by Nicole P. Marwell and Jennifer E. Mosley at HistPhil.