Why Randomized Controlled Trials?
The Gold Standard for Causal Evidence
Randomized Controlled Trials (RCTs) are widely considered the most rigorous method for measuring program
impact. But why? And when are they worth the investment?
The Core Problem: Correlation ≠ Causation
Scenario: Your education program serves 500 students. After one year, test scores improve
by 15 points.
Question: Did your program cause the improvement?
Alternative explanations:
– Students would have improved anyway (maturation)
– A new government policy changed curricula
– Motivated families self-selected into your program
– Economic growth improved nutrition, enabling learning
Bottom line: Without a comparison group, you can’t isolate your program’s effect.
How RCTs Solve This
The RCT Approach
- Identify eligible participants (e.g., 1,000 students)
- Randomly assign 500 to treatment, 500 to control
- Implement program for treatment group only
- Measure outcomes for both groups
- Compare: Treatment effect = (Treatment outcome) – (Control outcome)
Why Randomization Matters
Before randomization: Treatment and control groups are statistically identical on
all characteristics (observed and unobserved).
After randomization: Any difference in outcomes can be attributed to the program (plus
random noise).
Magic: Randomization eliminates selection bias—the #1 threat to causal inference.
When RCTs Are Worth It
Consider an RCT When:
-
Genuine Uncertainty Exists
– Program is new or unproven
– Stakeholders legitimately unsure if it works
– High stakes (scaling could affect millions) -
Randomization is Feasible
– Can assign treatment before rollout
– Enough units to randomize (100+)
– Control group won’t receive similar services elsewhere -
Results Will Inform Decisions
– Funders will scale if evidence is positive
– Or pivot/stop if evidence is negative
– Academic publication could influence field -
Budget Allows Rigor
– Typically $20K-$100K+ depending on scope
– Sufficient sample size for adequate power
– Timeline allows 12-24+ months
When RCTs Aren’t Right
Skip RCTs When:
-
Randomization is Unethical
– Denying treatment to control violates rights
– Program is clearly beneficial (e.g., clean water)
– Vulnerable populations without appropriate protections -
Randomization is Infeasible
– Program already rolled out universally
– Sample size too small (<50 units)
– Political constraints prevent randomization -
Research Question Doesn’t Need RCT
– Process questions (“How is it implemented?”)
– Mechanism questions (“Why does it work?”)
– Heterogeneity beyond effect size -
Better Alternatives Exist
– Strong quasi-experimental design possible (RDD, DID)
– Existing rigorous evidence from similar contexts
– Budget better spent on program improvement
Common Myths About RCTs
Myth 1: “RCTs are too expensive”
Reality: Costs vary widely ($20K-$500K+). Smart design can reduce costs:
– Focus on administrative data (cheaper than surveys)
– Cluster randomization (fewer units to track)
– Partner with government for data collection
Counter-question: What’s the cost of scaling an ineffective program?
Myth 2: “RCTs are unethical”
Reality: Ethical RCTs are common when:
– Phased rollout: Everyone gets treatment, just at different times
– Oversubscription: More applicants than spots? Lottery is fair.
– Equipoise: Genuine uncertainty means neither group is clearly harmed
Unethical: Launching untested programs at scale without learning if they work.
Myth 3: “We can’t randomize in our context”
Reality: RCTs have been successfully conducted in:
– Active conflict zones (DRC, Afghanistan)
– Remote rural areas (Sub-Saharan Africa)
– Urban slums (Kenya, India)
– Refugee camps (Jordan, Uganda)
Creative solutions:
– Cluster randomization (villages, not individuals)
– Encouragement designs (randomize offer, not actual take-up)
– Stepped-wedge designs (everyone gets treatment eventually)
Myth 4: “RCTs can’t capture complexity”
Reality: RCTs measure average treatment effects, but can also:
– Test mechanisms through mediation analysis
– Examine heterogeneity (who benefits most?)
– Combine with qualitative research for “why”
– Use multiple treatment arms to compare approaches
What RCTs don’t do: Explain every individual’s experience. But that’s not the point.
Next Steps