You’re designing an impact evaluation. You’ve identified
your research question, chosen your outcome measures, and decided on a randomized controlled trial (RCT)
design. Now comes a critical question that many organizations get wrong:

How many people do you need in your study?

This is where power analysis comes in—one of the most important (and most misunderstood)
concepts in impact evaluation.

What Is Statistical Power?

In simple terms, statistical power is the probability that your study will detect an
effect if one actually exists.

Think of it like a metal detector:

• A high-powered detector can find small coins buried deep underground
• A low-powered detector will only find large metal objects near the surface
• With a weak detector, you might walk right over buried treasure and never know it was there

In evaluation, power determines whether you’ll be able to detect your program’s true
impact. A well-powered study can detect small but meaningful effects. An underpowered study might miss
real impacts entirely.

The Four Key Concepts

Every power analysis involves four interrelated components:

These four are mathematically linked. If you know three, you can calculate the fourth.

Why Sample Size Matters: A Real Example

Let’s say you’re evaluating a girls’ education program. You believe the program increases secondary
school enrollment by 10 percentage points (from 60% to 70%).

The difference? Sample size. Too small, and you’re flying blind.

The Consequences of Being Underpowered

When studies are underpowered, several bad things happen:

1. False Negatives (Type II Errors)

Your program actually works, but your evaluation fails to detect it. You conclude the program is
ineffective and shut it down, wasting a potentially valuable intervention.

2. Wasted Resources

You spent money, time, and effort implementing an evaluation that was doomed from the start. All that
investment in data collection, analysis, and reporting yields inconclusive results.

3. Publication Bias

Journals and funders favor statistically significant results. Underpowered studies that show “no effect”
are less likely to be published, even if they were conducted rigorously.

4. Incorrect Conclusions

Sometimes, underpowered studies do find statistically significant results—but these are often false
positives or wildly inflated effect sizes. This misleads future program designers.

Conducting a Power Analysis: Step-by-Step

Here’s how to do a power analysis for your evaluation:

Practical Example: Sample Size Calculation

Adjustments for Real-World Factors

Common Power Analysis Mistakes

What If You Can’t Afford the Required Sample?

Power analysis might reveal that you need 2,000 participants, but you can only afford 500. What now?

Conclusion: Don’t Skip the Power Analysis

Power analysis is not an optional luxury—it’s a fundamental requirement for any rigorous impact
evaluation. Skipping it is like building a house without checking if the foundation can support the
structure.

Remember: An underpowered evaluation is
worse than no evaluation at all. It wastes resources and generates misleading conclusions.

About the Author

Aubrey Jolex is the founder of AJ Impact Evaluation Consulting, specializing in rigorous impact
evaluation for development programs. With 7+ years of experience with leading research organizations
such as IPA, IFPRI and IITA, Aubrey has designed and powered a number of impact evaluations across
multiple countries.

← Back to Blog | Home

Randomized controlled trials (RCTs) are the gold
standard for impact evaluation—when done correctly. But we’ve reviewed dozens of RCT
designs over our years in the field, and we see the same mistakes repeatedly.

These aren’t minor technical issues. They’re fundamental flaws that can invalidate your entire
evaluation, waste resources, and lead to incorrect conclusions about program effectiveness.

Mistake #1: Insufficient Statistical Power

The Problem: You design an RCT with too small a sample to detect meaningful program
effects.

Mistake #2: Poor Randomization Implementation

The Problem: Randomization is compromised by field staff or logistical errors,
undermining the entire design.

Mistake #3: Measuring Outcomes Too Soon (Or
Late)

The Problem: You collect data before effects materialize or after they’ve faded.

Rule of Thumb: Map your theory of change carefully.
Education often needs 1-2 years; health might need 6-24 months depending on the outcome.

Mistake #4: Ignoring Implementation Fidelity

The Problem: You assume the program was implemented as designed, but it wasn’t. A
“null result” might just mean the program never actually happened.

Mistake #5: Multiple Testing Without
Correction

The Problem: Testing 30 outcomes and reporting only the 2 “significant” ones. This
is statistical cherry-picking.

Bonus: Skipping Pre-Registration

Pre-registration prevents “p-hacking” and “outcome switching.” Always register your study design and
analysis plan on the AEA RCT Registry or OSF before collecting endline data.

Conclusion

Great RCT design requires care, technical knowledge, and commitment to rigor. Avoid these five
mistakes:

About the Author

Aubrey Jolex has designed and implemented dozens of RCTs across Asia and Africa with 7+ years of
experience at IFPRI. Learn from real-world experience—avoid costly mistakes in your
evaluation.

← Back to Blog | Home

Every year, billions of dollars flow into
international
development programs. Organizations are driven by a genuine desire to create positive change. Yet a
critical question often goes unanswered: Are these programs actually working?

This is where rigorous impact evaluation comes in—and why it matters more than ever.

The Problem: Good Intentions Aren’t Enough

Development practitioners work hard. But here’s the hard truth: activity doesn’t equal
impact.

Without rigorous evaluation, we’re operating in the dark. We might be wasting resources on programs
that
don’t work, or worse, cause harm.

What Makes Evaluation “Rigorous”?

A rigorous impact evaluation answers a specific causal question: Did this program cause the
observed outcomes?

The gold standard is the Randomized Controlled Trial (RCT). When that’s not
feasible,
Quasi-Experimental Designs (QEDs) like RDD or DID can provide credible evidence.

Common Evaluation Pitfalls

Why Rigorous Evaluation Matters

1. Learn What Works: Discover the truth about your
   program’s effectiveness.
2. Stop Wasting Money: Cut losses on ineffective
   programs
   and double down on what works.
3. Improve Programs: Learn which components drive
   impact
   and for whom.
4. Build Credibility: Funders demand evidence.
   Rigorous
   proof gives you a competitive edge.
5. Contribute to Knowledge: Help the global
   development
   community learn.

Common Objections (And Why They’re Wrong)

Conclusion: Evidence Matters

Rigorous impact evaluation isn’t a luxury. It’s a moral imperative. We owe it to the people we serve
to
ensure our programs actually improve their lives.

About the Author

Aubrey Jolex is the founder of AJ Impact Evaluation Consulting, specializing in rigorous impact
evaluation for development programs. With 7+ years of experience at IFPRI and 6+ peer-reviewed
publications, Aubrey helps organizations generate credible evidence of program impact.

← Back to Blog | Home

Randomized controlled trials (RCTs) are the gold standard
for impact evaluation. But let’s be honest: sometimes an RCT just isn’t feasible.

Maybe your program is already running, political constraints prevent withholding services, or you’re
evaluating a national policy. Does this mean you can’t conduct rigorous impact evaluation?
No.

Enter quasi-experimental designs (QEDs)—methods that approximate experimental conditions
without random assignment. When implemented carefully, QEDs can provide credible causal evidence.

When to Use Quasi-Experimental Designs

Four Common Quasi-Experimental Designs

1. Regression Discontinuity Design (RDD)

Best for: Programs with strict eligibility cutoffs (e.g., test scores, income
thresholds).

RDD compares people just above vs. just below the cutoff. If the cutoff is arbitrary, people on either
side are likely very similar, making the difference in outcomes attributable to the program.

2. Difference-in-Differences (DID)

Best for: Policy changes where you have pre- and post-data for treatment and comparison
groups.

DID compares the change over time in the treatment group vs. the change in the
comparison group. This removes time-invariant differences and common trends.

3. Propensity Score Matching (PSM)

Best for: When you have rich data on all factors influencing participation.

PSM creates a comparison group by matching each treated person with a non-participant who has similar
characteristics (age, education, motivation, etc.).

4. Instrumental Variables (IV)

Best for: When you have a variable (instrument) that affects treatment but not outcomes
directly.

IV uses “natural experiments” (like distance to a clinic or lottery numbers) to isolate causal effects.

Which Design Should You Use?

Strengthening Your Design

Since QEDs rely on assumptions, you need to work harder to prove your results are robust:

• Combine Methods: Use DID + PSM together for stronger evidence.
• Falsification Tests: Test for effects on “placebo” outcomes that shouldn’t change.
• Sensitivity Analysis: Show that results hold under different assumptions.
• Transparency: Be honest about limitations and assumptions.

Conclusion

Quasi-experimental designs offer rigorous causal inference when RCTs aren’t feasible. But they’re not a
free lunch—they require strong assumptions that must be justified and tested.

About the Author

Aubrey Jolex has designed and implemented both experimental and quasi-experimental evaluations across
multiple countries. Let’s find the right approach for your context.

← Back to Blog | Home

Randomized controlled trials (RCTs) are the gold
standard for measuring program impact—but what makes them so powerful? And how do you actually
design and implement one correctly?

Whether you’re evaluating a health intervention, education program, or poverty reduction strategy,
understanding RCTs is essential for rigorous impact evaluation.

What is an RCT?

An RCT is an experimental design where participants are randomly assigned to either a treatment group
(receives the program) or a control group (does not receive the program).

The Key Principle: Randomization

Random assignment ensures that, on average, treatment and control groups are identical except for the
program itself. This means any difference in outcomes can be attributed to the program—not to
pre-existing differences between groups.

The Anatomy of an RCT

Types of Randomization

Implementing Randomization with Integrity

Common Threats to Validity

Best Practices for Clean Randomization

Power Analysis: Getting Sample Size Right

A critical step in RCT design is determining how many participants you need. Too few, and you won’t
be able to detect program effects. Too many wastes resources.

Common RCT Design Challenges

Conclusion

RCTs provide the most credible evidence of program impact when done correctly. Key principles:

About the Author

Aubrey Jolex has designed and implemented dozens of RCTs across Asia and Africa with 7+ years of
experience at IFPRI. Learn from real-world experience to implement rigorous evaluations.

← Back to Blog | Home