• News
    • Latest news
    • News search
    • Health
    • Finance
    • Food
    • Career news
    • Content series
    • Try Devex Pro
  • Jobs
    • Job search
    • Post a job
    • Employer search
    • CV Writing
    • Upcoming career events
    • Try Career Account
  • Funding
    • Funding search
    • Funding news
  • Talent
    • Candidate search
    • Devex Talent Solutions
  • Events
    • Upcoming and past events
    • Partner on an event
  • Post a job
  • About
      • About us
      • Membership
      • Newsletters
      • Advertising partnerships
      • Devex Talent Solutions
      • Contact us
Join DevexSign in
Join DevexSign in

News

  • Latest news
  • News search
  • Health
  • Finance
  • Food
  • Career news
  • Content series
  • Try Devex Pro

Jobs

  • Job search
  • Post a job
  • Employer search
  • CV Writing
  • Upcoming career events
  • Try Career Account

Funding

  • Funding search
  • Funding news

Talent

  • Candidate search
  • Devex Talent Solutions

Events

  • Upcoming and past events
  • Partner on an event
Post a job

About

  • About us
  • Membership
  • Newsletters
  • Advertising partnerships
  • Devex Talent Solutions
  • Contact us
  • My Devex
  • Update my profile % complete
  • Account & privacy settings
  • My saved jobs
  • Manage newsletters
  • Support
  • Sign out
Latest newsNews searchHealthFinanceFoodCareer newsContent seriesTry Devex Pro
    • News
    • News

    Don’t Do Impact Evaluations Because…

    <p>William Savedoff outlines common objections to doing impact evaluations.</p>

    By Devex Editor // 01 June 2011

    EDITOR’S NOTE: Over the years, several objections to preforming impact evaluations have emerged, according to William Savedoff, senior fellow at the Center for Global Development. Some say evaluations cost a lot, while others see them as a waste of money, believing their programs work.

    Recently, I was called for advice by someone who will be running a workshop attended by people who implement and evaluate programs. She asked me to help her anticipate the main objections raised against doing impact evaluations—evaluations that measure how much of an outcome can be attributed to a specific intervention–and to suggest possible responses. The FAQs on our Evaluation Gap site offer some guidance, but in answering her question, I realized that five particular objections come up over and over again. They are:

    Objection #1: “We already spend a lot on evaluation”

    This may be true. Most organizations do spend money on evaluations and many of these evaluations are very useful for improving operations and monitoring outputs. The problem is that most of those studies don’t measure impact. So an organization may know that it implements its programs well, but it doesn’t know if its programs have the desired outcomes. A second common problem is that even when impact evaluations are undertaken, their rigor and quality may be poor. When we reviewed impact evaluations back in 2004, it was depressingly common to find a paragraph in the methodological section explaining that conclusions were compromised by the lack of baseline data. So, yes, your organization spends on evaluation, but it may not be commissioning impact evaluations and it may not be getting good ones.

    Objection #2: “Impact evaluation methods can’t be applied in our field”

    Debates over the best methodology are interminable when they are argued in the abstract. The only time you can decide if an impact evaluation can or cannot be conducted is when you have identified a particular question in a specific situation. (Thanks to Stef Bertozzi for pointing this out to me years ago and to Michael Clemens who illustrates this point with respect to a specific prominent initiative, the Millennium Villages Project).

    Researchers have demonstrated considerable creativity in developing ways to study programs that address such diverse issues as corruption, teacher absenteeism, women’s empowerment and ethnic fragmentation. The first step is to identify the question. The second step is to figure out which method will give you the most convincing answer. For impact evaluations, that usually involves making a serious effort to compare outcomes from your program with some alternative – a counterfactual that you can either directly observe or plausibly construct.

    Objection #3: “Impact evaluations cost too much”

    Relative to what? The cost of an impact evaluation should be judged relative to the value of the information it will produce. So, a $2 million evaluation of a $500,000 program might be extremely cost-effective – a bargain – if the study helps policymakers decide whether or not to scale up into a billion dollar national program. This also means that impact evaluations should not be required of every program – rather they should be commissioned strategically to assess those programs that are unproven and are either widely used or are new and promising. Being selective in this way makes the overall budget for impact evaluation manageable relative to the overall budget for operations.

    Objection #4: “We know that our programs work so it would be a waste of money”

    If you really know that your program works, then it would be wasteful to conduct impact evaluations. We don’t need a study to know that feeding starving people will keep them alive. But most social programs aren’t this obvious. For years, job training programs in the U.S. were thought to be highly successful with good placement rates–until impact evaluations showed that placement rates were higher for comparable people who didn’t participate. Initiatives like conditional cash transfer programs that are widely supported today were initially the subject of considerable worry, that they would lead, for example to dependence, increased alcohol consumption or violence against women. Only because of good evaluations has it been possible to allay these fears and document the broad benefits. As with medical treatments, we need to have more humility about social programs and responsibly assess whether they are truly beneficial or cause harm relative to alternatives.

    Objection #5: “Impact evaluations don’t affect policy decisions”

    Evidence from impact evaluations is only one of many factors that influence policy decisions. If you think that the passage from evidence to action is linear and direct, you will almost always be disappointed. But the influence of impact evaluations is more complex than that. The questions they answer shape public debates over appropriate and effective policies. They also provide a base of information that is available when critical moments or opportunities arise to influence key choices. Rigorous impact evaluations have a much longer shelf-life than you’d think, while studies that are less rigorous or less conclusive quickly fade. Certainly more can be done to encourage the use of evidence in policymaking but this is a case where more and better supply can, I believe, have an indirect and ultimately decisive influence over the course of time.

    My last suggestion for the workshop?  Foster a collaborative atmosphere by posing problems as a shared challenge to which you are seeking solutions and be honest about your own weaknesses and failures. People usually respond to such admissions by feeling safer about sharing their own experiences, doubts and mistakes. After all, we’re human beings and that’s how we learn.

    Re-published with permission by the Center for Global Development. Visit the original article.

    • Project Management
    Printing articles to share with others is a breach of our terms and conditions and copyright policy. Please use the sharing options on the left side of the article. Devex Pro members may share up to 10 articles per month using the Pro share tool ( ).

    About the author

    • Devex Editor

      Devex Editor

      Thanks a lot for your interest in Devex News. To share news and views, story ideas and press releases, please email editor@devex.com. We look forward to hearing from you.

    Search for articles

    Related Jobs

    • Program Coordinator Lebanon
      Beirut, Lebanon | Lebanon | North Africa and Middle East
    • Project Assistant (Hybrid)
      Lausanne, Vaud, Switzerland | Vaud, Switzerland | Switzerland | Western Europe
    • Project Manager
      Brussels, Belgium | Belgium | Western Europe
    • See more

    Most Read

    • 1
      How low-emissions livestock are transforming dairy farming in Africa
    • 2
      Opinion: Mobile credit, savings, and insurance can drive financial health
    • 3
      Opinion: India’s bold leadership in turning the tide for TB
    • 4
      How AI-powered citizen science can be a catalyst for the SDGs
    • 5
      Strengthening health systems by measuring what really matters

    Trending

    Financing for Development Conference

    The Trump Effect

    Newsletters

    Related Stories

    Sponsored by AmgenOpinion: Are health systems really measuring what matters?

    Opinion: Are health systems really measuring what matters?

    The future of US AidMarco Rubio sails through nomination hearing for US secretary of state

    Marco Rubio sails through nomination hearing for US secretary of state

    #Globaldev SkillsHow to succeed as a monitoring, evaluation, and learning specialist

    How to succeed as a monitoring, evaluation, and learning specialist

    Devex Pro Live5 key questions about the US foreign aid halt, funding freeze

    5 key questions about the US foreign aid halt, funding freeze

    • News
    • Jobs
    • Funding
    • Talent
    • Events

    Devex is the media platform for the global development community.

    A social enterprise, we connect and inform over 1.3 million development, health, humanitarian, and sustainability professionals through news, business intelligence, and funding & career opportunities so you can do more good for more people. We invite you to join us.

    • About us
    • Membership
    • Newsletters
    • Advertising partnerships
    • Devex Talent Solutions
    • Post a job
    • Careers at Devex
    • Contact us
    © Copyright 2000 - 2025 Devex|User Agreement|Privacy Statement