Impact evaluations to guide policy

Asian countries are increasingly turning to investing in dedicated development programs rather than relying entirely on economic growth to deliver better social outcomes. Evaluations of their actual impact have not always accompanied such decision-making, but where they have, it has made a key difference.

Impact evaluations can guide policy and investments based on evidence instead of them being driven mostly by special interests.

Evaluations and policy choices intersect in three respects. First, conducting assessments of policies or investments — like a social protection initiative — allows them to be expanded on the basis of the results obtained. Second, they bring out the crucial complementary factors that are necessary for development success — for example, road projects can improve inclusion if they are linked with programs addressing education and health care in the same area. Third, there are emerging avenues of action such as climate change, where past experience may not provide a sufficient guide for the future.

Careful evaluations in some social areas have given policymakers grounds to expand or wind down a program. Assessments of conditional cash transfers have provided a basis to expand these programs in Brazil, Mexico, the Philippines and elsewhere.

An impact evaluation of the Philippine government’s CCT program, which started with 6,000 households in 2008, showed gains in reducing stunting in children, increases in school enrollment and attendance and increased ante- and postnatal care for women — well worth the program costs. As a result, the program was massively expanded to cover more than 3 million households in 2013.

Impact evaluation initially focused on social programs, where information on beneficiaries and others before and after a program, is relatively easier to obtain. But it is now expanding into other areas of policy, such as transport, energy and the environment. All these areas present challenges for analysis, such as separating cause and effect, attribution, and assessing the validity of the outcomes across various locations, beneficiary groups or policy instruments.

Evaluations have often also highlighted the importance of complementary factors. In many instances, while the intervention did produce results, the gains were limited and full impact was not achieved. Impact evaluations help identify the barriers to achieving better outcomes, or the inadequacies in program design.

One consistent theme has been the limits of poverty-oriented efforts to reach the very poor. An impact evaluation of microfinance in Pakistan and Vietnam found that there were trade-offs between program impact and targeting of the poor. Pakistan’s program was able to target and include the poor, but the program had very little impact on their welfare. On the other hand, Vietnam’s mostly nonpoor clients enjoyed positive direct impact on income and enterprise employment.

A study on Bangladesh pointed to the gains from rural roads maintenance, but again the benefits missed the very poor. In these and other instances, the presence or investment in related areas such as education, health and financial literacy proved critical. Rural electrification will not only light communities, but can transform lives if education and income opportunities are promoted.

Among emerging issues, although biodiversity and the environment have received relatively little attention from impact evaluation, the results have been noteworthy when these issues have been considered.

An evaluation of forest protection worldwide found that establishing protected areas is effective in reducing the loss of forest cover. However, the impact of protected regimes — rates of deforestation before and after intervention — were sharper in areas closer to communities and economic activities than in settings far from economic activities.

Studies in Latin America found a strong correlation between forest protection and indigenous land tenure and property rights. While protecting pristine forests is equally important, protection of land competing with other economic uses can have relatively high payoffs too.

The application of lessons to new challenges remains difficult as there are limits to what can be inferred from historic data. Using an impact evaluation based on past experience to guide future action presents a tough challenge when the problems are new. Climate change is a case in point where climate models yield various projections with varying probabilities.

These are just some of the challenges and possibilities of applying impact evaluation to guide policy. The growing body of evaluative work directed at policy will be reviewed at an international conference co-sponsored by 3iE, the Philippine government and the Asian Development Bank on Sept. 5-6 at ADB’s headquarters in Manila.

Join the Devex community and access more in-depth analysis, breaking news and business advice — and a host of other services — on international development, humanitarian aid and global health.