Impact Evaluation: AKF’s MIAD Approach to Development in Badakhshan, Afghanistan, 2004-2014

Job Description

MULTI-INPUT AREA DEVELOPMENT-GLOBAL DEVELOPMENT ALLIANCE

(MIAD-GDA)

MONITORING, EVALUATION, RESEARCH AND LEARNING (MERL)

LEVEL 2 RESEARCH PROJECT, YEAR 3: A RETROSPECTIVE STUDY ON THE EVOLUTION AND IMPACT OF THE AKF-A MIAD APPROACH TO DEVELOPMENT IN BADAKHSHAN PROVINCE IN AFGHANISTAN, 2004-2014

TERMS OF REFERENCE

CORE OBJECTIVES OF STUDY:

• Identify and communicate the achievements in Badakshan over the last ten years by assessing the relevance, effectiveness, efficiency, impact, and sustainability of AKF Afghanistan’s evolving MIAD approach in a post conflict scenario.
• Provide findings, conclusions and recommendations with respect to implementation of AKF Afghanistan’s evolving MIAD approach in Badakhshan in order to draw lessons for future design and implementation.

1. INTRODUCTION

MIAD-GDA is a five-year (2013-2018), $60M rural development project being implemented by AKDN in Badakhshan Province in northern Afghanistan. Initiated in March of 2013, the project is funded jointly and in equal shares by AKDN and USAID. Because of the U.S. source of external funding, AKF-USA figures as the official USAID cooperating partner for the project. In that capacity, AKF-USA is ultimately responsible for compliance to USAID and accountable to that agency for achieving the project’s aims in an effective and efficient manner. Operations are carried out on the ground by AKF-Afghanistan (AKF-A), Aga Khan Health Service (AKHS), and the University of Central Asia (UCA), and are coordinated with the Aga Khan Development Network (AKDN).

The overarching goal of the project is to support long-term stability and growth in Badakhshan, which will be achieved through two strategic objectives: (1) to improve the quality of life for residents and to improve social and economic development in the province, and (2) to establish a model for a sustainable, replicable financing facility for regional social and economic development. The first objective will be achieved through intensive integrated socio-economic development interventions in Badakhshan, which will focus on four sectors: health, education, rural livelihoods, and governance. The second objective will be achieved through the establishment during the first five years of the project of a financial entity to channel profit-making investments toward economic activities. A portion of the returns on those investments, which remains to be determined, will go to a trust fund that will finance socio-economic development interventions in Badakhshan into the future.

Because it is both experimental and innovative, the MIAD-GDA model will be extensively scrutinized during the life of the project. To achieve this, AKDN has conceived of a two-track data collection and knowledge management program known as MERL (Monitoring, Evaluation, Research, and Learning). Track 1 embraces program monitoring and evaluation, which entail on the one hand conventional mid- and end-term process and impact evaluations, and on the other, routine measurement of project outputs and outcomes against standard indicators. Track 2 comprises research and learning, which together are designed to understand more fully the outcomes attained; disseminate and process among wider audiences, inside and outside of AKDN, the findings of the research being conducted; share with other practitioners the lessons learned and practices achieved through the project; and ultimately test the central hypothesis of the project, namely that the concentration of effort and resources in a confined geographic area will lead to improved livelihoods and quality of life among the participants. MERL is governed by a working group (MWG) composed of representatives of AKF-Afghanistan, AKF-USA, AKF-Geneva, and the University of Central Asia.

The strategy for carrying out the project’s research agenda contemplates developing and conducting research at two levels:

Level 1 entails research that is designed to sharpen targeting and programming with a view to enhancing the project and increase its impact and sustainability. Level 1 research is managed by MERL Unit, the Monitoring, Evaluation, Research, and Learning Unit, at AKF-Afghanistan. [1]

Level 2 research under the MERL strategy is directly related to documenting and testing the validity of the MIAD-GDA approach and contributes to higher level organizational learning. In particular, it examines the extent that project activities are integrated within and between program sectors. Furthermore, Level 2 research studies the impact of interventions as measured by the highest-level indicators (impact and outcome), focusing especially on livelihood and quality-of-life improvement.[2]

All research is oriented toward validating the central hypothesis of the project[3] and is fueled by three key assumptions:

Assumption 1: Dynamic integration of program inputs will over time contribute directly to improved quality of life. If multi-sectoral interventions are targeted, coherent, mutually reinforcing, and effectively implemented, then participants will perceive a substantial and demonstrable improvement in their wellbeing.

Assumption 2: Program integration, well executed, will mean greater value for the money spent. Multi-sectoral approaches that are interlocking and mutually supportive will lead to economies of scale. Moreover these economies will increase over time as interventions gather momentum and approach sustainability.

Assumption 3: Sustainability is achieved through effective program interventions that generate income and create an enduring resource base. Income generation not only benefits participants but also grows the macro-economies of rural villages and sub-regions, which in turn benefits more families. When incomes improve, the potential for sustainability is increased. In like fashion, sustainability is strengthened through sound investment initiatives that generate funds for reinvestment in social programming, which is at the heart of the private investment component of the MIAD-GDA project.

1. RESEARCH TO BE PERFORMED

Background

The Aga Khan Development Network (AKDN) has been operating in Afghanistan since 1995 through its various agencies and affiliates including FOCUS Humanitarian Assistance (FOCUS), Aga Khan Foundation (AKF), Aga Khan Health Services (AKHS), Aga Khan Fund for Economic Development (AKFED), and the University of Central Asia (UCA), among others. The MIAD approach has been evolving over a period of the last ten years as learnings from the field and international theory have been integrated into programming. AKDN activities in Badakhshan province were first initiated by FOCUS in 1999 through relief programming, and subsequently evolved into longer-term, sustainable development interventions implemented by AKF-Afghanistan (AKF-A) beginning in 2004.

Between 2004-2010, AKF-A employed a Micro-Area Development approach which aimed to have an immediate and visible impact on target group vulnerability through diversified livelihood options and the re-establishment of trust and synergy between households, villages and external service structures. Beginning in 2010, AKF-A’s approach shifted to adopt Integrated Area Development strategies based on the experience that poverty is usually caused by multiple, interlinked factors, all of which need to be looked at in order to identify and overcome the key bottlenecks to the socio-economic development of a region. The right mix of interventions with the right sequencing within a defined geographical region can maximise synergies and impact of various sectoral interventions.

Around 2010 is also where the term Multi-Input Area Development (MIAD) was first employed in AKF-A terminology and strategic planning exercises. Today, several other agencies now work in collaboration with AKF-A to implement the MIAD approach in Badakhshan, including FOCUS, AKHS, and UCA, as well as Aga Khan Education Services (AKES) and the First Microfinance Bank (FMFB).

While AKF-A and its sister agencies have carefully monitored progress in various program sectors including health, education, natural resource management, market development, human institutional development (civil society), and infrastructure, that monitoring has largely been done at the activity/output level in coordination with first-time impact level assessments measuring Quality of Life (QoL) in 2010 and Area Development in 2011. However, there has not been an integrated investigation of the entire MIAD experience in Badakhshan, to date.

Furthermore, the last decade has seen an evolution of AKF-A’s MIAD operations in the province, an evolution both in programming as well as in the administrative and support structures that shore up those operations in a post-conflict scenario.

Research

AKF-USA is seeking proposals from interested and qualified researchers (individual or institutional) to undertake a thorough and comprehensive study of the development, evolution, and socioeconomic impact of AKF-A’s MIAD approach to development in Badakhshan Province in Afghanistan. The study will cover a ten-year period beginning in 2004, will scrutinize all areas where MIAD approaches have been applied, and will focus on programmatic issues and the administrative, financial, and logistical support structures created to support them, analyzing the extent to which back-office support, as it has evolved over time, has helped or hindered program operations and effectiveness.

On the program side, the study will explore fully the issue of deliberate coordination of sectors to achieve impact through integration. Does integration exist? What does integration mean /require in this context? Is it effective? Is it critical or is it incidental in achieving the program goals of improved quality of life and livelihoods?

Program and back-office (understood to embrace administration, finance, and logistics) issues will each be studied within a framework of five key elements: relevance, effectiveness, efficiency, impact, and sustainability, defined as follows[4]:

1. Relevance: The extent to which the activities are suited to the priorities and policies of the target group, recipient and donor.

1. Effectiveness: A measure of the extent to which the activities attained their objectives. The researchers will establish parameters for measuring effectiveness and will employ that yardstick consistently throughout the study.

1. Efficiency: Efficiency measures the outputs - qualitative and quantitative - in relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted. The researcher(s) will establish clear guidelines for measuring efficiency and will apply that measure consistently in the study.

1. Impact: The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade, financial and security conditions.

1. Sustainability: Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding is withdrawn and can include aspects such as political support, cultural appropriateness, adequacy of technology, and institutional capacity as well as environmental and financial aspects.

These five elements will frame the analysis of the development, evolution, and impact of the MIAD approach during the period 2004-2014. The researchers will also pay attention to four cross-cutting issues:

1. Gender and equity issues, to determine whether gender-based and rights-based programming, both central to AKF’s core principles, is in fact a reality and is effective.

1. Integration of programming in the several sectors, whether it exists or not and whether it is deliberate or accidental. If possible, the researchers should seek to determine whether greater impact is achieved through integrated programming or single-sector programming.

1. Replicability of the MIAD “model” as it has evolved in Badakhshan. What elements of the Badakhshan experience would have to be reproduced (in other countries, regions, provinces) for replication to be successful?

1. Agency policy and how the achievements of MIAD over the years have influenced policy development and advocacy in AKF-A.

Finally, the study will conclude with a chapter on suggested future directions for AKF based on the findings of the in-depth study of MIAD in Badakhshan. What changes are recommended? What should stay the same? What should remain but in modified form? What are the implications of the study’s findings for MIAD-GDA now and in the future?

III. METHODOLOGY, SOURCES, AND TIMEFRAME

This study is conceived to be carried out by a team of researchers, led by a Team Leader, each with specialized knowledge in one or more of the key dimensions noted above including program and backoffice support. The timeframe for the study is 7 months beginning on 1 June 2015, and ending 31 December, 2015, with the delivery of a first draft of the report for review by AKF. The final draft will be due on 15 March 2016.

The research itself will be broken out into three steps including (1) documented synthesis of materials[5] provided by AKF-A and initial assessment in advance of fieldwork (1 June – 31 July 2015), (2) documented key findings after initial field and staff interviews in-country (2 August – 3 September 2015)[6], and (3) documented assessment after final round of staff interviews and meetings (3 – 30 September 2015)[7]. Breaking out steps 2 and 3 will ensure that interviewed staff have the opportunity to respond to key points in a transparent fashion.

AKF-A will assist in the coordination of all in-country activities, will facilitate access to key informants both in Kabul and in Badakhshan, and will make all necessary arrangements for lodging, transport, and security. Security issues are of paramount importance; the research team must be willing to alter plans on short notice and must commit to adhering to all aspects of AKF-A security policies.

1. DELIVERABLES

1. Research report (no page limit) (see Annex A for proposed content of final report)
2. Executive Summary of report – 15 pages

1. LEVEL OF EFFORT AND TIMEFRAMES

1. Preliminary document review, synthesis, and initial assessment: 1 June – 31 July 2015
2. Field work, key stakeholder interviews in Kabul and Badakhshan, and documented key findings presented: 2 August – 3 September 2015
3. Final key stakeholder interviews, findings developed, and exit briefing provided: 3-30 September 2015
4. First report draft developed and submitted: 1 October – 1 December 2015
5. Virtual meeting to discuss draft and comments: 15 December 2015
6. Rewriting, revisions, submission of final report: 1 January-15 March 2016
7. Executive summary submission of draft: 1 March 2016; final draft: 15 March 2016
8. A formal presentation to AKF of the research findings in both Washington DC and Kabul will be scheduled for the latter part of the month of March 2016.

Total proposed LOE is not to exceed 200 days[8]

1. PROPOSAL

Individuals or research institutions interested in participating in this research project must submit a technical and cost proposal by electronic mail by COB (EST) on 17 May 2015. Proposals may be submitted to Elizabeth Grant at elizabeth.grant@akdn.org.

1. Technical Proposal.

The technical proposal shall be no longer than 20 pages in length. It shall contain the following sections:

1. Introduction (1 page). Background, statement of purpose, succinct summary of proposal.

1. Statement of Work (5 pages). Detailed statement of work to be performed to fulfill the requirements of this Terms of Reference document. Must demonstrate full understanding of the issues to be addressed and the most efficient means of achieving that end.

III. Methodology (5 pages). Description of the proposed methods to be employed in achieving the goals of the ToR. Should mention in particular the composition of the research team and the mechanisms for coordination among team members throughout the research project.

1. Timeframe (3 pages). A detailed Gantt chart breakdown of the activities to be carried out for all components within the overall time limits outlined in the section “Level of Effort” above. Proposals should not include more than 200 individual days of LoE for entire team.

1. Proposed Team (3 pages). Names of proposed team members accompanied by a summary of credentials, professional qualifications, and a statement as to the role of each individual in the research project. Include as annexes the Curricula Vitae of proposed team members.

1. Capacity Statement (3 pages). Detailed description of background and capacity of the individuals or research institution to carry out work of this type. Should emphasize experience in designing and carrying out project/program evaluations and development research. Give examples of previous work that is relevant to this study. International experience (especially in Afghanistan and Central Asia region) should be highlighted.

1. Cost Proposal.

The technical proposal must be accompanied by a detailed cost proposal that covers all anticipated expenses for carrying out the research proposed, including honoraria, transportation, per diem, incidentals, and any administrative costs. A budget narrative outlining rationale for cost choices should be attached.

1. Writing Sample.

Please include a brief writing sample for each member of the proposed team.

Annex A: Expected Content of the Final Report

Executive Summary (15 pages)

Context of AKF-A’s MIAD approach in Afghan Badakhshan:

• Policy context, development agency and partner policies, objectives and strategies;
• Development context, including socio-economic, political and cultural factors;
• Institutional context and stakeholder involvement.

Intervention Logic

Validity and Reliability of Information Sources

Methodology

Framework of Analysis

Limitations

Ethical Considerations

Findings

Suggested Future Directions

[1] The MERLU Director gives final approval, on behalf of AKF-A, of all Level 1 research activities in consultation with the MIAD Program Manager and provides recommendations to the MIAD Program Manager for Level 2 research and approval.

[2] The MIAD Program Manager gives final approval, on behalf of AKF USA, of all Level 2 research initiatives in consultation with the ERLU Director and provides recommendations to the ERLU Director for Level 1 research and approval.

[3] Namely that integrated socio-economic interventions in a concentrated geographic area will lead to measurable improvements in livelihoods and quality of life.

[4] These five key elements are taken from OECD DAC’s criteria for evaluating development assistance. Further information on the DAC criteria can be found here: http://www.oecd.org/development/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm

[5] Materials to include, but not be limited to, reports from Quality of Life Surveys, Area Assessments, strategic plans, program plans, existing data sets, and evaluation reports found in agency archives.

[6] Field-based research will begin with preliminary discussions in Kabul with key stakeholders, followed by field visits to Badakhshan for observation, interviews, and discussions with key stakeholders. The documented key findings will then be shared with AKF for discussion and clarification on emerging patterns and trends prior final round interviews and meetings with staff.

[7] The research team will provide an exit briefing to key staff prior to departure and drafting of full report.

[8] 1 day is equivalent to 8 working hours

About the Organization

ABOUT AGA KHAN FOUNDATION U.S.A. (AKF USA)

Established in 1981, AKF USA is a private, non-denominational, non-profit international development organization committed to alleviating poverty, hunger, disease and illiteracy, primarily in Africa and Asia. Its mission is to strengthen the capacity of grassroots communities to solve their own problems and to promote opportunities that lead to long-term improvements in their income, health, education and the sustainability of local institutions and the environment. The Foundation supports community-based programs in the areas of health, education, rural development and civil society. Aga Khan Foundation is an agency of the Aga Khan Development Network (www.akdn.org), a group of 11 private development agencies working to empower communities and individuals, often in disadvantaged circumstances, to improve living conditions and opportunities, especially in Africa and Asia.