How to 'pilot test' your program the right way

The term ‘pilot’ can be overused and misused in the development field. It is often wrongly employed to refer to any small or early stage activities or projects — regardless of whether they have the capacity, potential or intention of operating at a larger scale. The lack of a clear, formal definition is problematic for many seasoned program managers, whose interpretation of the word is tied to scalability and eventual large-scale implementation. A pilot should always anticipate what it would look like if it were being delivered at scale, says Larry Cooley, president emeritus of Management Systems International. And planning for scale influences everything — from strategy and project design to the way you structure incentives and engage third parties. “I think a pilot really is just trying to look at a proof of concept,” Cory Heyman, chief program officer of Room to Read, told Devex. “I think that what makes a pilot different is the intention to learn something, and the intention to both prove the concept and to improve the model in a formal way.” There are now more donors and funders than ever, but the average pilot project size and duration have both shrunk. According to Cooley, this has contributed to the increasing number of small, one-off projects that never make the transition to full or large-scale implementation. But how can program implementers avoid falling into the trap of “pilots going nowhere?” Devex spoke with a few experts and experienced professionals to compile some vital tips. 1. Always begin with the end in mind. There are many one-time projects or activities that do add value in development, but these are not to be confused with pilots. To differentiate pilots from the other activities, some development workers prefer calling them early stage implementation programs. Naty Silorio, Plan International Philippines’ area manager for Visayas, listed viability, sustainability and replicability as a pilot’s main objectives. A pilot is not an end in itself, but a process — a helpful tool used to refine or further improve a product or service prototype. In order to help ensure that your activity does eventually scale successfully, you must be thinking about the finish line from day one. Cooley recommends imagining what a fully implemented version of your pilot would look like, and working backwards to determine what has to be done for that to take place. A pilot needs a plan, and that plan has to be anchored on your end goal. “Even if you don’t have the funds for a full scale long-term program, you should be building into the design of that program from the very beginning, the basic building blocks you need for scale,” Kirsten Gagnaire, executive director of Mobile Alliance for Maternal Action, told Devex. 2. Have a clearly defined model to test. In order for your pilot to serve its purpose — which is to provide insight and lessons toward improvement of the program — it must be based on a well-researched and clearly defined model, which is comprehensive and contextualized to the local setting. Without a clear design, you will not be able to accurately and correctly evaluate your program and the success of your pilot. While the components of a model will vary between different programs, funding is always an important part of any project design. Gagnaire suggests identifying the stakeholders who would have a financial interest in ensuring that your program works at scale. Consider traditional donors, the public sector and the private sector as well. Think about whether or not people might be willing to pay for your product or service, and what this might mean for your program’s impact and financial sustainability. “I think the whole idea is just understanding all the different pieces of infrastructure … what you will need to do when you’re reaching 500 people, 5,000 people, 50,000 people — and making sure that you’re putting that plan in place,” she said. 3. Get buy-in from all stakeholders and implement with high fidelity. For a pilot program to be successful, it needs the cooperation of government and local partners, as well as the readiness of the target communities or beneficiaries, says Manuel Madamba, Plan International Philippines’ country director for research, evaluation and dissemination. Room to Read’s Heyman offered similar advice. “There needs to be permission to try the model,” said Heyman. “And that’s something that we take for granted but something that’s very important because that also is part of the buy-in at the very early stages.” Support from everyone involved also makes it easier to provide the training necessary to implement the program with high fidelity, with its central methods and strategies intact. While a certain degree of flexibility is advised, staying true to the model or design of a pilot and ensuring it has been implemented as intended is fundamental to determining the real effectiveness of a program, as well as coming up with an accurate evaluation of its success. Before launching, determine which aspects of your program design need to adhere closely to your model and which ones can be modified as your pilot moves along. “If the model is implemented with low fidelity, then it’s always a question if it doesn’t work, whether it’s the fidelity or the nature of the program itself,” said Heyman. 4. Constantly collect information. Since a pilot seeks to provide insight and lessons for the improvement of a particular program, it is imperative that the right information is collected at every step of the way. A systematic data collection and monitoring system will help ensure that implementers do not miss any important details or issues, making it easier to draw out lessons in post-program review. In addition to airtight documentation, pilot programs benefit greatly from open communication lines between the implementers, governing bodies, stakeholders and beneficiaries. Feedback from everyone involved is an important knowledge resource that will not only influence later recommendations for improvement of the program, but it will also allow implementers to make real-time tweaks and modifications. “I think it’s all about having a continuous information loop and knowing what the critical components are that need to work, getting the data on those components and being able to make adjustments,” said Gagnaire. A successful pilot is always a team effort, so be sure to share the results with all stakeholders. Aside from building partnerships founded on transparency, honesty and trust, stakeholders can provide insights and solutions you might have otherwise missed. Check out more insights and analysis for global development leaders like you, and sign up as an Executive Member to receive the information you need for your organization to thrive.