What is the science of scaling?

By Catherine Cheney 30 September 2016

Evan Thomas, director of the SWEETLab at Portland State University, at the event, “The Science of Scaling.” Photo by: Catherine Cheney / Devex

Anti-poverty innovations can make real progress on sustainable global development when they transition from startup to scaleup. But scaling is easier said than done, particularly when it comes to technology transfer from universities to end users.

“Academics are notoriously bad at spinning out ideas from the ivory tower into the real world,” Temina Madon, executive director of the Center for Effective Global Action, told Devex at “The Science of Scaling.” Scaling was the theme of the 2016 State of the Science conference hosted by Development Impact Lab, which is funded by the U.S. Agency for International Development and headquartered at the University of California, Berkeley. “At the same time a lot of academics are trying to build institutions for spin out and scale up. USAID and other donors are constantly talking with us about scaling scaling scaling without a sense of the barriers and constraints we face. So we wanted to start unpacking what the pressure to scale means for us.”

On Monday, researchers, investors, technologists and practitioners gathered at UC Berkeley to answer big questions, including: “Why do some products and interventions scale quicker than others?” Here are some of the top takeaways.

1. Don’t plan. Iterate.

If your goal is to scale, it’s important to iterate rather than plan, said Ofir Reich, data scientist at CEGA, whose presentation focused on what lessons agile product development hold for effective global development.

The audience laughed at one of his slides, which read: “Being off course and wasting two weeks is :(, but still better than being off course and wasting six months.”

Pilot with the real thing, so that early users will give you feedback to improve the product, and fail early and often, he said. Ship out minimum viable products, which can be scaleable but not yet scale ready, and build in tight feedback loops, talking with real users from day one. Then conduct research on your pilot to inform the requirements for the scale up process, he said.

2. Manage expectations.

“One of the challenges with scaling is that we, the development community, simultaneously emphasize that an intervention needs to be ‘locally appropriate,’ ‘innovative’ and also scalable,’” Evan Thomas, director of the Sustainable Water, Energy and Environmental Technologies Laboratory, or SWEETLab, at Portland State University, told Devex. “Those first two characteristics, while not overtly in opposition to the third, are certainly in tension with it. Almost by definition, something that is both locally appropriate and innovative is not yet ready for scale.”

And yet, when donors put out a request for proposal, they expect a product to work out of the box. They are not necessarily attuned to the iterative cycle of design, so there is a need for expectation management, which can be a challenge, Thomas said.

3. Change the incentives.

One of the main talking points of the Development Impact Lab meeting was how the status quo — in which money comes from funders to support interventions — incentivizes new projects versus the sustained delivery of services.

“There really are misaligned expectations and structures between what it takes to make a technology product viable and what our customers in this context think it takes,” Thomas said at the event Monday.

The question, then, is how to monitor and sustain these interventions over time and not just over the course of the grant periods. New technology tools, such as cellular sensors, combined with new feedback tools, such as carbon credits, could lead the global development community to monitor and monetize improved outcomes, and therefore change incentives.

4. Monitor the outcomes that matter.

The event Monday uncovered some of the fundamental limitations to randomized control trials, or RCTs, from the high price tags to the challenge of generating transferable insights on what works in other settings.

Kenya Resilient Arid Lands Partnership for Integrated Development, or Kenya RAPID, is setting a new standard in its model to improve access to water and sanitation in five northern counties in Kenya. In addition to its unique public-private partnership model, RAPID represents the first USAID project funding the rehabilitation of water services, building sensors in as part of that project, with plans to respond to that data.

This is just one example of how technologies are providing the global development community with alternatives to RCTs to monitor outcomes that matter.

5. Remember that scale requires systems.

One of the major barriers standing in the way of scale is “the attribution versus contribution problem,” said Louis Boorstin, managing director of the Osprey Foundation in Washington, D.C. When donors seek to attribute outcomes to their own funding, in order to please those who authorize their budgets, they may not prioritize activities that contribute to the overall sustainability of systems, he explained. “That can drive crappy, if unintended, behaviors by implementers to deliver services more directly instead of engaging in the messy business of building the long-term capacity of the local systems to deliver those services,” he said at the event Monday.

This comes back to the problem of incentives, and if funders like USAID were to put in metrics for systems change in addition to, say, the numbers of toilets installed for the amount of money granted, that would change the game.

The Abdul Latif Jameel Poverty Action Lab, better known as J-PAL, launched the Government Partnership Initiative last year to enable the research center — which is based at Massachusetts Institute for Technology — to work with governments to scale up programs that have demonstrated their effectiveness in poverty reduction, said Claire Walsh, policy manager at J-PAL. Current partnerships include one with the Ministry of Education in Peru and another with the Tamil Nadu Innovation Initiative in India. The key ingredients for the success of these partnerships are being flexible and ready to respond to policy windows, becoming “part of the woodwork,” and sharing data and analysis throughout the partnership, not just a final report, she said.

6. Consider the community.

FloodRISE — a research project led by the University of California, Irvine, to promote flood resilience in Southern California — offers transferrable lessons for others looking to increase flood resilience in communities around the world. The team is working to combine drone imagery and citizen science at scale, given that this combination provides far better resolution than satellite imagery, but is time consuming and expensive, said Richard Matthew, professor of planning, policy, and design and political science at UC Irvine.

Projects like these demonstrate the need to develop scalable solutions for community engagement alongside big data tools in global development work, said Bessie Schwarz, cofounder of Cloud to Street, which is focused on dynamic flood vulnerability mapping.

“What you can do with these incredibly nimble and scalable tools only provides half the picture,” Schwarz told Devex. “Working with communities to understand the nuances of climate vulnerability is not scaled at the same speed but we have to prioritize that ground level information as much as the quick data we can detect from the sky.”

7. Test at scale.

When Boorstin of the Osprey Foundation was young, he took his bicycle apart then built it back up to the point where it rode. But just because he could assemble a bike in his garage, doesn’t mean he knows how to run a bicycle factory, Boorstin said, expanding on why he believes in working at scale, but not scaling up.

The recent wave of RCTs in global development has provoked criticisms regarding external validity, or whether the results of the study can be applied to other contexts, said Karthik Muralidharan, an associate professor at University of California, San Diego. But scale can improve external validity as long as researchers answer key questions about representativeness — whether the sample represents a policy relevant population scale — as well as scale of implementation and scale of unit of randomization.

Muralidharan presented on the key enabling factors for at-scale evaluations. They include deep and long-term engagement with governments, domain expertise in the area of intervention, and the use of examples from other settings to demonstrate feasibility and value.

“Show you care about the public interest and not just an academic paper!” he told the room of mostly academics, as well as representatives from Omidyar Network, Google.org, and Mercy Corps, to name a few organizations.

Still, it’s important to do pilot research even when you have a government ready to test at scale, Walsh of J-PAL said, explaining that a few thousand dollars can save several hundreds of thousands of dollars that otherwise might have been spent on an RCT.

8. Pressure test interventions.

The two flagship programs for Evidence Action, an organization with a mission to scale proven anti-poverty initiatives, are Dispensers for Safe Water and Deworm the World Initiative. But the organization has a new program called Evidence Action Beta, which is pressure testing whether promising interventions might be ready for massive scale up. This is similar to ways that companies might beta test for software, but in this case, the goal is to improve the lives of millions.

The keys for identifying promising interventions are rigorous evidence, low cost per beneficiary, big impact, and a widespread issue, said Guillaume Kroll, project manager at Evidence Action. And the keys for path to scale are scalable program design, implementation capacity, conducive policy environment, and sustainable funding plan, he continued.

One of the interventions they are pressure testing is a cash-based incentive for migration to urban areas of Bangladesh during the lean season, the time between planting and harvest. After the inputs (subsidies are offered), the pressure testing happens in the transition from outputs (subsidies accepted) to outcomes (migration induced) to impact (earnings and consumption increased), Kroll explained. In the first year, the focus was on program design, with 9,000 users reached, and next year the plan is an RCT at scale with 40,000 people. If, and only if that works, the plan would be scaled to implementation, up to 165,000 people.

Like scaling, key takeaways from the event  — from piloting with the real thing to responding to feedback quickly  —  are easier said than done, Matthew of UC Irvine said.

“Encouraging the development community — especially its donors — to experiment and accept failure is excellent advice, but politically very challenging,” he said.

Read more international development news online, and subscribe to The Development Newswire to receive the latest from the world’s leading donors and decision-makers — emailed to you FREE every business day.

About the author

Catherine cheney devex
Catherine Cheneycatherinecheney

Catherine Cheney covers the West Coast global development community for Devex. Since graduating from Yale University, where she earned bachelor's and master's degrees in political science, Catherine has worked as a reporter and editor for a range of publications including World Politics Review, POLITICO, and NationSwell, a media company and membership network she helped to build. She is also an ambassador for the Solutions Journalism Network and the Franklin Project at the Aspen Institute.


Join the Discussion