Data before scale: Glasswing awaits Salvadoran behavior therapy results

SAN SALVADOR, El Salvador — Like many people in El Salvador, Jonathan is obsessed with soccer. He wants to be a professional player and has traveled abroad with his youth team. But his intensity — often a survival skill in difficult parts of his country — has gotten the 14-year-old in trouble on and off the soccer field.

“I was kicked off the soccer team for a month and [a] half because of a foul. I ripped up another player’s ankle,” said Jonathan, whose last name Devex is withholding to protect his privacy. “I would really have a quick temper. When they had the ball I would chase them just to be able to tackle them.”

Jonathan said he was also quick to anger off the field, frequently losing his patience at home with his siblings and parents, and with friends in the schoolyard. But over the past year, he said his behavior has shifted.

Jonathan was selected by Glasswing International, an NGO with programs focused on vulnerable communities across Latin America, to participate in a controlled trial of cognitive behavioral therapy, or CBT, as part of a club at his school.

“Once I came into this club, I learned the tools to take a few seconds to recognize your emotions,” Jonathan said, referencing the six-second model of emotional management. “Now I feel like I am in more control of my own thoughts and feelings and actions on the field.”

Reducing stress and aggressivity

Kids growing up in El Salvador are exposed to violence and gang activity from very young and must learn to navigate a life where something as simple as going to school can be dangerous. Glasswing aims to mitigate the impact growing up in such an environment has on a child’s ability to succeed through its program in select Salvadoran schools. The pilot measures the impact CBT and mindfulness exercises can have in helping children such as Jonathan identify and process their emotions.

Both interventions, which are done in the form of separate school clubs led by youth guides, began in 2017 and aim to reduce stress and aggressive and impulsive actions. The goal is to help students improve their ability to make decisions. Students in the CBT group meet once a week for 45 minutes, while mindfulness students begin and end their school days with 10-minute meditation sessions.

Staff at the school say they’ve noticed a tangible change in the way kids who have participated in the Glasswing pilot are behaving at school. They said some students are not as quick to anger, are visibly better at managing their emotions, and are more productive members of the school community.  

“I would scold Jonathan a lot. He was always hanging around outside,” said Roxmery Lopez, a coordinator at Jonathan’s school, the Complejo Educativo Walter Arturo Soundy. “Now, after the school day is over, he’s very responsible in cleaning up the classroom and making sure everything is OK. It’s clear that he’s changed.”

Still, anecdotes like this aren’t enough evidence that the program should be scaled as is, said Glasswing co-founder and Vice President of Programs Celina de Sola. Current participants for the two groups were randomly selected and are part of a controlled trial funded by the Tinker Foundation to determine whether employing CBT and mindfulness in Salvadoran schools makes a real difference in student’s lives and their academic outcomes.

“We want to wait to see what kind of impact we have before you invest or seek funding for something that you don’t have evidence it works. We’re being a lot stricter about that. We can’t scale without knowing,” de Sola said. “We’re looking at a few different indicators. Some of them have to do with their outcomes in the school, the others have to do with conduct, the others have to do with their relationships they have with their peers.”

An iterative approach

Several years ago, Glasswing contacted the Chicago Crime Lab to discuss adapting its “Becoming a Man” program for Latin America, which works with high-risk male youth in Chicago, Illinois. The crime lab shared information about how its youth programming had increased academic performance and resilience and decreased violence. Glasswing wanted to model a similar CBT program, but needed a curriculum that would be appropriate for both boys and girls in El Salvador.

“It’s easy for them to fall into crime and gangs and drugs. We are working to try and make sure that youth can have awareness about what can happen if they get into these criminalized activities,” said Jenny Geraldine Valle, one of the youth guides that leads the CBT club at Walter Arturo Soundy. “For them, it’s natural because this is where they grew up, so they accept it as something … normal.”

By the time Glasswing secured funding for the project, Chicago Crime Lab was scaling its own program in the United States, and didn’t have the resources to expand internationally. So Glasswing chose to adapt a curriculum from Catholic Relief Services, and in 2017, it rolled out CBT clubs to 300 students in three schools. As the programming progressed, the organization realized that it needed to take an iterative approach, and adapt the curriculum further.

The sessions in the original CBT program were too long, de Sola said, and some of the content didn’t make sense for the students’ realities. For mindfulness participants, Glasswing had to secure buy-in from schools to take students out of class twice a day.

“In the end, we’re ideally looking for interventions that are going to work — and that we can implement and other organizations can implement — but ideally that the government can implement because if you want to reach scale, you need to go through public sector,” de Sola said. “When we have positive results what we want to do is meet with the Ministry of Education and present the evidence ... and talk about it — ‘how can we roll something like this out?’”

Evaluation

Before doing that, Glasswing needs to receive the results from its evaluation study, which is being conducted by the Center for Research and Statistics at the Salvadoran Foundation for Economic and Social Development, or FUSADES.

“When you’re looking at the data you need to gather, the baseline and the endline data, it’s administrative data that you need to get from the schools,” de Sola said. “It’s not digital, it’s on paper, and sometimes the attendance isn’t taken. So you’re relying on a public institution that’s underresourced.”

The NGO must also determine how to measure students who missed sessions or stopped attending altogether. Some participants said that while some of their friends were initially skeptical of what they were doing in the Glasswing clubs, their curiosity eventually got the better of them. This led participants to share techniques they’d learned.

“They said we were going to psychologist’s office because we were crazy,” said Nathalie, 14, a participant in the mindfulness club. “But now they ask us what we’re doing and they’re curious about it.”

While this sharing of knowledge allows it to benefit more students than those who are formally enrolled, it creates a challenge in measuring impact, said Margarita Beneke de Sanfeliu. As director of the Center for Research and Statistics at FUSADES, she is devising survey questions to measure the success of the interventions, including ascertaining whether there was any spillover to students who didn’t participate in Glasswing’s CBT or mindfulness activities but may have learned techniques from someone who did.

De Sola is anxious to get results from the survey, due by the end of the year, or early 2019.

“It freaks you out … because at the end of the day, they’re going to see everything you do and you have to be ready to accept that you’re not having an impact,” de Sola said.

“We’re all trying to reduce violence and we’re not regionally getting to where we need to be as fast as we need to be there. So, if we don’t start really evaluating rigorously and studying and being more critical about what we do, we’re not going to change the context. We have to innovate, but innovate based on evidence.”