When two Chicago nonprofits and a team of academic researchers conducted a low-cost randomized, controlled trial on a youth-violence prevention program, they didn’t just gain solid data that proved it works. They also attracted the attention of government officials and earned more than $8-million in grants to further their work and research.
The program, Becoming A Man (BAM) -- Sports Edition, pairs in-school counseling and after-school sports training to teach conflict resolution and social and cognitive skills to male students. It’s run by Youth Guidance and World Sport Chicago, which conducted the trial with the University of Chicago Crime Lab, which uses science to address concerns about violence.
With the endorsement of Mayor Richard Daley, who agreed to apply the results in his policy agenda, the lab and the nonprofits raised money to operate the program for a school year.
By collaborating effectively and using randomization to their advantage, the nonprofits and researchers revealed the benefits of the program and leveraged the results to gain additional funding.
The process wasn’t always easy, though.
“We knew evaluation is important, and the idea of having an external evaluation by an institution with the prestige of the University of Chicago was appealing,” says Jason Eby, program director at World Sport Chicago. “I don’t know that going into it we knew it was going to be quite the rigor of a randomized, controlled trial.”
Randomization: Making the Most of Limited Resources
Running a randomized, controlled trial doesn’t have to be prohibitively expensive, says Jens Ludwig, director of the Crime Lab, despite widespread perception to the contrary, because randomization costs basically nothing. It’s collecting the baseline data -- information about existing conditions -- that adds up fast, because of the high cost of research labor.
To avoid this problem, the Crime Lab relied on existing government data, such as student-level education records from the Chicago Public Schools system and crime records from the Illinois State Police, to select a pool of at-risk young men for the trial’s control and participant groups.
“We did the research part of this on a shoestring -- on a ratty, old shoestring,” Mr. Ludwig says. “And some analysts’ time.”
Having limited financial resources can be a useful constraint because it provides a rationale for turning people away from the program -- and creating a control group -- without raising ethical questions, Mr. Ludwig says. Other eligible students from the subject pool went into the control group and still benefited from the usual services the city provides.
“Even if we raised $10-million, it wouldn’t be nearly enough to serve every kid who we think needs help or could benefit,” Mr. Ludwig says. “As long as there is excess need in the programming, there is some sort of need to decide which kids get and which kids don’t.”
Running the Trial: Collaboration and Compassion
During the 2009-2010 school year, 2,740 boys in seventh through 10th grades at 18 public schools who were invited to volunteer for BAM -- Sports Edition were assigned by lottery to either the participant or control group.
Conducting the trial required collaboration among the two nonprofits and the Crime Lab researchers. Mr. Eby says the effort went smoothly, in part because the two nonprofits shared their service data.
“It actually came together very naturally,” he says. “It wasn’t until talking about the intervention and networking with other organizations that I realized how rare that is to do effectively. We stayed motivated for the same reasons and had multiple levels of communication.”
Among the challenges of staying true to the experimental design, Mr. Eby says, was when coaches had to explain to students in the control group why they couldn’t participate in the after-school sports programs.
Mr. Ludwig says that although conducting rigorous research is essential to solving social problems, there’s still room for compassion from nonprofit staff members.
“These are people who do incredibly difficult and challenging work...They do that because they’re deeply committed to helping people,” he says. “Research is important, but we understand that when a kid comes in and says their family was evicted yesterday and has no place to go, they’re not going to kick them out.”
The Crime Lab accommodated for such situations using statistical analysis.
The Results: Reduced Violence, Increased Attention
The data show that BAM -- Sports Edition made a big difference in the lives of participants, reducing violent-crime arrests during the program year by 44 percent, or 8.1 arrests per 100 youth.
Running the trial also made a difference for Youth Guidance and World Sport Chicago by garnering attention to their efforts and highlighting the issue of youth violence.
“Over the past two years, we’ve had appearances from the mayor, the first lady, and the president of the United States sitting in on sessions and witnessing program,” Mr. Eby says. “I think the precedent that was set was that interventions like this do work. Now there’s hard evidence to deliver that. It’s much easier for politicians and other supporters to get behind.”
Despite the extra attention, Mr. Eby says that his staff’s reactions to the data were mixed.
“It was great to have the evidence of stuff we’d already observed, but we were very confident in the effectiveness of the program prior to this study,” he said.
But that kind of confidence among nonprofit leaders is often misplaced, Mr. Ludwig says.
“Very few program providers tell you at the start of the study that they think their program doesn’t work,” Mr. Ludwig says. “The kids that they see are the ones that actually show up and are enthusiastic and keep coming. That’s different from understanding the causal effect of the program that’s changing people’s lives.”
Next Steps: Leveraging Small Trials to Gain More Support
The Crime Lab, Youth Guidance, and World Sport Chicago used the results of the trial to win a $6-million grant from the National Institutes of Health to run a new study on the program to look for the root cause of the change in student behavior.
The money will pay for the research and program costs for two years.
The Edna McConnell Clark Foundation also awarded the program $2.75-million over three years.
Mr. Ludwig says that one of the benefits of starting with a low-cost randomized, controlled trial is it provides the evidence needed to apply for larger grants for high-powered, follow-up experiments.
Although Mr. Eby is glad to see more research being conducted, he’s ambivalent about the idea of continually running trials.
“We view evaluation and constant inquiry in our work to be an important aspect of developing the program and advancing forward,” he says. “But we don’t feel that randomized, controlled trials, and something of that rigor, is something we have to constantly be doing. There are constraints with trying to do something of such rigor and trying to do something very clinical in a real-world scenario.”
To Mr. Ludwig, though, the more evidence the better, because it means that programs proven to work will get the funding they deserve.
“I think a huge segment of the American public has internalized the idea that social programs don’t do any good,” he says. “We need evidence to push back against that privileged and data-free conventional wisdom -- rock-solid evidence to convince even the most skeptical skeptic.”
Students who participated: more than 800
Program cost per participant: $1,100
Reduction in violent-crime arrests: 44 percent
NIH grant for follow-up research: $6-million