Chely Esguerra and Nisma Le Boul | December 2, 2020
In the conflict-affected countries where the International Committee of the Red Cross (ICRC) works, connectivity and access can be difficult. When COVID-19 hit earlier this year, ICRC worked quickly to ensure that they got useful information about the coronavirus to the people they serve. How did ICRC do it? They turned to radio. Radio is a time-tested low-tech solution to share messages with these communities, and it consistently ranks as a preferred and trusted medium for receiving information.
The ICRC’s institutional strategy is anchored on the statement “putting people at the center”. One of the ways it does this is by engaging with communities – through information, feedback, and participation – to ensure that the people it serves are able to shape the organization’s humanitarian programs. After successfully running COVID19 prevention radio spots and producing one of their first interactive live shows, the ICRC asked the people they serve for feedback to figure out if their radio work had made a difference. ICRC called radio listeners and surveyed them over the phone, asking questions such as “What did you think of the program?”, “What did you learn?”, and “What is your most trusted/reliable source of information?” In their phone surveys, ICRC heard from 48 radio listeners and got some useful feedback on the ICRC radio program.
For the next round of radio impact evaluation, the ICRC seeks to be more systematic. Representatives from the ICRC came to the LabStorm group for advice on making the impact evaluation process less laborious and more scalable, how to determine what the right questions to ask are, and how to measure the value of their interactive programs based on what happens when people call in to the interactive shows. Here’s what LabStorm attendees had to say.
- Adapt technology with time. When it comes to impact evaluation, there is no perfect technical product to make the process less laborious. LabStorm attendees from various industries validated that the ICRC was actually on the right track using excel to record survey responses. Some attendees recommended that in the future, the ICRC look into existing platforms that use Interactive Voice Response or SMS messaging in order to more efficiently manage and record data. Of course, or any written messaging efforts the ICRC must take the survey respondents literacy level into account. They can adjust the wording or language accordingly, or offer a voice-to-text option for those who cannot read or write.
- Learn which questions are best in the early stages. Measuring impact is challenging enough, but measuring the impact of the survey questions you ask to measure impact… that’s even harder! LabStorm attendees assured the ICRC that it is almost impossible to land on the perfect impact measurement questions the first time around. The ICRC can learn which questions worked in this round of phone interviews and use those insights to define the next phase of interviews. LabStorm attendees also assured the ICRC that subjective questions are good to ask because they can help the ICRC hone in on trends in the early stages.
- Measure behavior change. The goal of the ICRC’s coronavirus radio coverage is to encourage positive behaviors such as social distancing and mask-wearing. To measure the value of their radio programs, therefore, the ICRC can focus on how the programs have changed listeners behavior. LabStorm attendees shared a few ideas for how the ICRC could measure behavior change resulting from their radio programs. One attendee suggested that the ICRC identify three areas of behavior they would like to change through their radio programming, craft a questionnaire around those areas, and then compare questionnaire results before and after listening to the radio program. Another attendee shared a variety of options for measurement, including self reports, exposure vs. non exposure experiments, change over time experiments, and scientific observation. The ICRC can utilize any of these techniques to interact with their radio audience and see if their programs are really making a difference.
This LabStorm reminded us that impact measurement is an adaptive process. In order to figure out if their radio program is making a difference in behaviors around coronavirus, the ICRC will have to adjust their technology over time, learn which questions work, and experiment with ways to measure behavior change. If you have a great idea for ICRC to measure the impact of their program, please reach out to [email protected]!
