Motivating Organization Participation in Feedback
If you’ve seen the movie Field of Dreams, you likely recall the scene where a confused Kevin Costner wanders a corn field hearing the words, “If you build it, they will come.” Those experimenting with new tools in feedback loops may have felt like Kevin Costner at some point.
This summer I tested tools with a group of organizations to understand how to get them to use relevant community feedback which GlobalGiving had already gathered through our Storytelling Project. To understand the test, I should first tell you about our inspiration: Sesame Street.
“D” Is for “Distractor”
The creators of Sesame Street believed they could teach children through television, provided the programs engaged children. Until that point, show quality was largely measured by the number of viewers. Anyone who has sat through a terrible movie or a boring lecture understands that this is a poor measurement of quality.
To measure engagement, Edward Palmer and his team developed “the distractor.” This experiment involved children sitting in front of two televisions: one screen played an episode of Sesame Street, and the other flashed colorful images of puppies and paper clips that changed every seven seconds. Researchers recorded when a child’s attention diverted away from the episode of Sesame Street towards the “eye candy.
This empirical approach allowed the Children’s Television Workshop to revise their concept dozens of times based on what actually worked, often along paths that “experts” said would be damaging to children.
Understanding NGO Incentives
In the tradition of Sesame Street, we devised a similar experiment with two parallel offerings. We created a website where service organizations could learn what people were saying in the towns where they were working. “Would they listen?” we wondered. Or, would they be distracted by the list of funding and training opportunities gathered from across the web that appeared on the same page?
To conduct the experiment we needed foot traffic through the tool. Over the course of a month I emailed 104 Ugandan organizations telling them about the website. I also included a personalized report with an example of analysis related to that specific organization’s mission using stories from a community where they actually worked. I explained how this could be used to write a stronger funding proposal and offered to do a one-on-one training call. This is an example of the “concierge minimum viable product” approach to learning what the “product” should be from the Lean Startup book by Eric Ries. At the end of a month we ended up with an email response rate of 14%. Only about 5% connected on a call or tested the website with the compiled storytelling feedback.
Most organizations did not look at the tool. And without many giving it a glance, we couldn’t start to tell how useful our tools might be. Were these organizations reading their emails? Were we contacting the right people?
As a second test, we sent emails linking to a web form with over 100 funding and training opportunities, the “eye candy” that we were using to distract organizations. We asked them read the list and check a box next to the opportunities that interested them. We neglected to mention that all of the grant descriptions were hyperlinked to the grant application pages, but people figured that out anyway.
44 of 104 organizations responded in the first 12 hours (43% response). Over the next two weeks, people clicked on a link to a grant application 759 times. Organizations were far more eager to apply for funding than they were in improving their applications. Still, many people actually did follow one of the links to the evaluation tools we had wanted to test in the first place.
Keeping Their Attention
The lesson we took from here is that others won’t participate in your big idea if you aren’t willing to redesign it around exactly what they think is the core problem. At GlobalGiving, we believe that community feedback is valued when organizations consider it while writing their next grant proposal, but in order to even reach that step, we need to build relationships with organizations and inform them about grant opportunities first. We might even need to build relationships with funders, so that grant seekers will believe us when we tell them that funders believe community feedback is valuable. That is the pivot we are currently discussing.
This is a story about learning about our products and our customers. Although the technology to enable feedback loops exists, putting them into practice requires thinking about the incentives for adopting them. That’s messy and complex. And without strategies to test incentives like those from the Lean Startup principles or Sesame Street’s model for measuring behavior, it will take forever. And even if we build it, they will not come.