Author: Renee Ho
Once upon a time there was a little girl. Her name was Goldilocks. She had golden hair…
In the children’s story of Goldilocks and the Three Bears, a woman breaks into the home of three anthropomorphic bears–Baby Bear, Mummy Bear, and Daddy Bear.
While they are away, Goldilocks tears through the bears’ furniture and food until she finds that which is “just right”.
Recently, I had the pleasure of attending a pre-launch event for the Innovation for Poverty Action‘s (IPA) Goldilock’s Project.
The Goldilocks Project is an effort “to help organizations and funders build ‘right-fit’ monitoring and evaluation (M&E) systems that balance the need for information on impact with operational data to inform management decisions.” Concretely, they will be coming out with a book and an online toolkit.
According to the event (and directly from their website):
“A right-fit M&E system follows a core set of principles that all organizations–regardless of their size or ability to assess impact–can apply. The Goldilocks principles are organized around creating credible, actionable, responsible, and transportable data collection systems, or CART for short.
- Credible: Only collect data that accurately reflects what you are intending to measure.
- Actionable: Only collect data that your organization is going to use. To make data actionable, ask if you can use the information to change the course of action at your organization—if not, do not collect it.
- Responsible: Match data collection with the systems and resources your organization has to collect it. Think about the resources you have. Don’t overreach, as doing so could compromise data quality.
- Transportable: Apply what you learn to other programs and contexts—either your own program in future years or in other locations, or those of other organizations working on similar problems.
“
Overall, and without having seen the final materials (due out later this year) for the project, there is reason to be excited. The organization that has been at the forefront (along with J-PAL) of the randomized controlled trial (RCT) movement is effectively saying, “It’s okay not to do an RCT. In fact, it actually may be a bad choice.”
Along the spectrum of monitoring and evaluation, the Goldilocks Project seems to rightly push the needle in the direction of monitoring. In doing so, it suggests that a lot of projects struggle in implementation and therefore, monitoring the implementation— versus evaluating the impact —may be the more appropriate thing to do.
IPA is recognizing that many of us want and need to monitor the health of a project, not wait to conduct an autopsy.
There are a few questions that linger and likely, hopefully they’ll be addressed when the Goldilocks Project toolkit and book is formally unveiled. The following are meant to open a constructive discussion:
- The last letter T? The CART principles end by saying we should apply what we learn to other programs and contexts. I am not sure I understand what is really transportable and I worry that what we learn has limited external validity1. Will the Goldilocks Project explain how we can overcome the common mistake of taking average RCT results and assuming they hold elsewhere? (see Dani Rodrik 2008)
- Are RCTs still rigorous? The project still asserts that for certain organizations at the right time within their program maturity, the RCT is still a good and rigorous option. I don’t disagree with this entirely but I don’t fully agree that the word “rigorous” should be bandied about in a tautological way. Are they necessarily more rigorous than other evaluation methods? (see Michael Woolcock 2013)
- Are we there yet? Right-fitting is a great idea but how do we know when we’ve arrived at the “right-fit”? If we’re measuring correctly? Is there a way we can get there without being like Goldilocks and destroying all of the bears’ food and shelter?
- How do we know we’re measuring the right thing? C is for credible and says we should only collect what we intend to measure. What if we don’t know what things matter?
- How about, you know, the “beneficiary”?2 Will the book and toolkit address how to incorporate beneficiary feedback and constituent voice in either monitoring or evaluation? This could easily be part of “R”, responsible.
The Goldilocks Project is stimulating a much-needed initiative. We are excited to participate and follow its progress!
Once upon a time there was a woman. Her name was Morena. She had black hair…
In the children’s story of Morena and the “Beneficiaries” a woman politely enters a country upon invitation.
Morena sits down with the “beneficiaries” over a cup of tea to discuss what they want to make their lives better. Together, and with on-going feedback, they develop a project and adapt, iterate, and improve to achieve social impact.
2. The word “beneficiary” is used in quotation marks here because I don’t like the word: it is condescending and disempowering. While others sometimes use the word constituent and citizen, I also find these a little problematic. To the extent that people more clearly understand “beneficiary” because of historical use and path dependency, I continue to use it here until I figure out the word I like best. Another blog will follow on this subject of vocabulary.
