May 12, 2016 | Washington DC
In collaboration with Global Delivery Initiative
Facilitated by Reboot
When is feedback SMART?
Feedback Labs is a group of like-minded organizations committed to the belief that regular people — whether we call them beneficiaries, constituents, or citizens — should be driving the policies and programs that affect them. Most members and supporters of the Labs instinctively believe that feedback is the “right” thing to do in aid and philanthropy; however there is less of a consensus about the instrumental value of feedback.
Gathering and acting on feedback takes resources, so we must ask if it leads to better outcomes – in other words, is it the smart thing to do? If so, how can it help us design and implement better programs? Are there instances when we see measurable correlations between the use of beneficiary feedback and improved outcomes of interest?
This summit brings together practitioners and experts in aid, development, and philanthropy to interrogate the concept of feedback as the “smart thing,” and to bring about improvements in these fields.
Speakers
Michael Woolcock
Lead Social Development Specialist
World Bank’s Development Research Group
Dean Karlan
Professor of Economics
Yale University
Stefan Koeberle
Director for Strategy, Risk and Results
World Bank
Maria González de Asis
Head of the Science of Delivery team
Global Delivery Initiative
Fay Twersky
Senior Director
William and Flora Hewlett Foundation
Vijayendra Rao
Lead Economist, Development Research Group
Social Observatory, The World Bank
Tim Ogden
Managing Director
Financial Access Initiative
Basit Chaudhry
Founder and CEO
Tuple Health
Samir K. Doshi
Senior Scientist
USAID Global Development Lab
Erin Holve
Senior Director
AcademyHealth
David Evans
Senior Economist
World Bank
Kavita Patel
Nonresident Senior Fellow
Brookings Institution
Caroline Fiennes
Founder and Director
Giving Evidence
Fredrik Galtung
President and co-founder
Integrity Action
Alison Hemberger
Markets and Learning Advisor
Mercy Corps
Dave Algoso
Freelance Consultant
algoso.org
Smart Summit Paper
As part of this conference we are publishing a draft of the paper “Is feedback smart?” This paper is a first attempt to frame the issue conceptually, review existing empirical work, and suggest productive avenues for future exploration. Why do we call this a “draft?” Because, we need your feedback. The questions discussed in this paper can’t, and shouldn’t, be answered by a small group of people. Better understanding requires an open, inclusive, and ongoing conversation to which we hope you will contribute your own expertise, experience, and knowledge. Your feedback will inform the next wave of research and discussion on this topic.
Initial Feedback
We asked several leading thinkers to provide initial feedback on the discussion paper. Have a look, and add your own voice here.
Dan Honig
The Johns Hopkins University SAIS
The paper opens by telling the reader “this paper is motivated by the idea that regular people – not experts – should ultimately drive the policies and programs that affect them.” (p. 9) But in this work it’s the elites that do the acting, with feedback and input informing their decisions. The paper argues this is the best that can be done; donors will always be de facto in control and thus “feedback offers the next best alternative for ensuring that important tacit and time-and-place knowledge make their way into program” (p.13). The claim, then, is: 1) It would be best if locals ran things, 2) this however isn’t possible because of how aid is controlled; as such 3) feedback is the best alternative because it gets at time-and-place knowledge in the best way possible, given the constraints. I question both 2 and 3.
Why Do Elites Need to Be in Charge?
We have seen the intermediation of experts in aid program administration challenged in a variety of ways, not just in theory (e.g. Easterly’s tyranny of experts), but also via alternative mechanisms (e.g. cash on delivery aid, the work of GiveDirectly, and social entrepreneurship). It sure seems that, at least sometimes, we can avoid donors’ hands being directly on the operational controls. When can alternative mechanisms allow feedback loops to flow through markets, local accountability channels, civil society, etc. rather than via elite decision makers? What are the limits of this, and thus when are we in this 2nd-best world you describe?
Does Beneficiary Feedback Actually Capture Local Knowledge?
How exactly will beneficiary feedback convey the tacit knowledge elites need to know? The paper quotes Polanyi on “we know more than we can tell”. (p. 12) Why is beneficiary feedback immune from this, with tacit knowledge not lost in the attempt to “tell”?
Additionally, while subjective judgments are always contextual – of a time and place – that does not mean they need be tacit knowledge rich. A simple way of assessing this may be to ask whether one in fact “knows more than they can tell” in a given case. If I’m asked to predict when my toddler is going to have a meltdown my feedback leverages tacit knowledge; I can’t explain why precisely I know a tantrum is coming but I think my prediction is nonetheless informative. When I fill out a survey from the local gas company it’s unclear that what I’m communicating is best framed as ‘tacit knowledge,’ rather than ‘perceptions’ or ‘feelings’ or ‘level of satisfaction’; I don’t have deeper knowledge of the relevant context for decision making, and indeed my feedback may be shallow and unhelpful in improving gas services. This isn’t a distinction without a difference – while some elements of what the paper claims feedback will achieve (e.g. participant ‘buy in’ emerging from feedback solicitation) may still hold irrespective of what the feedback communicates, elites will only be able to use feedback to learn from tacit knowledge when that feedback actually contains tacit knowledge.
Moving Forward
I think this work needs to:
- Take on the ‘eliminate the donor intermediary’ argument more directly.
- Delve deeper into what beneficiary feedback captures in what circumstances.
- Be more pointed on the agenda going forward. What are the specific open questions that need addressing first? When is beneficiary feedback most likely to work – that is, where should we look first to see impact?
Samantha Hammer
Independent Consultant
Even though it’s tricky to pin beneficiary feedback down as a clear concept, it seems straightforward that feedback loops contribute to good development work. “Is Feedback Smart?” shows how far we are from being able to concretely demonstrate the value that feedback has in achieving outcomes. This is an important starting point. The next step is to piece what we know into concrete hypotheses that can guide enterprising practitioners in building up the empirical knowledge about what kind of feedback is smart, when, and why. These are a few ideas about how to build on the current analysis and usefully target future research:
• Identify the factors that are key to designing smart feedback loops. In the examples of successful feedback loops that the paper cited, 3 elements seemed especially critical to creating smart feedback loops: 1) the complexity of the problem; 2) beneficiaries’ involvement in the intervention; and 3) the feedback mechanism (or mechanisms). Exploring and testing the relationship between these elements may lead to a more nuanced way to tailor feedback processes to specific interventions. Even digging deeper into the few examples we have now points to ways we can break down these elements into a few key dimensions to consider in designing feedback processes. For instance, taking the Uganda scorecard example: in that case, the problem was fairly complex—interactions between end-beneficiaries and other stakeholders were important, and there were hidden factors affecting outcomes—and beneficiaries themselves had a close connection to the key lever of change (absenteeism). The feedback mechanism was used for collaborative knowledge production and used over a sustained period. Might there be something to say for how co-creative and sustained feedback is valuable for interventions that have a similar profile?
• Measure feedback’s value toward achieving donors’ big-ticket goals. It’s going to take more than establishing a link between feedback and increased trust to get donors excited to fund robust feedback mechanisms. Going forward, it would be useful to target research to see if/how feedback can contribute to achieving the goals that donors care about most and are hard to achieve. Sustainability and scale stand out as 2 likely and compelling candidates. It seems intuitive that the legitimizing and trust-building function of feedback should translate to greater sustainability of a given intervention—by, among other things creating users and providers that are invested in a service’s success. Scale may be more promising, if we look to private sector experience. Some research on the role of feedback in the corporate world suggests that feedback contributes to customer advocacy and therefore the spread of products and services in new user groups and contexts. Could that help shape experiments to see if the same holds true for behavior change interventions, for instance? Development projects have already picked up the Net Promoter methodology; those cases should provide a targeted way to assess the distinct contribution of feedback.
• Use a comparative research approach to hone insights efficiently. Moving forward, comparing the growing evidence base to lessons from the neighboring practices of participatory approaches and traditional customer feedback could help illuminate what’s truly distinct, promising, and problematic about feedback loops in development interventions. Evaluations of participatory approaches cover many of the themes explored in the paper and suggested for follow-on research. They may help define specific mechanisms, use cases, or other factors influencing whether or not feedback is smart. They could also provide a point of comparison to better distinguish feedback as a distinct concept—which could be helpful from a practical and a donor optics perspective. Similarly, looking closely and comparatively at how the private sector collects and uses feedback, and measures its value, may provide some inspiration about how to do so in development contexts. Acknowledging that the feedback loop between the producer and customer is broken in development projects because the donor isn’t the end-user, that doesn’t mean that there isn’t some insight to be gained.
Genevieve Maitland Hudson
Director
OSCA
April 27 2016
- Theory
I think the section on theory could do with drawing on a wider range of sources. I’m not sure that economists do enough of the leg work when it comes to thinking through individual and communal identity and “local” knowledge. The absence of a fully developed theory leads to some, I’d suggest, oversimplified assumptions, eg. that “tacit” knowledge is usefully differentiated from “scientific” knowledge in a way that will helpfully inform the development of good feedback mechanisms. This is a question begging opposition, to my mind:
• What does it mean to ask for “tacit” knowledge from constituents?
• How does asking in itself develop understandings of self and community?
• What is best thought of as ‘private’ knowledge, and what is public and shared?
• What are the relationships between these kinds of knowledge?
• What does “scientific” knowledge mean? (Whose science? Naive or hard? You can make lots of things countable, by starting to count…)
Developing a theoretical framework for these kinds of questions would help with, for instance, some of the subsequent analysis on bias and peer influences.
There is a considerable literature on the effects of counting on behaviour, going back to the first studies of suicide in the nineteenth century. Ian Hacking has studied the effects of description on behaviour and self-understanding. He calls this the looping effect of human kinds. I think this is an important consideration in thinking about how feedback works, in that asking questions generates new possibilities for intentional action. This helps to explain why closing the loop is so important. It also helps to explain why pre-prepared score cards don’t work in the same way as participative feedback. Pre-prepared score cards offer only limited scope for re-description. This kind of thinking should also highlight some of the potential risks with feedback (certain formulations may lead to unexpected kinds of self-understanding, I’ve attached a blog I’ve written on safe spaces and mental health apps to show what I mean).
- Evidence
I wondered about the use of very different kinds of evidence under the heading of feedback. But perhaps this is really, for me, a subsection of theory above. ie. I think there are questions about the kind of knowledge “ordinary” people have about education, health, politics, water supply… There’s an assumption that these are sufficiently similar to warrant being thrown together in a relatively unreflective way. I’d like to see that unpacked a little more I think. A good theory would actually help with that. I’m quite keen on prototype theory as a means of explaining how we formulate understandings of concepts and manage differences within them. I’ve written about it here.
I think the ‘Way Forward’ section needs to reflect the lack of a theoretical framework for local knowledge. This could usefully be developed further. It would make for the formulation of better research questions, and would support analysis of any evidence.
Radha Rajkotia
Senior Director, Economic Recovery and Development
International Rescue Committee
May 10, 2016
The paper provides an excellent overview of existing research and evidence on the “case” for feedback. The premise of focusing on why and how feedback makes a difference is key to pushing forward this agenda.This is question that I have struggled with for some time as there is a need for us to get beyond the theoretical and moral rationale for client feedback and into the realm of why it makes sense – how it makes design and delivery of aid more effective or why it helps policy-makers make better decisions.
I think the paper does a good job on the former, but is insufficient on the latter. My concern with the paper and perhaps with how we think about feedback more broadly is that the loop seems to be composed primarily of aid recipient, implementer agency and donor. This composition of actors runs the risk of de-emphasizing political economy considerations, which also feed into decision-making for policy-makers and donors. This might be considered a separate feedback loop that contributes to decision-making (between tax-paying constituents, politicians and funders), but could be one in which aid recipients might be connected to aid contributors. This addition might enable us to focus on use of feedback beyond service provision in a specific location and instead allow us to understand how feedback (and the tensions from different feedback loops) influence decision-making in reality.