Feedback

Smart Summit

In collaboration with Global Delivery Initiative

Facilitated by Reboot

When is feedback SMART?

Feedback Labs is a group of like-minded organizations committed to the belief that regular people — whether we call them beneficiaries, constituents, or citizens — should be driving the policies and programs that affect them. Most members and supporters of the Labs instinctively believe that feedback is the “right” thing to do in aid and philanthropy; however there is less of a consensus about the instrumental value of feedback.

Gathering and acting on feedback takes resources, so we must ask if it leads to better outcomes – in other words, is it the smart thing to do? If so, how can it help us design and implement better programs? Are there instances when we see measurable correlations between the use of beneficiary feedback and improved outcomes of interest?

This summit brings together practitioners and experts in aid, development, and philanthropy to interrogate the concept of feedback as the “smart thing,” and to bring about improvements in these fields.

Speakers

Michael Woolcock

Michael Woolcock
Lead Social Development Specialist
World Bank’s Development Research Group

Michael Woolcock

Michael Woolcock

Lead Social Development Specialist
World Bank’s Development Research Group

Michael Woolcock is Lead Social Development Specialist with the World Bank’s Development Research Group, and a (part-time) Lecturer in Public Policy at Harvard University’s Kennedy School of Government. He is the author of over 75 journal articles and chapters, as well as eight books, most recently ‘Building State Capability: Evidence, Analysis, Action’ (forthcoming with Oxford University Press; with Matt Andrews and Lant Pritchett). He is a co-founder of the World Bank’s global ‘Justice for the Poor’ initiative (2004); from 2006-2009 he was the founding Research Director of the Brooks World Poverty Institute at the University of Manchester; and in 2012 co-founded the ‘Building State Capability’ program at Harvard’s Center for International Development. In 2012 he was a co-recipient of the ‘best book’ prize, and in 2014 the ‘best article’ prize, from the American Sociological Association’s section on International Development. He is currently based in Malaysia, helping establish the World Bank’s first Knowledge and Research Hub. He is on the editorial board of numerous journals, and is the Asia-Pacific representative on the Scientific Advisory Committee of UNESCO’s largest research program (Management of Social Transformations). An Australian national, he has an MA and PhD in comparative-historical sociology from Brown University.

dean karlan

Dean Karlan
Professor of Economics
Yale University

dean karlan

Dean Karlan

Professor of Economics
Yale University

Dean Karlan is a Professor of Economics at Yale University and President of Innovations for Poverty Action, a non-profit dedicated to discovering and promoting solutions to global poverty problems. He is on the Board of Directors of the M.I.T. Jameel Poverty Action Lab. As a social entrepreneur Karlan is co-Founder of stickK.com, a website that uses lessons from behavioral economics to help people reach personal goals. In 2015, Karlan founded ImpactMatters, a nonprofit dedicated to producing impact audits, which assesses whether an organization uses and produces appropriate evidence of impact.

Karlan received a Presidential Early Career Award for Scientists and Engineers, an Alfred P. Sloan Foundation Fellowship, and a Guggenheim Fellowship. He was awarded distinguished alumni awards from the University of Chicago Booth Graduate School of Business and the Duke University Talent Identification Program. In 2011 he co-authored More Than Good Intentions: How a New Economics is Helping to Solve Global Poverty. His education includes a Ph.D. in Economics from M.I.T., an M.B.A. and an M.P.P. from the University of Chicago, and a B.A. in International Affairs from the University of Virginia.

Stefan Koeberle

Stefan Koeberle
Director for Strategy, Risk and Results
World Bank

Stefan Koeberle

Stefan Koeberle

Director for Strategy, Risk and Results
World Bank

Dr. Stefan G. Koeberle is the Director for Strategy, Risk and Results, Operations Policy and Country Services of the World Bank since August, 2013. Dr. Koeberle previously held a variety of positions at the World Bank including: Country Director for Indonesia, Director, Strategy and Operations in the Latin America and Caribbean Region and Director, Operations Services in Latin America. Since joining the World Bank in 1993, Dr. Koeberle has worked in a variety of countries and regions, including assignments in Thailand, Vietnam and Lao PDR. Prior to joining the World Bank, Dr. Koeberle worked as a post-graduate fellow at the German Development Institute in Berlin. He holds a Ph.D. in economics from Cambridge University. He has published a number of articles and several books on competitiveness, conditionality and budget support.

maria gonzalez

Maria González de Asis
Head of the Science of Delivery team
Global Delivery Initiative

maria gonzalez

Maria González de Asis

Head of the Science of Delivery team
Global Delivery Initiative

Maria González de Asis is the Head of the Science of Delivery team and the Global Delivery Initiative at the World Bank. She has been at the World Bank since 1997, where she worked in operations in different regions focusing on public sector reforms; managing anticorruption programs, disseminating emerging best practice in governance and anti-corruption worldwide, leading the area of Legal and Judicial Reform Learning Programs and the sectoral work and researching and advising countries on governance and development. Over the past years she has pioneered lending and non-lending operations and capacity building approaches for governance in different countries. Ms. González is a frequent speaker on governance issues in different fora (Stanford University, Kennedy School at Harvard, Georgetown University, among others). Ms.González de Asis has a Ph.D. in Law from the Universidad Autónoma of Madrid, and a M.P.P. from Georgetown University. Before joining the World Bank, she worked at Transparency International in Washington DC, Berlin and Peru, and for the Spanish Lawyer Firm Abogados Asociados.

Fay Twersky

Fay Twersky
Senior Director
William and Flora Hewlett Foundation

Fay Twersky

Fay Twersky

Senior Director
William and Flora Hewlett Foundation

Fay Twersky is Director of the Effective Philanthropy Group at the William and Flora Hewlett Foundation. She oversees five functions including cross-foundation strategy support, evaluation and organization learning as well as grantmaking in support of organizational effectiveness and a strong philanthropic sector. Twersky spent 2010–2011 working in Jerusalem, advising Yad Hanadiv (the Rothschild Family Foundation).

Twersky served for four years as Director and member of the leadership team of the Bill & Melinda Gates Foundation, designing and developing the Impact Planning & Improvement division. She was also a founding principal of BTW – Informing Change, a strategic consulting firm.

Twersky has authored many articles and reports. Recently, she published “The Artful Juggler,” in the Stanford Social Innovation Review on what it takes to be a successful Foundation Chief Executive Officer. She was principal author of Listening to Those Who Matter Most, the Beneficiaries and A Guide to Actionable Measurement. Twersky is a member of the board of directors for The Center for Effective Philanthropy and the UBS Optimus Foundation in Zurich, Switzerland. She serve on the Curriculum Advisory Committee for Philanthropy University, a newly launched Massive Open Online Course offered in collaboration with UC Berkeley’s Haas School of Business, Twersky holds two bachelor’s degrees in Rhetoric and Middle Eastern Studies from the University of California, Berkeley, and a master’s degree in City Planning from the Massachusetts Institute of Technology.

Vijayendra Rao

Vijayendra Rao
Lead Economist, Development Research Group
Social Observatory, The World Bank

Vijayendra Rao

Vijayendra Rao

Lead Economist, Development Research Group
Social Observatory, The World Bank

Vijayendra Rao is a Lead Economist in the Development Research Group of the World Bank. He heads the Social Observatory, a multi-disciplinary approach to build adaptive learning in large-scale participatory projects. He integrates his training in economics with theories and methods from anthropology, sociology and political science to study the social, cultural, and political context of extreme poverty in developing countries. Dr. Rao has published on a variety of subjects that include dowries and domestic violence in India, the economics of celebrations, sex work in Calcutta, participatory development, village democracy and deliberation, and inter-disciplinary approaches to public policy. He co-edited Culture and Public Action, and History, Historians and Development Policy, and co-authored the 2006 World Development Report on Equity and Development. Most recently, with Ghazala Mansuri, he co-authored the World Bank’s Policy Research Report on Localizing Development: Does Participation Work?

tim ogden

Tim Ogden
Managing Director
Financial Access Initiative

tim ogden

Tim Ogden

Managing Director
Financial Access Initiative

Tim Ogden is managing director of the Financial Access Initiative, a research center housed at NYU-Wagner and executive partner at Sona Partners, a thought leadership communications firm. He is co-author of Toyota Under Fire, and author of the forthcoming Experimental Conversations, a collection of interviews with economists about the use of randomized control trials in development economics and policy. He has developed and edited more than 20 books, including award-winning business books on lean business practices and innovation. Tim is a senior fellow of the Aspen Institute’s Financial Security Program, a contributing editor for Alliance Magazine, and serves on the board of GiveWell.

basit chary

Basit Chaudhry
Founder and CEO
Tuple Health

basit chary

Basit Chaudhry

Founder and CEO
Tuple Health

Dr. Basit Chaudhry is an internal medicine physician and medical technologist whose expertise focuses on health care payment, clinical service redesign, and the use of data analytics to improve clinical and financial performance in health care. He is the founder and CEO of Tuple Health, which works with providers, payers, and purchasers to transition to new models for value-based payment and collaborative, population health. Prior to starting Tuple Health, Dr. Chaudhry was a medical scientist at IBM Research where his work focused on using data analytics and information technology to drive innovation in health care. In addition to technology development, Dr. Chaudhry worked on developing IBM’s private and public-sector business strategies for the healthcare industry.

After finishing his medical training in internal medicine, Dr. Chaudhry completed the Robert Wood Johnson Clinical Scholars program at UCLA, earning a Ph.D focused on biomedical informatics and health services research. He has provided expertise to the U.S. Department of Health and Human Services and the Institute of Medicine and served on a working group of the President’s Council of Advisors on Science and Technology, co-authoring its policy report, “Realizing the Full Potential of Health Information Technology.”

samir dosh

Samir K. Doshi
Senior Scientist
USAID Global Development Lab

samir dosh

Samir K. Doshi

Senior Scientist
USAID Global Development Lab

Samir K. Doshi is a Senior Scientist at USAID’s Global Development Lab. Samir leads the Real-Time Data for Adaptive Management initiative, with a focus on how local communities can use digital technologies in complex environments to better monitor, evaluate, learn and adapt to emergent and dynamic situations. Samir also supports USAID’s work on the Ebola response and the strengthening of health systems in West Africa.

Samir has held teaching and research appointments at the University of Cambridge, the Stockholm Resilience Centre, Queen’s University and the University of Vermont. He was also a Fulbright-Nehru Senior Scholar in India focusing on grassroots innovations and sustainable development. Samir was a fellow of Jawaharlal Nehru University’s Institute for Advanced Study, the Santa Fe Institute, the Environmental Leadership Program and other institutions. Prior to his academic career, Samir worked as an engineer for local organizations on sustainable development projects in indigenous communities around the world. His Ph.D. research specialized in Systems Ecology, and his MS and BS focus was in Development Economics and Computer Systems Engineering, respectively.

erin holve

Erin Holve
Senior Director
AcademyHealth

erin holve

Erin Holve

Senior Director
AcademyHealth

Erin Holve, Ph.D. M.P.H. M.P.P is a senior director at AcademyHealth where she leads AcademyHealth’s work on research resources for the field of health services and policy research. Dr. Holve directs several projects related to patient-centered outcomes research, quality improvement and health system analytics, including serving as principal investigator of an AHRQ-funded grant, the Electronic Data Methods (EDM) Forum. She is founding editor and editor-in-chief of AcademyHealth’s open access peer-reviewed journal, eGEMs (Generating Evidence & Methods to improve patient outcomes) and is a senior advisor on numerous national-level programs to advance the goals of learning health systems. She holds a Ph.D in Health Services Research from Johns Hopkins University and Masters’ degrees in public health and public policy from the University of California, Berkeley.

david evans

David Evans
Senior Economist
World Bank

david evans

David Evans

Senior Economist
World Bank

David Evans is a Senior Economist in the Chief Economist’s Office for the Africa Region of the World Bank. He coordinates impact evaluation work across sectors for the Africa Region. In the past, he worked as Senior Economist in the Human Development Department in the Latin America and the Caribbean Region of the World Bank, and as an economist designing and implementing impact evaluations in Africa. He has designed and implemented impact evaluations in agriculture, early child development, education, governance, health, and social protection, in Brazil, the Gambia, Kenya, Mexico, Nigeria, Sierra Leone, and Tanzania. He coordinated the World Bank’s efforts to estimate the economic impact of the West African Ebola epidemic of 2014-2015. He has published in Demography, Economic Development and Cultural Change, the Lancet, the Lancet Global Health, and World Development. He teaches economic development at the Pardee RAND Graduate School of Public Policy, and he holds a Ph.D. in economics from Harvard University.

kavita

Kavita Patel
Nonresident Senior Fellow
Brookings Institution

kavita

Kavita Patel

Nonresident Senior Fellow
Brookings Institution

Kavita Patel is a Nonresident Senior Fellow at the Brookings Institution. Previously, Dr. Patel was managing director of clinical transformation at the Center for Health Policy and lead research on delivery system reform, health care financing, physician payment reform, and healthcare workforce development. She is a previous Robert Wood Johnson Clinical Scholar, and while at Brookings, she will return to providing clinical care as an internal medicine practitioner.

Dr. Patel is also a practicing primary care internist at Johns Hopkins Medicine. She also served in the Obama Administration as director of policy for the Office of Intergovernmental Affairs and Public Engagement in the White House. As a senior aide to Valerie Jarrett, President Obama’s senior advisor, Dr. Patel played a critical role in policy development and evaluation of policy initiatives connected to health reform, financial regulatory reform, and economic recovery issues. In addition, Dr. Patel has a deep understanding of Capitol Hill from her time spent on the late Senator Edward Kennedy’s staff.

Caroline Fiennes

Caroline Fiennes
Founder and Director
Giving Evidence

Caroline Fiennes

Caroline Fiennes

Founder and Director
Giving Evidence
@carolinefiennes

Caroline Fiennes is one of the few people whose work has appeared in both The Lancet and OK! Magazine. She is a leading advocate and campaigner for effective philanthropy, in which she has worked for twelve years. She serves on boards of: Charity Navigator; The Cochrane Collaboration (leading global research house at the centre of evidence-based medicine); and the Center for Effective Philanthropy (US philanthropy think-tank & research house). She works with Innovations for Poverty Action and formerly with J-PAL at MIT. A former award-winning charity CEO herself, she founded Giving Evidence which advises donors of various descriptions and in many countries about effective giving, and conducts research to improve it. Caroline speaks and writes extensively about the need for and barriers to effective giving, e.g., in the Financial Times, Forbes, The Economist, BBC Radio 4, Freakonomics, the Daily Mail, the philanthropy sector press. Her book It Ain’t What You Give, It’s the Way That You Give It is dedicated to all those who miss out because donors make the wrong call, and was described in the press as “indispensable… relentlessly logical… engaging, informative, irreverent … long overdue… Thank goodness somebody’s finally written this book… a tour de force”. Caroline has taught about effective giving at Oxford, Cambridge and Yale. She holds a surprisingly useful degree in physics and philosophy.

frederik

Fredrik Galtung
President and co-founder
Integrity Action

frederik

Fredrik Galtung

President and co-founder
Integrity Action

Fredrik Galtung is the president and co-founder of Integrity Action, an NGO that helps communities fix failing development projects. Over the past 20 years, Fredrik has consulted on strategic corruption control in more than forty countries, working with governments, international organisations (Council of Europe, World Bank, UN secretariat, UNDP, UNESCO, Unicef, UN Office of Drugs and Crime, etc.), several companies, foundations and governments and development agencies. Fredrik began his international career as the founding staff member and Head of Research of Transparency International (TI), the world’s first global anti-corruption NGO, with national chapters in some 100 countries. Fredrik is the founder of the Integrity Education Network which started as a joint project with the Central European University. IEN comprises more than 400 universities in 60 countries intent on developing an effective teaching, training and research programme in the field of public integrity. He started the Network for Integrity in Reconstruction, a group of NGOs and policy makers from post-war countries who address the integrity and corruption challenges of reconstruction. Fredrik is an Ashoka Fellow in recognition of his role as a social entrepreneur.

Alison Hemberger

Alison Hemberger
Markets and Learning Advisor
Mercy Corps

Alison Hemberger

Alison Hemberger

Markets and Learning Advisor
Mercy Corps

Alison Hemberger is Mercy Corps’ Markets and Learning Advisor, where she guides agency efforts to support adaptive management and advises on market systems interventions. Alison serves on the leadership team for Mercy Corps ADAPT partnership with IRC, aimed at driving institutional and program-level change using evidence of adaptive programming. Prior to her current role, Alison was Mercy Corps Liberia’s Director of Results Management where she was responsible for leading the team’s approach to monitoring and learning in market development programming. Before joining Mercy Corps, Alison worked across sub-Saharan Africa on projects to integrate M&E design with program strategy and policy, including work with Skoll Foundation, USAID, Technoserve, CARE, and Innovations for Poverty Action (IPA). Alison holds a Master’s in Public Policy from the Harvard Kennedy School with a focus on Political and Economic Development.

dave algoso

Dave Algoso
Freelance Consultant
algoso.org

dave algoso

Dave Algoso

Freelance Consultant
algoso.org

Dave Algoso is a manager, facilitator, and writer working as a freelance consultant in the international development and social impact sectors. Previously, as Managing Director at Reboot, he led a variety of governance and civil society projects for clients at the World Bank, USAID, and elsewhere. While working on peacebuilding programs with Mercy Corps, he launched a microgrants fund supporting youth groups in Kenya. His professional experience weaves across research and evaluation, political campaign organizing and strategy consulting, program design and organizational development. Dave also writes on aid, development, politics, and related issues on his blog (Praxis), on Twitter (@dalgoso), and elsewhere.

Dennis Whittle

Dennis Whittle
Co-Founder and Executive Director
Feedback Labs

Dennis Whittle

Dennis Whittle

Co-Founder and Executive Director
Feedback Labs
@DennisWhittle

Dennis Whittle’s adolescence was spent working as a newspaper delivery boy, dishwasher, busboy, ice cream scooper, mechanics assistant at a used car dealership, clerk in a bookstore, and maintenance assistant scraping the paint off wooden bleachers at a football stadium.

Dennis is co-founder and director of Feedback Labs. Previously, he worked as Lead Economist and Senior Partner at the World Bank, where his team created Innovation and Development Marketplaces which have been replicated in over 100 countries by the World Bank and other aid agencies, foundations, and impact investors. He also co-founded GlobalGiving, the first global crowdfunding + crowdsourcing website, where he was CEO from 2000-2010. He has since served as Executive Chairman of Ashoka Changemakers, Visiting Fellow at the Center for Global Development, Visiting Lecturer at Princeton University, Robin Richards Donohoe Professor of the Practice and Entrepreneur in Residence at UNC-Chapel Hill, and Board Director of Internews.

During his tenure at the World Bank from 1986-2000, Dennis lived and/or worked for many years in Indonesia and Russia. In 1984-85, Dennis worked for the Asian Development Bank and USAID in the Philippines, where he was an extra in the Chuck Norris film, Missing in Action. He has a BA in Religious Studies from UNC-Chapel Hill, where he was Morehead Scholar, and an MPA from the Woodrow Wilson School at Princeton University.

panthea

Panthea Lee
Principal
Reboot

panthea

Panthea Lee

Principal
Reboot

Panthea Lee is Reboot’s Principal and lead designer, focused on the practical applications of ethnography, design, and systems thinking in delivering effective international development and governance programs. She oversees all aspects of the program management process, including research, design, implementation, and evaluation.

Panthea has directed over $10 million of programming in over 20 countries, primarily across Asia, the Middle East and North Africa, and Sub-Saharan Africa. Her experience includes work in open government, civic innovation, public sector reform, ICT4D, education, and public health. She is a frequent leader of co-creation efforts that bridge the gap between government, civil society, communities, and the private sector.

Prior to co-founding Reboot, Panthea worked to leverage the use of technology in international development with UNICEF Innovation. Her work has been featured in The Atlantic, The Stanford Social Innovation Review, Al Jazeera, MIT Innovations Journal, Touchpoint: The Journal of Service Design, and Fast Company. She has lectured at Columbia, Harvard, NYU, and the School of Visual Arts, and is a graduate of McGill University.

Background Image

Draft Agenda

  • 8:30-9:00 Registration and Coffee
  • 11:00 – 11:15 Break + Coffee

9:00 – 9:30 Is Feedback Smart?

Does beneficiary feedback improve program design and implementation? Does it correlate with improved outcomes of interest? During this session we define what we mean by “feedback” and “smart”. This short session lays the groundwork for the rest of the day, beginning with an introduction from the Global Delivery Initiative.
Speakers: Dennis Whittle, Maria González de Asis with Panthea Lee

9:45 – 11:00 What is good evidence?

No one doubts that “evidence” is important for program implementers and policy-makers. The important questions are about what makes for good, operationally useful evidence? How is beneficiary feedback potentially useful ‘evidence’ throughout a project’s lifecycle? When is it appropriate to use a randomized control trial? When might other methods be better and why? What should we do when evidence from various sources conflicts?
Speakers: David Evans, Michael Woolcock, Vijayrendra Rao
Moderator: Timothy Ogden

11:15–11:45 “Unpacking” Evidence

Discussion leaders: Caroline Fiennes, Fredrik Galtung, and others

  • 4:30 – 5:15 What’s Next?
  • 5:15pm Cocktails and Conversation with GDI

12:00 – 1:15 Feedback and RCTs + lunch

Innovations for Poverty Action will be further exploring if beneficiary feedback is the smart thing to do.

Panelists: Dean Karlan
Moderator: Fay Twersky

1:30 – 2:45 Does patient feedback improve outcomes?

How can patient centered outcomes research and patient feedback help medical professionals deliver more effective healthcare? Under what conditions might patients know more than doctors and how can doctors and other medical professionals better harness and make use of this knowledge?

Speakers: Erin Holve, Kavita Patel
Moderator: Basit Chaudhry

3:00 – 4:15 It’s only SMART if it WORKS

How can and should we apply the concepts discussed today? What implementation, management, and measurement opportunities emerge- and what are the challenges? During this “unconference” style session, we will self-organize around the topics most relevant to attendees.

Introduction: Stefan Koeberle
Discussion leaders: Samir Doshi, Alison Hemberger, Maria González de Asis
Moderator: Dave Algoso

Smart Summit Paper

As part of this conference we are publishing a draft of the paper “Is feedback smart?” This paper is a first attempt to frame the issue conceptually, review existing empirical work, and suggest productive avenues for future exploration. Why do we call this a “draft?” Because, we need your feedback. The questions discussed in this paper can’t, and shouldn’t, be answered by a small group of people. Better understanding requires an open, inclusive, and ongoing conversation to which we hope you will contribute your own expertise, experience, and knowledge. Your feedback will inform the next wave of research and discussion on this topic.

Initial Feedback

We asked several leading thinkers to provide initial feedback on the discussion paper. Have a look, and add your own voice here.

dan honig

Dan Honig
The Johns Hopkins University SAIS

dan honig

Dan Honig

The Johns Hopkins University SAIS

Dan is Assistant Professor of International Development at Johns Hopkins SAIS. Dan’s research focuses on the relationship between organizational structure, management practice, and performance in developing country governments and organizations that provide foreign aid. Dan holds a Ph.D. in Public Policy from Harvard’s Kennedy School, where his dissertation focused on the optimal level of autonomy in the delivery of foreign aid, and holds a BA in political science and philosophy from the University of Michigan. Outside the academy Dan has worked for developing country governments and both local and international NGOs in his time living in Liberia, East Timor, Thailand, India, and Israel.

The paper opens by telling the reader “this paper is motivated by the idea that regular people – not experts – should ultimately drive the policies and programs that affect them.” (p. 9) But in this work it’s the elites that do the acting, with feedback and input informing their decisions. The paper argues this is the best that can be done; donors will always be de facto in control and thus “feedback offers the next best alternative for ensuring that important tacit and time-and-place knowledge make their way into program” (p.13). The claim, then, is: 1) It would be best if locals ran things, 2) this however isn’t possible because of how aid is controlled; as such 3) feedback is the best alternative because it gets at time-and-place knowledge in the best way possible, given the constraints. I question both 2 and 3.

Why Do Elites Need to Be in Charge?

We have seen the intermediation of experts in aid program administration challenged in a variety of ways, not just in theory (e.g. Easterly’s tyranny of experts), but also via alternative mechanisms (e.g. cash on delivery aid, the work of GiveDirectly, and social entrepreneurship). It sure seems that, at least sometimes, we can avoid donors’ hands being directly on the operational controls. When can alternative mechanisms allow feedback loops to flow through markets, local accountability channels, civil society, etc. rather than via elite decision makers? What are the limits of this, and thus when are we in this 2nd-best world you describe?

Does Beneficiary Feedback Actually Capture Local Knowledge?

How exactly will beneficiary feedback convey the tacit knowledge elites need to know? The paper quotes Polanyi on “we know more than we can tell”. (p. 12) Why is beneficiary feedback immune from this, with tacit knowledge not lost in the attempt to “tell”?

Additionally, while subjective judgments are always contextual – of a time and place – that does not mean they need be tacit knowledge rich. A simple way of assessing this may be to ask whether one in fact “knows more than they can tell” in a given case. If I’m asked to predict when my toddler is going to have a meltdown my feedback leverages tacit knowledge; I can’t explain why precisely I know a tantrum is coming but I think my prediction is nonetheless informative. When I fill out a survey from the local gas company it’s unclear that what I’m communicating is best framed as ‘tacit knowledge,’ rather than ‘perceptions’ or ‘feelings’ or ‘level of satisfaction’; I don’t have deeper knowledge of the relevant context for decision making, and indeed my feedback may be shallow and unhelpful in improving gas services. This isn’t a distinction without a difference – while some elements of what the paper claims feedback will achieve (e.g. participant ‘buy in’ emerging from feedback solicitation) may still hold irrespective of what the feedback communicates, elites will only be able to use feedback to learn from tacit knowledge when that feedback actually contains tacit knowledge.

Moving Forward

I think this work needs to:

  • Take on the ‘eliminate the donor intermediary’ argument more directly.
  • Delve deeper into what beneficiary feedback captures in what circumstances.
  • Be more pointed on the agenda going forward. What are the specific open questions that need addressing first? When is beneficiary feedback most likely to work – that is, where should we look first to see impact?

samantha hammer

Samantha Hammer
Independent Consultant

samantha hammer

Samantha Hammer

Independent Consultant

Samantha Hammer is a researcher and strategist specializing in using ethnography and human-centered design to shape public policies, programs, and services. Her work has spanned fields of public health, human rights in fragile states, criminal justice, and technology for development. Across projects, she aims to advance social and economic equality and strengthen relationships between individuals, communities, and governments. As research manager with the social impact firm Reboot, she advised major development donors, civil society organizations, and government agencies, leading research and participatory design processes to create systems to better meet the needs of individuals and communities. Earlier, she managed research and community development projects advocating for women’s rights in Kosovo, Georgia, and Guatemala. She has worked with organizations including UNICEF, USAID, Mercy Corps, the NYC Mayor’s Office of Criminal Justice, the University of Chicago Crime Lab, and the Pacific Council on International Policy. She holds a Master of International Affairs from Columbia University’s School of International and Public Affairs and lives in Brooklyn.

Even though it’s tricky to pin beneficiary feedback down as a clear concept, it seems straightforward that feedback loops contribute to good development work. “Is Feedback Smart?” shows how far we are from being able to concretely demonstrate the value that feedback has in achieving outcomes. This is an important starting point. The next step is to piece what we know into concrete hypotheses that can guide enterprising practitioners in building up the empirical knowledge about what kind of feedback is smart, when, and why. These are a few ideas about how to build on the current analysis and usefully target future research:

• Identify the factors that are key to designing smart feedback loops. In the examples of successful feedback loops that the paper cited, 3 elements seemed especially critical to creating smart feedback loops: 1) the complexity of the problem; 2) beneficiaries’ involvement in the intervention; and 3) the feedback mechanism (or mechanisms). Exploring and testing the relationship between these elements may lead to a more nuanced way to tailor feedback processes to specific interventions. Even digging deeper into the few examples we have now points to ways we can break down these elements into a few key dimensions to consider in designing feedback processes. For instance, taking the Uganda scorecard example: in that case, the problem was fairly complex—interactions between end-beneficiaries and other stakeholders were important, and there were hidden factors affecting outcomes—and beneficiaries themselves had a close connection to the key lever of change (absenteeism). The feedback mechanism was used for collaborative knowledge production and used over a sustained period. Might there be something to say for how co-creative and sustained feedback is valuable for interventions that have a similar profile?

• Measure feedback’s value toward achieving donors’ big-ticket goals. It’s going to take more than establishing a link between feedback and increased trust to get donors excited to fund robust feedback mechanisms. Going forward, it would be useful to target research to see if/how feedback can contribute to achieving the goals that donors care about most and are hard to achieve. Sustainability and scale stand out as 2 likely and compelling candidates. It seems intuitive that the legitimizing and trust-building function of feedback should translate to greater sustainability of a given intervention—by, among other things creating users and providers that are invested in a service’s success. Scale may be more promising, if we look to private sector experience. Some research on the role of feedback in the corporate world suggests that feedback contributes to customer advocacy and therefore the spread of products and services in new user groups and contexts. Could that help shape experiments to see if the same holds true for behavior change interventions, for instance? Development projects have already picked up the Net Promoter methodology; those cases should provide a targeted way to assess the distinct contribution of feedback.

• Use a comparative research approach to hone insights efficiently. Moving forward, comparing the growing evidence base to lessons from the neighboring practices of participatory approaches and traditional customer feedback could help illuminate what’s truly distinct, promising, and problematic about feedback loops in development interventions. Evaluations of participatory approaches cover many of the themes explored in the paper and suggested for follow-on research. They may help define specific mechanisms, use cases, or other factors influencing whether or not feedback is smart. They could also provide a point of comparison to better distinguish feedback as a distinct concept—which could be helpful from a practical and a donor optics perspective. Similarly, looking closely and comparatively at how the private sector collects and uses feedback, and measures its value, may provide some inspiration about how to do so in development contexts. Acknowledging that the feedback loop between the producer and customer is broken in development projects because the donor isn’t the end-user, that doesn’t mean that there isn’t some insight to be gained.

genevieve

Genevieve Maitland Hudson
Director
OSCA

genevieve

Genevieve Maitland Hudson

Gen is a researcher, evaluator and project designer. She has wide-ranging experience of academic and social research. For the last six years she has worked with public, private and third sector organisations to design programmes that draw on a robust evidence base and demonstrate their effectiveness using appropriate metrics and methodologies. She has a particular interest in the use of human-centred measurement in assessing the effectiveness of social programmes.

Gen leads on research and evaluation at UK consultancy Osca. She works on projects in health, education and organisational change often developing new research methods to suit different people and different styles of delivery. Recent projects have included evaluation of Year of Care implementation for the London Borough of Islington, developing new evaluation measures for a youth employability project and rethinking measurement use in management training for Kent County Council.

Gen also writes about evaluation and social measurement, and has published widely on this topic.

Gen started her career in academia with a doctorate in philosophy and a particular interest in identity. She gravitated from research work on identity towards social research and has lectured on social research methodologies and evaluation at the Ecole Normale Supérieure in Paris and Birkbeck College London. She was formerly Head of Social Impact at The U, a social venture developed by the Young Foundation, and founder and director of GLUE, a social enterprise working with young people excluded from school.

April 27 2016

  1. Theory

I think the section on theory could do with drawing on a wider range of sources. I’m not sure that economists do enough of the leg work when it comes to thinking through individual and communal identity and “local” knowledge. The absence of a fully developed theory leads to some, I’d suggest, oversimplified assumptions, eg. that “tacit” knowledge is usefully differentiated from “scientific” knowledge in a way that will helpfully inform the development of good feedback mechanisms. This is a question begging opposition, to my mind:
• What does it mean to ask for “tacit” knowledge from constituents?
• How does asking in itself develop understandings of self and community?
• What is best thought of as ‘private’ knowledge, and what is public and shared?
• What are the relationships between these kinds of knowledge?
• What does “scientific” knowledge mean? (Whose science? Naive or hard? You can make lots of things countable, by starting to count…)
Developing a theoretical framework for these kinds of questions would help with, for instance, some of the subsequent analysis on bias and peer influences.

There is a considerable literature on the effects of counting on behaviour, going back to the first studies of suicide in the nineteenth century. Ian Hacking has studied the effects of description on behaviour and self-understanding. He calls this the looping effect of human kinds. I think this is an important consideration in thinking about how feedback works, in that asking questions generates new possibilities for intentional action. This helps to explain why closing the loop is so important. It also helps to explain why pre-prepared score cards don’t work in the same way as participative feedback. Pre-prepared score cards offer only limited scope for re-description. This kind of thinking should also highlight some of the potential risks with feedback (certain formulations may lead to unexpected kinds of self-understanding, I’ve attached a blog I’ve written on safe spaces and mental health apps to show what I mean).

  1. Evidence

I wondered about the use of very different kinds of evidence under the heading of feedback. But perhaps this is really, for me, a subsection of theory above. ie. I think there are questions about the kind of knowledge “ordinary” people have about education, health, politics, water supply… There’s an assumption that these are sufficiently similar to warrant being thrown together in a relatively unreflective way. I’d like to see that unpacked a little more I think. A good theory would actually help with that. I’m quite keen on prototype theory as a means of explaining how we formulate understandings of concepts and manage differences within them. I’ve written about it here.

I think the ‘Way Forward’ section needs to reflect the lack of a theoretical framework for local knowledge. This could usefully be developed further. It would make for the formulation of better research questions, and would support analysis of any evidence.

Radha Rajkotia

Radha Rajkotia
Senior Director, Economic Recovery and Development
International Rescue Committee

Radha Rajkotia

Radha Rajkotia

Radha Rajkotia is the senior director for economic programs at the International Rescue Committee. In this role, she sets the vision for IRC’s global strategy for economic programming and supervises a team of technical experts who provide support to IRC’s economic programming, globally. Radha has fifteen years of experience in economic and youth-focused programming and joined the IRC after completing her PhD in Refugee Studies at the University of Sussex. During and preceding her doctoral studies, Radha supervised research projects for UK-based clients on issues relating to migrant workers, refugee and asylum-seeking women and children and black and minority ethnic groups. She also led the development of employment and enterprise development programs. Radha has degrees in Refugee Studies, Social Research Methods, Migration Studies and Political Science. She lives in Washington, D.C., with her husband, Chris, and daughter, Esme. You can follow her on Twitter at @RadhaRajkotia.

May 10, 2016

The paper provides an excellent overview of existing research and evidence on the “case” for feedback. The premise of focusing on why and how feedback makes a difference is key to pushing forward this agenda.This is question that I have struggled with for some time as there is a need for us to get beyond the theoretical and moral rationale for client feedback and into the realm of why it makes sense – how it makes design and delivery of aid more effective or why it helps policy-makers make better decisions.

I think the paper does a good job on the former, but is insufficient on the latter. My concern with the paper and perhaps with how we think about feedback more broadly is that the loop seems to be composed primarily of aid recipient, implementer agency and donor. This composition of actors runs the risk of de-emphasizing political economy considerations, which also feed into decision-making for policy-makers and donors. This might be considered a separate feedback loop that contributes to decision-making (between tax-paying constituents, politicians and funders), but could be one in which aid recipients might be connected to aid contributors. This addition might enable us to focus on use of feedback beyond service provision in a specific location and instead allow us to understand how feedback (and the tensions from different feedback loops) influence decision-making in reality.

The Smart Summit is made possible with special thanks to the Global Delivery Initiative and Reboot.


Global Delivery Initiative

reboot-logo