Nonprofit Client Surveys: To anonymize or not to anonymize?

Nate Mandel, Center for Employment Opportunities  | October 27, 2021

For any organization wanting to receive feedback from its clients, the benefits of a fully anonymous survey may seem obvious at first. When a client’s name is not associated with their comments, they have no reason to fear any consequences and may feel free to be more honest and unfiltered with the feedback they provide. Ultimately, this type of candor can help organizations learn how they’re really doing in the eyes of their constituents and course-correct as needed. But within the nonprofit sector, where programming is available at no financial cost to clients, obtaining honest feedback from those receiving services can be a trickier task. 

The reason for this has to do with “courtesy bias.” Any client feedback received by a nonprofit is likely to be impacted in some way by inherent power dynamics that are unique to this setting. If clients worry that free services may be discontinued at any time, they may be more likely to avoid providing any form of criticism that may threaten that relationship. For example, clients might be extra cautious in responding to a survey if they perceive that their feedback might upset their case manager or result in them being treated differently or discharged from a program. 

In my work managing client feedback at the Center for Employment Opportunities (CEO), one client summarized this phenomenon succinctly during a focus group, asking me, “Why would I bite the hand that feeds?” We were examining the reasons behind our organization’s consistently high feedback scores. Clients agreed they would never rate their job coach anything less than a perfect score, even if there was room for improvement because they were simply grateful to have access to services.

Herein lies a tangible, pressing problem within our sector – if we cannot hear honestly from those we’re serving, how can we know what individuals really need to find success in their lives? And at the same time, if we don’t know who said what, how can we follow up with individuals to ensure they’re getting what they want? High-quality feedback loops continue to be an important tool within our field, and we must also address the logistical challenges around implementation and the very real power dynamics that can impact who is willing to share what they really think and how we can help people feel safe and comfortable speaking out freely.

Comparing anonymous and non-anonymous survey design methodologies can help us begin to unpack and understand this challenge. I believe both types of surveys are necessary tools for nonprofits to hear honest feedback from clients as well as respond directly to their needs. Recognizing that nonprofits have limited time and capacity, I hope to offer guidance on implementing both channels with minimal resources and lay out some benefits and drawbacks of each for organizations needing to choose just one method.

Non-anonymous surveys

The main benefit of non-anonymous surveying is being able to follow up with individuals based on what they have shared in surveys. Organizations can identify, for example, clients who:

• need more assistance or a different kind of support;
• have negative experiences that need addressing;
• are confused or utilize the survey to ask questions; or
• are generous with their feedback and likely want to share more.

Non-anonymous feedback helps managers maintain a pulse on the quality of service and variety of client experiences within a program. This is an impactful way to ensure that your program is being responsive to your clients’ needs, even when time and resources are limited. Most nonprofits can’t afford robust customer support departments, but SMS platforms like Mogli and Twilio offer elegant integrations with Salesforce (and nonprofit discounts), making it easier for teams with limited capacity to appoint champions to manage this work and remain in touch with client needs.

Anonymous surveys

The primary benefit of asking questions anonymously is that they can lead to more candor and honesty, as clients know that their name will not be tied to their response. In a part-time consulting role that I hold as a feedback coach with Listen4Good (L4G), I’ve supported dozens of nonprofits in administering surveys via SurveyMonkey. Based on my experience coaching organizations how to administer the simple L4G survey template, which asks a set of key questions on client satisfaction, I’ve clearly seen the benefit of asking questions anonymously.

At CEO, I wanted to see if we could use some of the L4G questions to compare anonymous and non-anonymous feedback. We developed an anonymous survey and set it up on tablets at each office. The survey itself included three of the exact same questions that we asked over our non-anonymous SMS platform.

A slide from the TV screen in the lobby of the CEO’s NYC office, drawing participants’ attention to the anonymous survey tablet.

In comparing the results across both survey methods over the same period of time, we found a fascinating statistical display of “courtesy bias” in action: The average raw scores for each question were lower among those responding via the anonymous channel (lower scores indicate less satisfaction). The statistical difference is what we might call a “courtesy inflation rate” and is displayed below as the percentage increase in average raw scores between the same survey question, asked anonymously vs. non-anonymously:

Anonymous vs. Non-Anonymous Survey Data

On each of the three questions, clients who were not anonymous responded with ratings that were 7 – 29% more positive than those who were anonymous. While organizations can mitigate bias and encourage candor through survey design or during orientation (e.g. how the survey process and purpose is explained to new clients), these findings show us that those efforts may produce only limited success when working in non-anonymous channels.

To anonymize or not to anonymize – why not both?

A poster to hang in CEO offices with a QR code for participants to scan and respond to the anonymous survey

As I have laid out, many nonprofits could benefit from administering both anonymous and non-anonymous surveys to their clients. Each method involves implementation that requires some technical and logistical planning, but need not be expensive or time-consuming.

At CEO, we re-designed our anonymous survey to be scannable as a QR code (no tablets needed, simply print and hang a poster or email/text your clients). Many websites offer QR services for free, and Google Forms and a basic SurveyMonkey account are also free to use. Organizations that don’t want to invest in SMS can still collect non-anonymous surveys by simply having clients fill in identifying information or write their name atop a paper survey. 

Regardless of administration method, having both anonymous and non-anonymous channels helps organizations get underneath the true experiences of their clients in ways that can lead to improvements in programming and stronger, more authentic connections with clients.

More about the author

With more than three years of experience designing and scaling beneficiary feedback systems, his approach uses a lens of social justice and incorporates both human-centered designs and evidence-based practices.

To learn more about Nate Mandel and his work, feel free to connect with him through LinkedIn.