Feedback comes in many forms. This can be a particularly important realization for a young social enterprise. At 990 Consulting, we’ve learned to look out for feedback in unexpected forms – ones that promote innovation in the use of data science to understand the nonprofit sector. When we find that feedback, we ask ourselves:
“What would we do if we could just start over?” And it turns out, if you plan for it, you can do just that—again and again.
Readers of the Feedback Labs Blog may recall a 2017 DataStorm facilitated by Dr. David Borenstein, at the time the Director of Data Science at Charity Navigator, and now co-founder of 990 Consulting and its free data site, Open990. The DataStorm featured a then-upcoming gathering of the Nonprofit Open Data Collective – a group of individuals and organizations devoted to making nonprofit tax returns more usable.
The previous year, the IRS had made over 1.4 million nonprofit tax returns (aka Form 990s) publicly available in machine-readable format. The returns, which today number over 2.4 million and span back to 2010, created the possibility of dramatically increased transparency in the nonprofit sector. But wrangling the data into a standardized dataset conducive to analysis was proving difficult. The Nonprofit Open Data Collective’s first meeting made great progress, but there was much work left to be done. The Collective arranged to meet again to validate the work from the first event and to tackle new sections of the IRS form.
- Get many perspectives at once…twice. In the months since the first meeting, members of the Collective had been working independently and with their colleagues at Charity Navigator, Guidestar, Aspen Institute, Urban Institute, and universities across the country. The second meeting reconvened the same key participants and some new ones; the group included open data advocates, data scientists, journalists, accountants, professors, students, and more. Their feedback at the meeting was based not only on their independent work with the data, but also comments, questions, and goals gathered from their own feedback networks during the intervening months. Meanwhile, feedback during the hands-on part of the meeting showed us where the bottlenecks were.
- Recognize unexpected challenges as a form of feedback. The IRS form 990 data is rich, but inconsistently structured. In reflecting on the second gathering’s slow progress, we realized that most of the time was being spent keeping track of the relationships between the many data points. To solve this, we designed a data “pipeline” capable of managing these hierarchical relationships.
Having solved this problem, though, we hit a new bottleneck: the meaning of the data was obscured by pages of accounting rules, often involving calculations from other parts of the form. So we engineered an internal web app purpose-built to keep track of these relationships. After a couple months of using this tool, we found that its Excel-like interface made our growing network of relationships impossible to interpret. So we started a third improvement cycle, rebuilding from the ground up. - Welcome unsolicited feedback from outside your network. Although ongoing feedback from our network was very helpful, some of the most important insights came unsolicited from users of open990.com. Their feedback alerted us to the potential for insights within non-financial parts of the tax returns—mission, program accomplishments, grants. Now we are developing a natural language algorithm to help organizations connect with others who share similar purposes and activities.
Sometimes the challenge isn’t getting useful feedback, but rather recognizing feedback for what it is. Successful integration of feedback requires recognizing feedback wherever it appears— formal or informal, solicited or unprompted—and planning for concerted, meaningful iteration.
Heather Kugelmass is co-founder of 990 Consulting and serves as its Director of Research. Her prior research investigated inequalities in higher education and in access to mental health care. Heather earned a Ph.D. in Sociology from Princeton University and an M.A. in Quantitative Methods in the Social Sciences from Columbia University.
David Borenstein is co-founder of 990 Consulting and serves as its Chief Data Scientist. Formerly the Director of Data Science at Charity Navigator, he created open-source software to turn the IRS Form 990 into spreadsheets. Prior to Charity Navigator, he worked as a computational biologist at the Broad Institute of MIT and Harvard. He holds a Ph.D. in Quantitative and Computational Biology from Princeton University.
