Morine Ngumi, Feedback Labs | October 21, 2024
In a world where AI is transforming everything from business to education, have you ever wondered how it could revolutionize the way we gather, analyze, and act on feedback to better support communities? We recently had a chance to speak with Jennifer de Forest, Interim Vice President at YouthTruth, to learn how YouthTruth, with the support from the Bill and Melinda Gates Foundation, used AI to analyze feedback at scale. We heard how YouthTruth used AI in their Math Learning and Identity Project and Civic Empowerment Project to amplify student voices and help high schools evaluate civic readiness and improve equitable K-12 mathematics outcomes.
YouthTruth’s Civic Empowerment and Math Learning & Identity Projects
YouthTruth used generative AI to efficiently analyze large sets of open-ended student feedback in both projects, extracting meaningful insights at scale. Over the 2023-2024 school year, YouthTruth set out to evaluate how well public schools are preparing students for civic life. Over 115,000 high school students nationwide participated in YouthTruth’s student experience survey, uncovering a stark reality and showcasing challenges that philanthropy has the potential to tackle. Meanwhile, the Math Learning and Identity Project, an initiative that stemmed from the Gates Foundation’s desire to incorporate student feedback into their strategy, focused on student perceptions of math education. The project gathered insights into students’ sense of belonging, teacher support, and the relevance of math to real-world contexts. AI tools enabled YouthTruth to highlight statistically significant differences in student experiences across demographic groups.
Funding to Choose the Right AI Tool
In the fall of 2023, the Gates Foundation launched a pilot AI initiative aimed at supporting equitable K-12 math outcomes. As part of this initiative, YouthTruth, one of the selected partners, received funding to test innovative AI tools. This support allowed YouthTruth to carefully evaluate and choose the most suitable AI platform for their needs, Canvs AI, which could effectively analyze large-scale student feedback. This natural language processing tool allowed for sophisticated qualitative analysis, and enabled researchers to dig into the back-end processes to see how the AI was conducting its analysis. As exciting as this tool is, Jennifer was clear about one important lesson: AI comes with a cost. The research tool they used greatly increased the efficiency of their qualitative analysis, but there was a significant learning curve for her team, and the tool itself would have been prohibitively expensive if it weren’t for the upfront investment in the technology and staff capacity. This is where funders like the Gates Foundation play a critical role. By supporting nonprofits as they adopt AI tools, funders can help lower the barrier to entry and ensure these technologies are accessible across the sector.
Human-Centered AI: Students at the Core of Understanding Feedback
AI is a powerful tool, but it comes with its own risks and limitations, such as the potential for bias. For YouthTruth, the solution was to put the people behind the data at the center of the analysis process. When it was time to create their codebook for the civic empowerment project, the tool used to categorize feedback, YouthTruth brought in students to help shape how their own feedback would be understood. Essentially, a codebook is a guide used to systematically categorize open-ended responses based on recurring themes or ideas. The students were given verbatim responses from their district-level data (comments or feedback in the students’ own words). They then worked together in groups, physically organizing these responses into “buckets” based on similar ideas or patterns. By doing this, the students created their own coding schemes and code definitions, ensuring that the interpretation of the data genuinely reflected the students’ own voices and perspectives. This collaborative approach highlights a crucial lesson: AI should enhance, not replace, human involvement in feedback processes. By engaging students directly, YouthTruth ensured that the data analysis remained grounded in the realities of participants. Furthermore, YouthTruth’s data analysis team regularly checked the output of the AI analysis by looking at the sources behind the analysis, and making adjustments when necessary, to ensure that the results were an accurate representation of the students’ feedback.
Key Takeaways and Lessons for Others
YouthTruth’s experience offers valuable lessons for organizations looking to integrate AI into their feedback loops. Some of the lessons that we took away from our conversation with Jennifer include:
- Select the right tool: Before getting started with AI, do your homework by asking important questions about AI tools. How well does the tool align with your feedback goals? What is the quality of the tool and is it accessible to your team? Are there privacy concerns, potential biases, and equity issues that you need to consider? While organizations can easily access and leverage free or inexpensive open-source tools like ChatGPT, publicly available tools are often less transparent, adaptable, and sophisticated than more expensive tools like Canvas AI. The key is to find the right fit for your needs, resources, and staff capacity.
- Keep the Human in the Loop: While AI offers significant power in data analysis, it is essential to maintain human involvement throughout the entire feedback process. YouthTruth involved people in analysis by first engaging feedback participants in coding a sample set of responses, and later having staff scrutinize how AI tools made decisions. By asking for evidence of the AI’s analysis and validating its results, organizations can ensure that the insights generated are a more accurate reflection of the lived experiences of those they aim to serve.
As part of our continued journey into understanding AI’s role in feedback systems, we’re uncovering innovative approaches and challenges that organizations face today—some of which will be covered in Feedback Labs’ upcoming webinar on Unlocking AI for Feedback Tools: Challenges and Opportunities. To learn more about this ongoing work, please reach out to Alexis Banks at [email protected].
Dr. Jennifer de Forest, Ed.D., has nearly a decade of classroom and school leadership experience. She holds a master’s from Teachers College, Columbia, and a doctorate from Harvard. At YouthTruth, Jen helps school leaders turn community insights into actionable data. Connect with Dr. Jennifer to learn how she can support your school’s improvement efforts.
