The Open Contracting Partnership (OCP) is a silo-busting collaboration across governments, businesses, civil society, and technologists to open up and transform government contracting worldwide. They bring open data and open government together to make sure public money is spent openly, fairly and effectively. They focus on public contracts as they are the single biggest item of spending by most governments. They are a government’s number one corruption risk and they are vital to make sure citizens get the services that they deserve.
Learning and evaluation is a key part of OCP’s work. They have an internal learning framework that they update as a team every quarter. They re-evaluate if they are tracking the right things at least once under each new strategy and then they course-correct as needed both in terms of indicators as well as targets. They then incorporate lessons from every quarter into their individual work plans. They track the following indicators for measuring community engagement and empowerment:
- Number of contacts who have sent them at least 3 emails in the last year
- Number of new non-OCP projects, programs, and organizations that get funding
- Network average interconnectivity
- Number of people and organizations in our wider network who engage in open contracting conversations
- Empowerment and Net Promoter Scores
- Field survey feedback
- Trainings
This LabStorm focused on the first four engagement-focused indicators, specifically the email indicator. OCP is planning to use email activity as an engagement metric for the foreseeable future, and they want to make sure they do it right. They came to the LabStorm group for guidance on using this indicator, and for suggestions on other indicators they could use to measure community engagement. Here is what LabStorm attendees had to say:
Figure out what three emails really say about someone’s engagement, and adapt accordingly. In order to measure their community’s email engagement, OCP ran an analysis to identify how many people sent three emails to OCP in the past year. They chose the number three because it at least means something was being followed up on between a member of OCP and a third party. Yet some participants suggested that three emails does not necessarily equate to meaningful engagement; sometimes, three emails can just be back-and-forth to set-up a call that does not lead to any action. LabStorm attendees recommended that OCP measure email engagement by tagging emails as they came in to mark which ones measured more significant actions between OCP and a third party. By looking at how three emails relates to engagement and new member conversion, OCP can ensure that their email metric actually provides value.
Add more context to your indicators without compromising privacy. While using additional data can help OCP determine the significance of three emails, LabStorm attendees noted the potential privacy concerns that could emerge from using email information. Attendees suggested OCP consider several actions to protect user privacy such as only collecting high-level thematic data and using a plug-in that would grant implicit permission. OCP is already very careful and only uses the email recipients and email frequency data. OCP can further use data to inform their engagement metrics while strengthening relationships with their community by seriously considering what information is being used and how it is being protected.
Define the “conversion points” in network members’ journeys, so that OCP can choose indicators that clearly reflect those conversion points. Attendees suggested that identifying the “conversion points” — common points in OCP’s offerings when new members join — could help OCP understand how to attract and better serve new members. Finding appropriate indicators will help OCP identify conversion points and what might attract new members. LabStorm attendees suggested identifying those conversion points and then creating a targeted feedback collection process to figure out if those conversion points resonate with the community. By connecting data to identifying conversion points, OCP will make it even easier for prospective members to join the community.
This LabStorm reminded us of the importance of gathering meaningful feedback and indicators of connection while respecting the privacy and time of the people we are working with. Do you have any suggestions for Open Contracting Partnership on how they can collect high-quality feedback and indicators? Please leave a comment below or email [email protected].
