mySociety operates in a field that we term the Civic Power sector. This sector includes a wide range of organizations, including non-profits like Ushahidi, The Sunlight Foundation, Avaaz and other companies like Change and Nationbuilder. There are many differences between these organizations, but they do share one thing in common: in the context of the wider civil society & development world in which we are situated, they are very young indeed.  mySociety, celebrating it’s tenth birthday this year, is one of the oldest in this sector –  but we are a spring chicken compared to the likes of Oxfam, Amnesty International and Plan.

Like many of our peers, the fervor of youth leads us to focus on tangible deliverables.  For mySociety that has involved building and growing websites so that they have meaningful numbers of users – numbers that would show that users liked and wanted the services we offered. And now, as we approach middle-age, we’ve definitely achieved this. But millions of yearly visitors is ultimately not what we are about.  How do we know the feedback loops we create are truly effective in bringing about change?

Theory of change

Our underlying philosophy – our theory of change – is that enabling (and encouraging) politically inexperienced people to take actions like reporting broken street lights or asking for government information will make people more aware of their own power to get things changed, and that will benefit both them and the communities they live in. But just because lots of people perform these actions doesn’t mean we have affected those users in any profound way.

As we have matured we have started to ask ourselves some tough questions, including:

– Does the use of our sites and services (and those of our partners) make people more powerful in the civic and democratic aspects of their lives?

– Does this power genuinely deliver tangible beneficial impacts (particularly in the face of potentially unresponsive or corrupt governments)?

– Do our tools risk increasing the power of the relatively richer, better educated and technically adept minority at the expense of the majority?

Questions are cheap, answers are expensive

Right now, our answer to these questions is simply, “We don’t know for sure.”  Getting answers to these questions is much, much harder than simply reading Google Analytics.  For us to maintain moral credibility as an organization, to satisfy our funders and donors that their money is being well spent, and simply to sleep at night as individuals, that answer is clearly not good enough.

For this reason, research will be a core focus for mySociety in the coming years, we are currently working to secure funds to support this work at scale.  In the meantime the first couple of research projects are in the process of kicking off.

Theoretical challenges 

One of the challenges we face is that within our field there is not an easy or categorical connection between action and impact.  If you immunize a child against disease, then you can be certain that the child has a materially higher chance of remaining healthy.  There are of course wider discussions around whether immunization should be carried out by foreign NGOs or whether governments should work to improve their own health provisioning, but there is no doubt that the immunization itself is a good thing.

What about writing to a politician?  Is that a good thing?  We believe that it is.  We believe that it drives engagement and accountability and strengthens democracy.  But we can’t prove it, and we might be wrong. We must find out.

We have a great deal of data – page impressions, unique visitors, Freedom of Information requests raised, international re-uses of our code bases, messages sent to politicians, etc. – but no way of linking this to true impact.  In order to address this gap we will conduct methodologically rigorous, experimentally-driven research on both UK and international deployments of our technologies. We will then use the findings and the method we develop to encourage increased rigor in impact assessment by other organizations working in the Civic Power sector.

It is quite likely that some of these outcomes will be challenging for us, potentially suggesting that some of our workstreams have little or no true impact as things currently stand.  Nonetheless, we are committed to sharing the all of the results – good and ill – as they start to come through.

Paul Lenz – Head of Finance and International Projects, mySociety

Leave a Reply