We're trying out a new experiment. Many of our clients ask us how should we measure our advocacy efforts? There are indeed many ways to measure advocacy, as we will discuss below. So to test what works, we're going to try to measure the impact of this Blog posting. We've decided to advocates for a more transparent and accountable nonprofit sector. So we're asking you to do two things:

1. Learn: Go to the Center for What Works, a charity that helps other charities build capacity to measure and benchmark their performance. You'll find helpful resources and a list of common outcomes and metrics.
2. Take Action: Take 2 seconds and Join as a Free Member of Center for What Works. You'll help support the movement toward greater accountability in the nonprofit sector.

We're going to measure three outcomes: the extent to which we are able to inform our readership about this issue, whether we build visibility and support for the issue, and ultimately, whether we can increase the number of nonprofits tracking and reporting their results. We'll report back to this blog periodically with our results.

In the meantime, how are others in the sector measuring advocacy or policy change?

About Measuring Advocacy

Let's start with the obvious: measuring advocacy/policy change is difficult. The difficulty relates to the fact that you cannot "control" for outside factors in a political system. Program evaluators like to have control groups to isolate outside variables and see if the program itself is the only reason why a change occurred. This is an example of the limitations of an evaluation methodology. While it may be difficult to prove impact, we can certainly measure progress.

Many of our clients, especially foundations and nonprofits but sometimes corporations too, are trying to advance a particular social agenda. They fund public policy research and advocacy efforts on issues like education reform, youth development, access to healthcare, healthy lifestyles, and other pressing social issues. And after millions of dollars spent, the question always lingers about whether the time and resources spent had an impact.

So we did some research, tapping into our own measurement database, and interviewing leading evaluators and experts in the field. Here are a few of the ways that researchers and organizations are measuring this type of social change:

Contacts: This simple measure can often be dismissed as too "output-ty". Still, there is value in knowing how much support is being generated by a particular campaign. Moreover, while some legislators ignore emails or mail, many officials do ask staff to calibrate the level of correspondence for certain issues and ascribe importance to this form of constituent interaction. Contacts also don't have to be so arm's-length; the number, frequency and favorability of personal contacts, meetings and discussions can be a good indicator of a legislator's (or stakeholder group's) affinity to support a cause.

Secondary Impacts: Even if the policy itself does not change, there are many intermediate outcomes that can be affected, and measured. For example: has a constituency been developed; how much awareness or visibility of an issue was created; whether champions for an issue were developed; and how many of the key stakeholder groups have been engaged in support of the issue. Each of these can be easily measured, and together, these intermediate outcomes can be used to gauge progress toward changing public policy.

System Change Continuum: This approach, developed by Mission Measurement, takes the view that change as progressive, and measures movement along a standardized change continuum. At one end of the continuum there is research being done to understand a particular issue. At the other end, there is legislation that is passed, and enforced, to support a particular policy change. Many things happen in between - education, issue advancement, visibility, stakeholder engagement, public support, etc. Some organizations can measure their progress on social change by creating their own change continuum. The continuum can be calibrated, with hash marks along the way, from 1 to 10. 1 being status quo, and 10 being sustainable policy change. A policy expert inside the organization, or an outside set of stakeholders, can determine progress on a quarterly or semi-annual basis.

Bellweather Rating: This is a technique developed by the Harvard Family Research Project to track whether a policy issue is getting traction with legislators. I.e. is it on the "agenda"? To determine this, evaluators ask a number of unprompted questions to a sample of legislators, policymakers, media, think-tanks and thought leaders. For example, "what are the top 3 policy issues on the agenda as you see it?". The evaluator can then determine whether their client's issue shows up near the top. If so, the evaluator can determine whether there is some attribution between their client's work and the issue visibility.

The bottom line in all this is that measuring the impact of an outreach or advocacy effort, in any form, is possible with creativity and reflection. As our experiment evolves, we'll share what we've learned. In the meantime, know that we in the social sector are not alone in this quest: see this article in the Wall Street Journal on measuring the impact of blogs!