Measurement is Trending
Measuring outcomes and impact isn’t a new idea - impact statistics in annual reports have been a staple of the sector. What’s changed is the extent to which funders, governments, corporate partners, and other stakeholders are tying evidence of impacts to funding and resource access, and setting expectations about those impacts. In other words, the ability to ‘prove’ the increasing effectiveness of our programs is becoming necessary to ensure non-profit survival.
An Imperfect Model
This interest in linking non-profit work to measurable results has launched many of us into the uncertain territory of program assessment and evaluation. Overall, it’s a helpful shift; a good evaluation framework can help us better understand ourselves - how we operate as an organization, where we can lean into our strengths, opportunities for growth, and the ‘value’ our communities attribute to us. Together, these data help determine what’s working and what isn’t, which is how we continuously improve and innovate our programs (and raise money!).
The challenge is that funders and other stakeholders often use corporate frames of reference to set expectations about non-profit impacts. Since the fundamental goal of any corporation is to drive revenue, there is a bias toward quantitative (numbers-based) measures of impact (e.g., number of participants served), and often, financial measures (e.g., total fundraising dollars).
But the goal of a non-profit organization isn’t to drive revenue - it’s to fulfill a social mission connected to improving the quality of life for others in the community, province, country, or the world. Consequently, the numbers many funders and stakeholders ask us to focus on (e.g., finances, participation) do not accurately represent our impacts (i.e., our ability to fulfill the mission).
Being held accountable for suboptimal impact measures is more than an inconvenience; it pulls our focus away from our mission and redirects precious resources to achieve goals that do not bring value to the communities we support. Moreover, it can give a false impression of our (in)ability to deliver impact, because the measures being used to evaluate us aren’t the measures our organizations are set up to achieve.
Impact Measurement: From Liability to Superpower
There is good news amidst this tension: we have a wealth of data at our disposal (really!), which can help us redefine our impacts and re-educate our funders and stakeholders. In many ways, the current tension is an ironic example of intent versus impact: our stakeholders want us to succeed and are applying a model to help us get there. By approaching impact measurement as a productive tool instead of an evaluation of our worth, we can transform it into a superpower.
Getting Clear About the Terms
Before getting into how we ‘do’ impact measurement, let’s clarify the landscape. There are lots of inconsistencies in how measurement terminology is used, contributing to measurement angst. We have put together a glossary of terms for quick reference to help everyone speak the same measurement language.
Theory of Change (ToC): How change happens.
This outlines the relationships between the communities/participants we support, the intended outcomes we are trying to deliver, and how our programs/offerings produce those outcomes. The ToC tells the story of how we create change in the world, and should be specific enough to map the steps from inputs to impacts.
Logic Model: What the program does and how.
This is specific to a given program or set of programs (offerings an organization provides to create change), and shows how the program objectives, activities, and expected outcomes are connected. The logic model clarifies who will be served, by what/whom, the expected outcomes, and how those outcomes will be accomplished.
Inputs: Resources committed to the program.
These are the ingredients that go into the program - funding, time, staff, expertise, methods, facilities. Together, these bring our programs to life.
Outputs: What is tracked and counted.
These are the direct derivatives of the program activities - how many people were served, what activities were conducted, what services were provided. Outputs are typically quantitative (numbers) and easily measured (usually they are counted). It can be tempting to stop at outputs because they are easy to produce. But outputs are not equivalent to outcomes, and they do not guarantee impacts.
Outcomes: What the program produces/achieves.
Outcomes are meaningful changes for the population/communities served, for example, changes in knowledge, skills, attitudes, behaviours, living conditions, status, etc. These changes are directly linked to the program. They are a measure of the extent to which the program fulfilled its core purpose. Measuring outcomes requires more time and resources, but they are essential to connecting our programs to broader impacts.
Indicators: What helps the program to stay on course.
These are specific, observable, and measurable characteristics, actions, or conditions that show whether change has occurred. Indicators are the measures, which can be quantitative (numbers) or qualitative (observations, first-hand reports, testimonials, etc.) that show progress over the course of the program. Indicators give us information about how things are going before the program is over, which allows us to take action if needed. Positive indicators tell us to keep doing what we’re doing; negative indicators tell us to explore what’s going on and consider making changes.
Impact: What effect took place because of the program and/or the organization
Impacts are the longer-term ‘results’ that are attributable to the outcomes of a program and/or represent the contribution our organizations have made to broader changes (e.g., sector system changes, policy changes, etc.). Impacts are indirect effects of the program or organization - there are other factors that can influence them beyond our efforts. The fact that impacts are indirect measures emphasizes the importance of measuring program outcomes (the direct measures of our programs) so that we are held accountable for what we can fully control.
In our upcoming posts, we will look at how to measure impacts and how to report on impacts to fulfill both the needs of our organizations and the priorities of our stakeholders.
This blog post was written by Lindsay Valve, Lead Consultant at Quilin and member of the EPIC team.