By Alnoor Ebrahim, Associate Professor of Business Administration, Harvard Business School, and V. Kasturi Rangan, Malcolm P. McNair Professor of Marketing, Harvard Business School
Conventional wisdom in the social sector suggests that one should measure results as far down the results chain as possible, counting not only the goods and services produced by an organization (outputs), but also their long-term effects on the lives of people (outcomes) and on society (impacts). Yet it is worth considering whether, and under what conditions, such measurement makes sense.
Take, for example, an international NGO such as the Red Cross or Doctors Without Borders that is engaged in emergency relief work. Measuring the work of such an organization is conceptually fairly straightforward: count the timeliness and delivery of emergency supplies such as tents, food, water, and medical supplies, as well as the numbers of people reached. Emergency relief is typically measured in terms of the delivery of outputs, as it is focused on meeting immediate survival needs rather than long-term development goals.
Outcome measurement, on the other hand, requires answers to a more complex causal question: Are the outputs leading to sustained improvements in the lives of affected people? An emergency relief organization that has done excellent work during a natural disaster might still fall short on outcomes of resettling people displaced from their homes and rebuilding their livelihoods. But that’s not necessarily its job, and it would be a mistake to assess the performance of emergency relief in terms of outcomes.
Beyond outputs
So when should an organization step forward into the domain of outcomes and impacts? Two organizations that have confronted this problem in different parts of the world are Aravind Eye Hospital in southern India and the Harlem Children’s Zone in New York City.
In 2012 alone, Aravind Eye Hospital performed over 340,000 surgeries, most of them for cataracts in one province in India; and it screened over ten times that number of people. Its outputs are remarkable: providing vision correction to over 3 million individuals since its founding in 1979. What really matters is not the number of patients screened or treated (outputs), but how many of those patients are satisfactorily cured (outcomes). Without a high-quality intervention, these outputs do not convert to outcomes.
Aravind engineered a superb operational process by which the surgery was performed at the highest quality. The rate of complications year after year were less than half that of hospitals in the United Kingdom. With such a high quality of service, Aravind was able to establish a tight causal linkage between outputs and outcomes.
We may never know whether those outcomes result in poverty reduction (impact). The organization does not measure impact—that individuals with recovered eyesight from cataract treatment will be able to lead productive lives once again and thereby contribute to society. While this causal assumption seems reasonable, the organization has cautiously stayed away from making that leap. Why? Because doing so would distract it from its highly focused goal—to improve vision, regardless of economic impact.
We may never know whether those outcomes result in poverty reduction (impact). The organization does not measure impact—that individuals with recovered eyesight from cataract treatment will be able to lead productive lives once again and thereby contribute to society. While this causal assumption seems reasonable, the organization has cautiously stayed away from making that leap. Why? Because doing so would distract it from its highly focused goal—to improve vision, regardless of economic impact.
In contrast, there is Harlem’s Children Zone (HCZ) in New York. HCZ offers a broad scope of activities aimed at developing a child’s capabilities through various stages of educational development. The organization runs pre-natal programs for parents; early childhood and after-school programs for children; charter schools, vocational training, and college preparation for youth; health programs to help families address asthma and obesity; and even programs for adults that aim to create a supportive and caring community for children.
Unlike Aravind, which focuses on a singular intervention, HCZ attempts to provide a “pipeline” of interventions “from cradle to college to community” that collectively enhance the chances of a child in Harlem making it from school to the workplace and on to self-sufficiency and responsible citizenry. HCZ’s “theory of change” is that educational support has to be continuous from pre-school through high school, and it has to be supplemented by extracurricular and community support to solidify the young person’s all-round development.
In order to provide this pipeline of comprehensive interventions, HCZ has concentrated its activities in a narrow geographical region of nearly 100 city blocks of Harlem, under the assumption that it will be better able to control the child’s overall environment. In 1998, before HCZ scaled its programs across Harlem, 60 percent of the children lived in poverty and only 20 percent of children in elementary schools were able to read at grade level. In 2011, the sixth graders in its two main charter schools had shown significant improvements: approximately 80 percent were at or above grade level in statewide math exams, and between 48percent and 67 percent (depending on the school) were at or above grade level in English. Moreover, 95 percent of seniors in public schools who attended HCZ after-school programs were accepted into college. The grade level metrics are primarily output measures, while college acceptance may be considered an outcome measure.
The time horizon for these interventions is long (five to 19 years), and the organization is undertaking longitudinal studies to better assess its results. Even then, drawing a causal link between HCZ’s interventions and longer-term outcomes such as lifetime incomes of its graduates, and impacts such as a decline in poverty in Harlem, remains complicated due to numerous social and economic factors that HCZ cannot control.
Lessons in impact measurement
What can we learn from these two examples? A key distinction between Aravind and HCZ is how they make the leap from outputs to outcomes. Aravind can measure outcomes because of the tight causal linkage between its outputs (corrective surgery) and outcomes (quality vision), which hinges on the quality of its surgery. HCZ can measure outcomes because it is able to vertically integrate a comprehensive set of interventions (a pipeline from cradle to college) that it controls in a tightly bounded geographic space (100 city blocks).
The time horizons for achieving outcomes can vary considerably. While Aravind may be able to gather evidence of improved vision fairly soon after surgery, HCZ’s interventions require a decade or more to see their outcomes materialize.
More generally, measuring outcomes is possible under two conditions that are uncommon in the social sector:
- The organization implements a narrow scope of activities where the causal link between outputs and outcomes is clearly established through evidence; or,
- The organization implements a broad scope of activities that is vertically integrated to increase control over outcomes.
All too often, social sector organizations seek to measure, or take credit for, results that extend well beyond the scale and scope of what they actually do. Sometimes they are pressured to do so by their funders or investors. At best, this is a misallocation of scarce resources. At worst, it is a deception that fails to make progress on the problems they are trying to solve. Their real challenge is a strategic one: being precise about their end goals, and clarifying how the scope of their activities will get them there.
The authors serve on the faculty of the Social Enterprise Initiative at Harvard Business School. This post is based on their article, “What Impact: A Framework for Measuring the Scale and Scope of Social Performance” in California Management Review (Spring 2014, pp. 118-141).
- See more at: http://tcbblogs.org/philanthropy/2014/07/22/what-impact-resist-taking-credit-for-results-you-cant-achieve/?sf28848245=1#sthash.qevphSDc.oxfdehTg.dpuf
No comments:
Post a Comment