Why the Nonprofit Sector Needs Predictive Data

Social investing and philanthropy is one of the few sectors of the economy where we only measure after we invest.  Think about it: most investors on Wall Street would not take a position in a public company without first predicting the stock price using Bloomberg or Factset; most bankers would not issue a loan to someone without first running a credit score through Experian or Equifax; and most lawyers would not try a case without first researching legal precedent via Westlaw or Lexis-Nexis. But in the nonprofit and public sector we invest first, and then measure to see what we got.  That may be a key reason why we aren’t making as much progress as we could be.

Predicting the success of social programs before we fund them holds great promise for the future of social impact.

  • Rational resource allocation.  Policymakers, government budget analysts and funders could use data to predict the likely success of a prospective social intervention. As opposed to politics, cronyism and bad data like overhead ratios.
  • Benchmarks.  Standardized data for social programs will enable us to compare relative performance of one program versus another.
  • Level playing field.  Currently, only those programs that can afford to hire an evaluator and conduct a fancy evaluation are considered “evidence-based.”  Predictive data analyzes all programs on the same basis and uses evidence-based factors to determine which are most likely to succeed.
  • Attract private capital.  I’ve never heard of a bond being issued without a rating.  But social impact bonds are, and as a result, there is no ability to attract real risk-seeking capital.  With predictive data, real market-based investors could invest in philanthropy, and capital markets could emerge to fund social change.

Some argue that predictive data is irresponsible.  We can never be 100% sure whether the prediction is correct, so using this type of data would unfairly penalize certain programs.

Others argue that data doesn’t tell the whole story – there are many other “non-quantifiable” factors like quality of leadership, context, geography, etc. that go into determining a program’s success.

Still others say that predictive data would stifle innovation – because innovative programs have no track record they would likely come up short in any analysis.

Finally, many say that predicting social impact is just plain impossible.  It’s never been done before.  There’s no way to predict human behavior.

These are all reasonable concerns.  However, each argument fails on its merits.  The standard for predictive data is not perfection, it’s being directionally right.  Predictive data isn’t the only consideration in other sectors; it’s just one input into a calculus of professional judgment.  Think about it: when Moneyball was invented they didn’t fire all the baseball scouts – they just gave them an additional tool to use.  And predictive data is unlikely to kill innovation; indeed, it’s likely the opposite.  Currently, limiting funding to only ‘evidence-based” programs that have been fully measured and evaluated is limiting innovation. Predictive data levels the playing field among innovations and established programs by shifting the analysis to risk and expected outcomes, rather than how well the program evaluation was conducted.

Finally, with regard to the concern about impossibility: if we can predict outcomes in the fields of medicine, economics, finance, entertainment, law, sports and weather, we can surely find a way to predict outcomes in social change.  We have a number of things on our side:

  1. Advances in meta-analysis
  2. A robust evidence base
  3. Accessibility of database technology
  4. Data science and statistical modeling

The time is now.  The demand for data is growing. But if you think about it, data is most valuable ex ante¸ not post hoc. The analyses that government policymakers and funders are trying to accomplish these days are predictive:

“which program is most likely to produce the outcome we want?”

“which program is going to give me the best bang for my buck?”

“will this program achieve the outcomes we want?”

“what is the ROI for investing in this program?”

“how do we design a program to maximize its effectiveness?”

Professor James Heckman, one of the world’s leading experts on early childhood development and a Nobel Laureate at University of Chicago told me “I get calls every day from policymakers asking me ‘how do I design the best early childhood program?’ and I tell them ‘I don’t know, I just know that the one I studied worked.”  We need to crack the code on social impact.  We need to figure out what works.  And predictive data is the only way to do that.  It may not be easy, and it may not be perfect.  But it is possible.  And I would argue that we are more irresponsible as a sector for not trying to do it at all, than for trying and failing to get it right.

 

 

 

 

Why the Nonprofit Sector Needs Predictive Data

by Jason Saul – May 06 2016

Social investing and philanthropy is one of the few sectors of the economy where we only measure after we invest.  Think about it: most investors on Wall Street would not take a position in a public company without first predicting the stock price using Bloomberg or Factset; most bankers would not issue a loan to someone without first running a credit score through Experian or Equifax; and most lawyers would not try a case without first researching legal precedent via Westlaw or Lexis-Nexis. But in the nonprofit and public sector we invest first, and then measure to see what we got.  That may be a key reason why we aren’t making as much progress as we could be.

 

Predicting the success of social programs before we fund them holds great promise for the future of social impact.

 

  • Rational resource allocation.  Policymakers, government budget analysts and funders could use data to predict the likely success of a prospective social intervention. As opposed to politics, cronyism and bad data like overhead ratios.

 

  • Benchmarks.  Standardized data for social programs will enable us to compare relative performance of one program versus another.

 

  • Level playing field.  Currently, only those programs that can afford to hire an evaluator and conduct a fancy evaluation are considered “evidence-based.”  Predictive data analyzes all programs on the same basis and uses evidence-based factors to determine which are most likely to succeed.

 

  • Attract private capital.  I’ve never heard of a bond being issued without a rating.  But social impact bonds are, and as a result, there is no ability to attract real risk-seeking capital.  With predictive data, real market-based investors could invest in philanthropy, and capital markets could emerge to fund social change.

 

Some argue that predictive data is irresponsible.  We can never be 100% sure whether the prediction is correct, so using this type of data would unfairly penalize certain programs.

 

Others argue that data doesn’t tell the whole story – there are many other “non-quantifiable” factors like quality of leadership, context, geography, etc. that go into determining a program’s success.

 

Still others say that predictive data would stifle innovation – because innovative programs have no track record they would likely come up short in any analysis.

 

Finally, many say that predicting social impact is just plain impossible.  It’s never been done before.  There’s no way to predict human behavior.

 

These are all reasonable concerns.  However, each argument fails on its merits.  The standard for predictive data is not perfection, it’s being directionally right.  Predictive data isn’t the only consideration in other sectors; it’s just one input into a calculus of professional judgment.  Think about it: when Moneyball was invented they didn’t fire all the baseball scouts – they just gave them an additional tool to use.  And predictive data is unlikely to kill innovation; indeed, it’s likely the opposite.  Currently, limiting funding to only ‘evidence-based” programs that have been fully measured and evaluated is limiting innovation. Predictive data levels the playing field among innovations and established programs by shifting the analysis to risk and expected outcomes, rather than how well the program evaluation was conducted.

 

Finally, with regard to the concern about impossibility: if we can predict outcomes in the fields of medicine, economics, finance, entertainment, law, sports and weather, we can surely find a way to predict outcomes in social change.  We have a number of things on our side:

 

1. Advances in meta-analysis

2. A robust evidence base

3. Accessibility of database technology

4. Data science and statistical modeling

 

The time is now.  The demand for data is growing. But if you think about it, data is most valuable ex ante¸ not post hoc. The analyses that government policymakers and funders are trying to accomplish these days are predictive:

 

“which program is most likely to produce the outcome we want?”

 

“which program is going to give me the best bang for my buck?”

 

“will this program achieve the outcomes we want?”

 

“what is the ROI for investing in this program?”

 

“how do we design a program to maximize its effectiveness?”

 

Professor James Heckman, one of the world’s leading experts on early childhood development and a Nobel Laureate at University of Chicago told me “I get calls every day from policymakers asking me ‘how do I design the best early childhood program?’ and I tell them ‘I don’t know, I just know that the one I studied worked.”  We need to crack the code on social impact.  We need to figure out what works.  And predictive data is the only way to do that.  It may not be easy, and it may not be perfect.  But it is possible.  And I would argue that we are more irresponsible as a sector for not trying to do it at all, than for trying and failing to get it right.

“Pray” for Success: How to Take the Guesswork Out of Social Finance

by Jason Saul and John Hoeppner – Sep 21 2015

The high-profile failure of a Social Impact Bond (SIB) experiment at Riker’s Island begs the question: how can we de-risk investment in SIBs, and social programming generally?  SIBs are largely predicated on private investors assuming the risk that programs will succeed at producing desired social benefits.  In the Riker’s Island example, Goldman Sachs lost $1.2M and Bloomberg Philanthropies lost $6M because a juvenile recidivism intervention failed to work.  Some backers hailed it a success nonetheless, insofar as the government didn’t have to pay for the program.  But was it really a success?  Will investors be more attracted to SIBs in the future?

 

Investors can only assume risks that they understand, and quantify.  In the case of SIBs, it appears unlikely that Goldman, or other SIB investors were adequately able to evaluate the risk.  And if they did try to assess the risk, they certainly didn’t guess right.  While the program underlying the Riker’s SIB, Adolescent Behavioral Learning Experience (ABLE) intervention, was purported to be “evidence-based,” the designation remains fairly arbitrary.  What does it take to deem a program “evidence-based”?  Does the program need to be 20% effective?  50% effective?  80% effective?  Proven effective once?  And how reliable does the evidence have to be?  Indeed, a comprehensive academic meta-analysis of the program’s core model, Moral Reconation Theory (MRT), suggests that the intervention was not very effective.  In fact, the meta-analysis, based on 33 underlying studies, found an overall r-squared of .16 (a statistical measure of efficacy), indicating that the MRT intervention was barely effective in the past.  The researchers even suggest that the MRT was “more successful with adult than juvenile defenders” in institutional settings, such as Rikers.  Finally, the vast majority of the evidence on MRT was produced by Correctional Consulting, Inc. the for-profit company that invented the program.

 

So how can an investor, with no formal training in social science, be in a position to interpret fairly dense academic literature and make a judgment about the likelihood of success for a particular intervention?  They can’t.  No more than an investor can personally assess the underlying creditworthiness of an individual loan or mortgage.  These financial investors typically rely on a qualified third party like a rating agency or credit bureau to appraise that credit risk.  This is absent in the structure of SIBs.  The focus of measurement is ex-post, not ex-ante.  And therein lies a major challenge of SIBs preventing mass adoption.  The rigor, the measurement, happens on the back-end, not the front-end.

 

“This can unlock new pieces of funding, private capital especially,” said Jim Anderson, who leads the government innovation program at Bloomberg Philanthropies. “It also brings a laser-like focus to measurable data. Everyone has an incentive. We’re not doing this to feel good. We want positive results.”  The fact is, failures like Rikers won’t unlock new pieces of funding like private capital; it’ll scare them away.  And that is the entire proposition of SIBs.

 

Here’s what we can do about it:

 

  1. Bonds need ratings – standardized, independent, predictive assessments of risk and return. There is no other reliable way for investors to assume capital markets risk.  To date, SIB investors like Goldman Sachs have assumed philanthropic risk, and maybe some reputational risk.  But the only way that SIBs or any form of social finance can attract true risk-capital is if investors are able to pre-determine the likelihood of success for an investment.  With advances in meta-analysis and evaluation modeling, like the Impact Genome Project®  investors can derive directionally-right estimations of effect sizes of a social program, before it runs its course.

 

  1. Measurement needs to be done upfront, not after the fact. Most SIBs are designed based on a particular charity or social program that an intermediary has deemed to be “evidence-based.” This program is then brought forward to government supporters and financial underwriters by a SIB intermediary as the basis for a bond. Rigorous measurement happens after the program has been completed, to determine its effectiveness.  This may not be the most efficient way to design a SIB.  Witness Riker’s Island.  Ideally, a government agency would first identify the outcome that they would like to advance via a SIB.  Then the intermediary would partner with a third-party evaluator upfront to evaluate the best, most effective programs that can produce that outcome.  A program would then be selected based on its efficacy score and its cost-per-outcome, based on the broader evidence base of what works.  Governments should be purchasing outcomes, not programs that they hope will produce outcomes. See, e.g., our learnings from the State of Illinois Budgeting for Results Commission.

 

  1. Be intentional about the purpose of the SIB. We’ve seen a variety of explanations of why SIBs are being issued: to teach governments how to be accountable and results-driven, to test innovations that wouldn’t otherwise be funded, to finance the most effective interventions, or to shift the risk of social programs and tap into private sector funding.  The design of the SIB should be tied to its purpose.  If the purpose is to finance the most effective interventions, then this merits a careful upfront analysis of which interventions are indeed the most effective of all interventions ever studied. Finding a program that happens to have been evaluated once or is deemed “evidence-based” is not the same.  If the SIB is designed to test innovative approaches that are too risky for government to invest in, then programs should be evaluated for their highest impact potential regardless of track-record.  The ABLE program selected for the Riker’s Island SIB clearly didn’t fit either of these criteria: it would be hard to justify that this was the most effective recidivism program ever evaluated, and it would be hard to say that (with a goal of 10% reduction in recidivism) this program held the most innovative, breakthrough potential.

 

 

We believe that the future of SIBs and social finance is brighter than ever.  But if we are to create a true social capital market, we must be more rigorous, intentional and outcomes-driven than the current approaches.

 

Me to We: The Story of its Founding and Impact


About 10 years after Free The Children’s (FTC) launch in 1995, co-founders Marc and Craig Kielburger, struggled to find a sustainable funding source for their international development and youth empowerment nonprofit organization.  The creation of Me to We, a social enterprise, became the organization’s answer to securing sustainable funding.

 

In their search for a solution, Marc and Craig were fortunate to be mentored by billionaire Jeff Skoll, eBay’s founding CEO. Skoll (founder of Participant Media, an award-winning film and media business, with a social mission baked into its DNA) was using his fortune to incubate innovative ideas to create social impact.

 

Four years after starting FTC, Marc and Craig had incorporated a small company, Leaders Today, to run volunteer trips and leadership camps. At the time, the designation of “social enterprise” didn’t even exist.  In Leaders Today, Skoll spied an opportunity. He encouraged the brothers to grow Leaders Today into a social enterprise (a business with a social mission at its heart) that would turn a profit to provide a long-term, predictable source of funding for FTC. With Jeff as a mentor, Leaders Today transformed into Me to We, a fast-growing social enterprise.

 

Jeff provided start-up capital to help bring the vision to scale, including the building of the Bogani Cottages in Kenya’s Maasai Mara so adults and corporate travel groups had a place to stay when they visited. He also provided seed money to start Me to We Artisans to support the work of Massai mamas. The mamas are women in the region who have found much-needed employment and a reliable source of income, working on mostly beaded items that are sold through Me to We and its retail partners.

 

The vision was for Me to We to offer sustainable products and life-changing experiences that would forever change the way consumers shop, travel and learn. And Me to We would help support FTC’s transformative international development projects.

 

The funds Me to We donated to FTC enabled the non-profit to have an extremely low administration rate (10 per cent) compared to other Canadian charities.  Half of Me to We’s profits are given to FTC to ensure the sustainability of the nonprofit’s international development work. The other half are invested back into the social enterprise to ensure its own sustainability and growth. To date, Me to We has provided over $6 million in cash, and $10 million in in-kind contributions, to FTC.

 

The social enterprise continues to drive innovation.  For example, all Me to We retail products now come with a ‘Track Your Impact’ code. This is an online tool that shows consumers exactly what social impact their purchase had, and where the donation was delivered. And, the purchase of a Me to We backpack might fund education for a child in Sierra Leone for one year.

 

Me to We was created out of necessity and with great care. In countries such as the UK, charities can form separate for-profit trading companies to generate revenue to meet charitable goals. But in Canada, (unlike the UK), the social enterprise model is relatively new and is not yet recognized by the laws that regulate businesses and charities.

 

To ensure that the relatively new concept of social enterprise met Canadian legal requirements, the distinct and separate governance and legal structure of FTC and Me to We was reviewed and given formal approval by the Public Guardian Trustee of Ontario, as well as the Ontario Superior Court. These critical steps have ensured Me to We’s success to-date.  Me to We still maintains separate financials, governance, and headquarters from its charitable partner, FTC.

 

In an environment in which charitable transfers from governments—and public donations in Canada and many countries—are declining, social enterprises have the potential to provide a sustainable source of funding for non-profits like Free The Children so that they can continue to deliver significant social impact.

 

 

Me to We – An innovative Social Enterprise


Me to We – An innovative Social Enterprise

 

Over the past four years, Mission Measurement has conducted four studies on the work of Free The Children and Me to We. In addition to these studies on social impact, our organization has independently assessed the following questions:

 

  1. Is Me to We a viable social enterprise?
  2. Is it well run? Innovative? Sustainable?
  3. Does it offer a meaningful and differentiated value proposition for Free The Children?

 

Here are my findings:

 

Me to We is intentionally designed to be a financial support mechanism for the organization Free The Children (FTC). This commitment is both transparent and internally consistent with the way in which the programs and services are aligned.

 

Me to We has also created a new class and generation of consumer that is powerful and very influential. Their purchases in the West create transformational change in the lives of individuals and their families overseas. As a social enterprise, Me to We enables consumers to solve many problems that government and nonprofits cannot.

 

Me to We donates 50% of its net profits to its charitable partner, Free The Children. This funding supports the long-term, charitable goals of FTC. The remaining half of Me to We’s profits are invested back into the social enterprise to secure its sustainability and growth.

 

This specific model has drawn recognition, validation and support from renowned social entrepreneurs such as Jeff Skoll, founder of the Skoll World Forum and Participant Media, and Sir Richard Branson of the Virgin Group. Both men have served as mentors to the Me to We leadership team.

 

The studies done by Mission Measurement have shown the significant impact of Me to We and Free the Children, more broadly. One of the most powerful outcomes that we have seen is that engagement with the brand and the organization have raised youth’s awareness of socially conscious consumption and made them more critical consumers.  Our research shows that young people involved in Me to We and FTC are 40% more likely to reward companies that commit to solving social problems than youth who are not involved with the organizations; 50% of these youth identify as socially conscious consumers.

 

What I have found most compelling is how Me to We has created innovative for-profit initiatives that actually extend the social impact of Free The Children’s international development model.

 

For example, The Me to We Artisans line of products help support alternative sources of income for women in developing communities. The Track Your Impact code on Me to We licensed and handmade products sold in North America and Europe let’s you track in which developing community your life-changing gift was delivered.  Me to We takes the charitable model and makes it more sustainable. Non-profits can’t live off the leftovers of the economy. A leading social enterprise, such as Me to We, has shown how you can inject a stream of commerce into a charity.

 

Based on experience with, and study of, Me to We and its leadership, I believe that it serves as a model for best practices in the field of social enterprise.

Free The Children: An Organization that Drives Significant Impact on Youth


Over the past five years, our team at Mission Measurement has been studying Free The Children (FTC) as a charity and a social enterprise model that empowers youth—at home and abroad—to be the catalyst for social change.

 

We have conducted numerous studies and examined the opinions and actions of an array of stakeholders- from program participants to educators to corporate partners. We’ve also had the opportunity to travel to Kenya to see FTC’s international projects firsthand, and spoken at length to the staff, beneficiaries and community leaders. Both through our studies and in personal observations, FTC has demonstrated an innovative, sustainable and exciting model of social change.

 

In Canada, the U.S. and the UK, Free The Children runs an innovative year-long program, We Schools (formerly known as We Act), that nurtures compassion in young people and gives them the tools to create transformative social change—both globally and locally. We Day is the organization’s celebration of youth who have made a difference in their local and global communities. Both of these programs are funded by the organization’s corporate sponsors.

 

Overseas, Free The Children empowers communities to lift themselves out of poverty, using a holistic and sustainable development model, We Villages (formerly known as Adopt A Village.) Youth in Canada, the U.S. and U.K. raise funds to support FTC’s international work.  FTC’s projects are designed with an eye to local empowerment so that the communities are empowered to ultimately take control of their funding and maintenance.

 

Free The Children demonstrates significant impact on youth involved in its programming.  The facts are compelling. Our research shows that:

 

  • FTC kids have grit- they are encouraged to be more purposeful and successful in school.
  • They are also a lot more interested in school. They are more passionate and able to connect their curriculum to real-world events.
  • And they are more likely to be leaders across issues, and to vote.
  • These kids exhibit 21st century skills- they are more likely to work better in teams, lead projects, speak up, work in collaboration, all of which are critical professional development skills.

 

The organization is truly developing youth who are more likely to be successful in life. What’s more, our research has found that 67% of educators state that their FTC students experienced transformational growth in their ability to project self-confidence because of FTC.

 

Moreover, FTC Participants note significant changes themselves:

 

  • 82% of youth who have participated in the organization’s programming feel inspired to be a role model to peers or siblings.
  • 70% of these youth state that because of their involvement with FTC they have become strong leaders.
  • 81% of alumni state that their FTC experience has helped them identify career goals;
  • 62% say FTC played a role in their acceptance to college / university.
  • 63% feel more confident in their ability to graduate from high school

 

These results speak to the dramatic impact that FTC has on youth as a result of their involvement with the organization.  Foundations, governments, corporations, school boards and parents want to see meaningful change in youth.  These results speak to the significant outcomes that FTC delivers and how the organization positively impacts youths’ lives.

Giving Accountability


Today, The Wall Street Journal reported that charitable giving in the U.S. last year hit a record $358.4B.   It is certainly great news that Americans continue to be among the most generous philanthropists in the world and that our rate of giving, post Great Recession, continues to grow.  But, even as we spend more, important questions remain–What do we really know about the impact of our contributions?  And to what expectations should we hold our charities? As funders, we must demand better answers about the impact our contributions generate which, in turn, will help to set higher expectations for the sector.

 

Measuring impact is one of the greatest challenges facing the social sector, and we need to be thoughtful about how we define success.  Is it enough to know that an organization helped 450 homeless veterans in a given community to have a place to live?  Without further information, this metric provides little insight.  To really gauge the impact, it helps to understand what percent of those veterans are still homeless. Furthermore, we should assess how that rate compares to total community homelessness.

 

Moreover, we should understand how one organization’s success compares to the effectiveness and efficiency of other organizations.  The bottom line in deciding which charity to support, requires that we must do more than assess a range of statistics, and instead determine exactly which outcomes we need to achieve.  Recipients of your charitable giving should go beyond reporting how many individuals they’ve reached and clearly articulate the change in status, condition or behavior they are aiming to produce. Then they can regularly track their progress toward achieving those changes.

 

The reality is not every organization is reporting meaningful outcomes data yet.  So here’s what I recommend you do before donating. Understand exactly what changes the prospect organization is seeking to create, and review the evidence they’ve provided.  Ask questions. Are these goals realistic? How do they compare to similar programs in other communities?

 

For instance, imagine a literacy program in a distressed neighborhood that provides one-on-one sessions one afternoon a week during the school year.  Is it realistic to expect participation in this program to result in 90% of participants reading on grade level by the end of the school year?  What, if anything, about this program makes you believe it can achieve significantly greater impact than other literacy programs?  Does the program’s intervention have a greater frequency or duration than average?  Is the quality of the tutors that much better?  Is there a proven, evidence-based  model it is following?

 

There is so much need in the world, and so many nonprofits vying for your dollars. Yet, as we seek ways to invest in social change, we frequently are only provided with accountability metrics such as administrative costs or percentages.  Next year, rather than tracking charitable spending increases, wouldn’t it be great to quantify the return on the billions of dollars we invest in social change?

Why People Love Chipotle

by Leeatt Rothschild – Mar 10 2015

The recent New York Times article “At Chipotle, How Many Calories Do People Really Eat?”  (which reveals what we really eat when we dine at Chipotle) struck a chord with many readers- most lauded Chipotle’s tasty menu, some were surprised by the number of calories in favorite dishes, while others blamed consumers for making poor food choices.  The volume of responses highlighted an important point–consumers are passionate about Chipotle, and for that reason Chipotle’s lines are out the door, literally.

 

In fact, data explains the chain’s success.  A recent Mission Measurement consumer research survey identified how strongly Chipotle performs at delivering the social benefits that matter to consumers.  Of the 50 restaurants included in the study, Chipotle falls within the top quartile of brands that deliver these important social benefits.  So what are those benefits that underscore Chipotle’s high performance?

 

Chipotle serves healthy food at affordable prices, made of high quality and fresh ingredients.  Mission Measurement’s consumer research study found that these three benefits are most important in driving consumers’ Quick Service Restaurant (QSR) choice.  Moreover, Chipotle makes it easy for consumers to customize their meals so they can maximize their choice (i.e. order just one taco for portion control, no sour cream for fat count, spicy salsa for mouth burning joy), and ultimately satisfaction with their Chipotle experience.  It also enables consumers to make smarter decisions by transparently breaking down the nutritional content of meals (check out their easy-to-use nutrition calculator).  Transparency and customization were also identified as important benefits that consumers highly value.  For all these reasons and more consumers eagerly reward Chipotle with their lunch money!

 

It’s no wonder Americans love Chipotle.  Chipotle delivers on the very things that matter most to consumers: it serves flavorful food made with fresh, healthy ingredients, quickly, at palatable prices.  If that isn’t enough, it allows (or dare I say encourages) customers to customize their meal, so that they get exactly what they want.  If only all other Quick Service Restaurants would follow suit….

The Platinum Standard

by Jason Saul – Jan 16 2015

The rise of results-driven strategies like evidence-based policymaking, impact investing, social impact bonds, collective impact, shared value, and public value has been largely dependent on the premise that we can reliably measure social outcomes.  But evaluating impact is hard stuff – studies can take years to complete and cost millions of dollars for just one program.  Few can argue with investing in social programs that work, or the premise that government should use evidence for budgeting decisions.  But the dilemma facing policymakers and philanthropists remains: how can you be “evidence-based” if we don’t yet have an ample and actionable evidence-base?  Indeed, only a handful of “top tier” evidence studies exist.  The answer may be found in the burgeoning field of applied social research known as meta-analysis.

 

Many consider randomized control trials, or RCTs, to be the “gold standard” for evidence.  Randomization provides a high degree of proof for a very narrow set of facts: a particular program, under a particular set of conditions, for a particular population of people, at a particular time, made a difference.  But while the specificity of RCTs can make them very credible, their precision also creates a number of critical limitations.  Moreover, if the goal is to develop a universal evidence base for policymakers to use, getting there one RCT at a time could take decades and cost billions of dollars, even using low-cost RCTs. 

 

There might be another way.  Meta-analysis is a research approach that capitalizes on the value of multiple RCTs and brings predictive power to social impact.  Think of meta-analysis as conducting “research about previous research.”  Meta-analysis allows researchers to analyze and synthesize the characteristics of programs and the effects of those programs in a systematic, replicable manner (Lipsey, Effectiveness of Juvenile Justice Programs).  Meta-analysis can improve the inferential value individual RCTs.  According to RA Fisher, one the early pioneers of randomization: “when a number of quite independent tests of significance have been made, it sometimes happens that although few or none can be claimed individually as significant, yet the aggregate gives an impression that the probabilities are on the whole [better] than would often have been obtained by chance.”

 

We would not be the first sector to use analytics as a way to predict outcomes.  In fact, we may be one of the last.  Many other sectors – from finance, to healthcare, to music, to baseball – have successfully combined meta-analytic techniques with applied statistics and database technology to improve outcomes.  Credit bureaus such as Experian have developed statistical models to isolate the variables that are statistically correlated to loan repayment.  This capability has transformed lending, allowing lenders to successfully predict creditworthiness and financial outcomes, in turn enabling them to make more accurate and efficient decisions.  The Human Genome Project has aggregated RCTs and other genetic research to help scientists understand common diseases, design more effective medications, assess risks, and more generally predict health outcomes.  AndPandora Radio created a highly detailed analytical system called the Music Genome Project® that quantifies all musicological and experiential aspects of a song or musical work in a standardized manner within each music genre. Every song has its own “genomic imprint” and can be algorithmically matched to other songs with correlated attributes that are already favored by a listener; this “meta-analytical” approach enables Pandora to successfully predict which new songs will match a listener’s existing musical preferences.

 

Benefits and Challenges of Meta-Analysis

 

Indeed, one of the most powerful benefits of meta-analysis may be the ability to quickly and affordably create a “quasi” evidence base for all programs, even those that haven’t been specifically studied. According to Mark Lipsey, author of Practical Meta-Analysis, and a leading evaluator in the field of juvenile delinquency: “Evidence-based practice can be extended beyond brand-name model programs to those many local and home-grown programs that are more generic instances of program types whose effectiveness is adequately supported by research.”  In other words, meta-analysis has the potential to turn gold standard studies into a “platinum standard” evidence base.

 

Another benefit of meta-analysis is that it can help us identify the factors across contexts and programs that lead to differential success. This is critical. Because if we can derive the social impact “genes” that determine outcomes, we can more consistently measure and evaluate existing programs, and design better programs.  For example, in John Hattie’s “Visible Learning” –a meta-analysis on student achievement, he analyzes evaluations across many educational programs and academic contexts to understand the degree of influence that certain factors have on student success. His findings present an opportunity for educators and practitioners to improve student outcomes by introducing these proven factors into programming. By identifying the factors and contexts that lead to program success across a large number of evaluated programs, a properly done meta-analysis can lead to actionable, predictive conclusions about the likely effectiveness of programs not yet studied.

 

A growing number of researchers are using meta-analysis to codify the evidence in disciplines such as juvenile justice, prisoner re-entry, student achievement, service learning, after-school programming, obesity prevention, and more.  Aggregating this knowledge, codifying it, and making it available to policymakers could create a fast-track to the universal, predictive evidence-base we’ve all been waiting for.

 

No doubt, there are challenges to meta-analysis as well: the number of studies available may not be sufficient; they may be of poor quality; and effects may not be homogeneous, so that grouping different causal factors may lead to meaningless estimates of effects. Moreover, meta-analysis requires that the analyst make a number of consequential decisions that could allow biases to creep in: what kind of studies (or program evaluations) should be included in the meta-analysis? Should only “gold-standard” RCTs be included?  If so, how do we decide which ones meet that standard? How should outcomes be standardized so they are comparable across studies? How do we incorporate qualitative information into the analysis? Or should we? If these kinds of questions can be answered well, the widespread application of meta-analysis to questions of social policy will vastly expand the actionable evidence base beyond the relatively small number of narrowly-focused RCTs and existing meta-analyses available for most social outcomes.

 

We may get there sooner than we think…

 

The Impact Genome Project™

 

In 2013, Mission Measurement launched an ambitious, field-wide meta-analysis to codify RCTs, consider the results of other meta-analyses, and incorporate findings from existing quantitative and qualitative research in every field of social policy.  We call it the Impact Genome Project™ (IGP). The IGP was announced at the Skoll World Forum in Oxford on April 10, 2014 and later at the White House on June 25, 2014.  The promise of the IGP is to create a universal evidence base upon which we can derive predictive analytics for policymakers and practitioners. The IGP will do this by standardizing the definition of universal outcomes in every field of social policy; collecting, grading, and codifying the existing evaluation literature in each given field; and statistically deriving the efficacy factors that can help us predict the outcomes of any social program.  Using our universal evidence base, we can then derive benchmark data to encourage more efficacious and cost-efficient programming to create more meaningful impact throughout the sector. Policy makers, program directors, and philanthropic leaders alike can use this data to make more informed decisions about programming. Industry response to the IGP thus far has been very encouraging; a recent SSIR webinar on the IGP attracted over 2,700 registrations and generated 365 questions. Subsequent to this webinar, nearly 400 social-sector organizations have signed-up to participate in the IGP.

 

Genomes are already under development in a number of different fields, including critical human needs, education, science & technology, and international development.  The first genomes are expected to be announced at the Clinton Global Initiative Winter Meeting.  In all, there are twelve genomes in the IGP: Education, Critical Human Needs, Youth Development, Health, Culture & Identity, Criminal Justice, Economic Development, Sustainability & Environment, International Development, Science & Technology, Arts, and Disaster Relief.

 

The IGP is not without controversy.  Some have questioned the sufficiency of available evidence to produce meta-analyses in some fields.  Others have questioned the ethics of using predictive data in the field of social impact, or the IGP’s ability to recognize innovation in social programming.  Still, even with these caveats, the value of meta-analysis likely outweighs the alternatives: guessing, on the one hand, or waiting until we have “perfect” data, on the other.

 

Conclusion

 

Policy makers today need more than just a static library of RCTs to make good decisions; they need analytic tools that can help predict the outcomes of programs, compare benchmarks, and design more effective social interventions. Meta-analysis and the Impact Genome Project™ have the potential to create a step-change in the way we govern and invest in social change. Turning “gold” into “platinum” will dramatically lower the cost of evaluation for all social programs, help us identify the “gaps” in research where RCTs are best served, and make evidence-based policymaking real within our lifetime.

 

Jason Saul, Founder and CEO, Mission Measurement

 

Nolan Gasser, Ph.D., Chief Genomic Officer, Mission Measurement, Chief Musical Architect of the Music Genome Project, and Adjunct Professor in Medieval–Renaissance Music History, Stanford University

 

Randy Stevenson, Ph.D., Director of Data Science, Mission Measurement and Professor of Political Science, Rice University

Mission Measurement Trendletter – November 2014

Nov 12 2014

Defining and Predicting Hunger Outcomes

 

In addition to forming a Microfinance Genome Consortium, we also recently worked with the Clinton Global Initiative to launch a Hunger Genome Consortium. This is an ever-growing group, open to practitioners, funders and researchers in hunger-related fields. For more information or to join the group, contact us at impactgenome@missionmeasurement.com 

 

How Your Organization Can Attract More Year End Donations

 

Whether it is the spirit of the holidays or a personal squaring of accounts ahead of tax season that drives donations, nearly a quarter of household charitable giving happens between Thanksgiving and New Year’s. So during this opportune time for giving, how can your organization secure more funding by convincing donors that it best meets the needs they care about most? Jason Saul,”4 Ways to Sell Your Impact to Year End Donors” The Chronicle of Philanthropy 

 

Identify the Social Benefits that Truly Motivate Your Consumers 

 

Consumer demand is shifting, with consumers now rewarding brands that deliver social benefits. This profound change in the marketplace presents an opportunity for brands to market to consumers in new ways. How can you determine which causes and social strategies will actually move the business needle? Perry Yeatman and Sue Tobias, “Capturing Consumer Demand for Social Impact,”The Huffington Post 

 

What Research Predicts for the Future of Ready-to-Drink Beverages

 

Much has been made about consumers shifting away from traditional carbonated soft drinks. In response, large beverage companies are launching new products. Our research suggests that those that deliver fresh, healthy and natural benefits will win in the market. Matt Fisher, “Is There a Future for Carbonated Soft Drinks?”

Capturing Consumer Demand for Social Impact

by Perry Yeatman – Oct 23 2014

This article was originally published on The Huffington Post. 

 

My Mission Measurement colleague, Sue Tobias, and I were struck by a recent BCG report entitled “An Imperative for Consumer Companies to Go Green.” This report highlights the increasing consumer preference for “organic, natural, ecological, fair trade” and other socially responsible products. Notably it indicates that even in historically difficult growth categories, “responsible” products generated two-thirds of the growth.

 

This profound change in the marketplace presents an opportunity for brands to market to consumers in new ways. And leading brands, whether they sell clothes, cars, or sandwiches, are overcoming the difficulty of differentiating based on price and convenience by delivering products that offer meaningful social impact, in the form of intrinsic social benefits — those things that have a direct, positive impact on the life of that consumer, their family or their community.

 

The question is: how can you determine which causes and social strategies will actually move the business needle? It is a particularly challenging issue given so many CSR programs and cause marketing campaigns don’t. Indeed, this very challenge was forefront in our minds when Mission Measurement created the Social Value Index™ (SVI) — which provides unique and actionable insights to help companies better leverage social impact for business growth.

 

Our recent Quick Serve Restaurant (QSR) category research is a great example. Using discrete choice analysis, our review of 40 QSR chains quantified consumer demand for 57 benefits, both traditional (taste, price, convenience) and social (health, freshness). The results were telling –intrinsic social benefits accounted for 7 of 10 of the top benefits that influence consumer choice. Even more surprising perhaps was the fact that things like recyclable packaging, giving to charity and encouraging employee volunteerism mattered little to consumers. While these activities may have very legitimate purposes — like reducing costs or boosting reputation — they have little direct effect on purchase behavior in QSR.

 

Net net, those brands like Chipotle, Panera and Pret-A-Manger, which deliver benefits that improve the quality of their consumers lives and make consumers feel good about their meal purchases came out on top. Pret A Manger emphasizes its “good natural food.” Panera has successfully focused on using additive free ingredients and sustainably farmed chicken. Chipotle’s transparency about its supply chain brings credibility and believability to its brand purpose of serving “Food with Integrity.” So they were at the top of the QSR SVI. But more important than being on the top of the list, it’s no coincidence that these same chains are also those growing fastest. In fact, there is a strong correlation between a high score on the SVI, and revenue growth.

 

The bottom line is this: consumer demand is shifting, and consumers are rewarding brands for delivering a new category of benefits. This pattern is the same across categories ranging from ready to drink beverages to household cleaning products. Today’s consumers prefer brands that can make a meaningful social impact on their lives. And if you want to effectively capitalize on this, you must first identify which social benefits truly motivate your consumers.

Is There a Future for Carbonated Soft Drinks?

by Matt Fisher – Oct 20 2014

Much has been made about consumers shifting away from traditional carbonated soft drinks (CSDs) in North America.  Concerns about obesity, sugar content and safety of artificial sweeteners have all contributed to consumers seeking alternative options.  In response, the largest beverage brands have launched new products like Coke Life and Pepsi True.

 

Based on our just completed research, the Social Value Index (SVI) for ready to drink (RTD) beverages, these launches are a step in the right direction but don’t likely go far enough.

 

We looked at RTD beverages expansively, including other options such as juices, teas, flavored and bottled waters.  Additionally, we focused well beyond traditional benefits and closely tracked social benefits such as healthy living, environmental progress, community improvement and ethics.

 

In what is becoming increasingly common across our SVI studies, the role of social benefits is much greater than previously expected.  For example, in RTD beverages, social benefits drive 36% of consumer purchase behavior.  Even more, the most important social benefits such as having fresh, healthy and natural ingredients were as influential as traditional benefits such as taste and price.  With that in mind, it’s not surprising that CSDs were the worst performing beverage type.

 

So rather than focus on relatively minor changes to traditional formulas, we believe that CSDs must provide truly different options that deliver more fully on fresh, healthy and natural – brands like Pepsi’s Izze with 70% pure fruit juice or Spindrift which uses fresh squeezed fruit or berry purées. Consumer demand in the CSD category is changing.  To win, all brands will need to take note of this shift.

Mission Measurement Trendletter – October 2014

Oct 9 2014

Free SSIR Recording on Using Big Data to Predict Social Impact

 

Jason Saul, Founder and CEO of Mission Measurement, and Nolan Gasser, architect of Pandora’s famed Music Genome Project™ explain how big data increases the utility of measurement, reduces the time and costs associated with evaluation, and enables users to confidently predict the success of a program before it even launches.The webinar is free with registration. Click here to view the recording>> 

 

New Research Simultaneously Helps CMOs and Society

 

The increasing difficulty in differentiating a brand based on price, quality or convenience has created an opportunity for marketers to seek out new ways to make their products matter. One of the most promising new sets of benefits are intrinsic social benefits — those non-traditional purchase drivers that have a positive impact on the lives of consumers, their families or their communities. Perry Yeatman and Leeatt Rothschild, The Huffington Post 

 

A Simple Tool for Measuring Your Outcomes 

 

For organizations that have embraced outcomes measurement, one useful tool for tracking progress is the Success Equation. This tool is valuable both for communicating what you do and for helping your organization clarify its thinking and its work related to outcomes. Jason Saul, The Chronicle of Philanthropy

 

Sue Tobias to Speak at Cause Marketing Forum in Toronto

 

On October 28th, Mission Measurement Principal Sue Tobias will present “Straight Talk on Measuring Impact” to Companies and Causes Canada. Her presentation will explain how marketers can align cause-marketing efforts with consumer demand and business goals. She will also offer advice on how measure the ROI of social impact programs. Companies and Causes Canada

New Research Simultaneously Helps CMOs and Society

by Perry Yeatman – Sept 16 2014

This article was originally published on The Huffington Post.

 

As a CMO myself, I’m experiencing firsthand the profound changes in the marketplace and in how we provide value for clients, consumers or customers. Amidst these changes, one of the developments my colleague, Leeatt Rothschild, and I are most excited about is the growing importance of social value drivers for consumers.

 

Yes, the increasing difficulty in differentiating a brand based on price, quality or convenience has actually created a new opportunity — or dare I say a new imperative — for all marketers to seek out new ways to make their products matter. One of the most promising new sets of benefits are intrinsic social benefits — those non-traditional purchase drivers that have a positive impact on the life of a consumer, their family or their community.

 

Using discrete choice analysis, our research across categories ranging from Ready-To-Drink Beverages to Insurance to Retail Pharmacy has identified the most influential social value drivers per category. The big news is that in some categories, social benefits account for nearly 50 percent of consumers’ purchase decisions!

 

Many leading companies are already reaping the benefits of leveraging social value drivers – benefits that range from increased revenue to enhanced loyalty. Fox example, Chipotle has leveraged consumer’s desire to eat fresh food that is produced using sustainable practices and resources. Their Food with Integrity philosophy has helped increase annual sales by 44 percent (from $2.5 billion to $3.6 billion) in the past two years. Similarly, in the competitive global car market, electric car company Tesla has grown annual sales from $150 million to $2.9 billion, a whopping 1800 percent increase in the past two years.

 

Think about it this way… expanding on the healthier food example above… if my favorite quick serve restaurant (QSR) offers healthier fare and I opt for that when I eat there, I will feel better about my choice. I may also have more energy or lose weight. That’s a valuable social benefit to me. My family may also benefit by my better choices if they lead to improved health or longer life-span. And finally society may benefit by my potentially reduced medical costs and extended productive years. Of course this is an exaggerated example but I think you get the point. And if one individual can achieve these benefits, think about the impact this could have if every person who eats at a QSR in the USA were to do the same?

 

So you can see that these new social value drivers are powerful, which is why we CMOs must embrace them if we are to continue to build uniquely relevant and meaningful brands — brands that consumers prefer and repeatedly choose in this evolving market. And if consumers are more often choosing brands that deliver social benefits, not only would we benefit by offering them, but society at large would benefit as well. Wouldn’t that be the ultimate win/win!

Mission Measurement Trendletter – September 2014

Sept 11 2014

Forbes on the Impact Genome Project™

 

“Mission Measurement is pioneering the movement for change with the Impact Genome Project. Other organizations such as Charity Navigator are making strides with their efforts to evaluate philanthropic programs; however, Mission Measurement’s methodology is unique in how they use predictive analytics to measure success.” “Using Big Data for Social Good,”Forbes 

 

Consumer Research Quantifies Growth Opportunities for Wall Street

 

Companies like Chipotle, Whole Foods and Tesla have capitalized on emerging consumer demand that financial analysts have largely overlooked. In today’s challenging market, better consumer data is increasingly valuable, and understanding what truly drives consumer decisions is essential to forecasting growth.  Recent Mission Measurement research quantifies the nuances of consumer behavior in the quick–service restaurant category.“The Customer Knows Best,” John Hoeppner,Journal of Sustainable Finance and Banking 

 

5 Questions You Should Ask Before Launching a Cause Marketing Campaign

 

Frequently, cause-marketing efforts are chosen with too little consideration given to how they align with consumer demand and business goals. To avoid that pitfall, and successfully implement a cause marketing campaign that drives business and social outcomes, you should consider 5 key questions.  “There’s A Reason ‘Marketing’ is Part of Cause Marketing,” Sue Tobias and Leeatt Rothschild, Companies and Causes Canada 

 

New Research Identifies, Quantifies and Ranks Beverage Brands on Delivery of Social Benefits

 

Mission Measurement recently completed a study of the US market for carbonated soft drink, juice, tea, flavored water and water brands.  The study reveals how brands perform in delivering  social value to consumers and their potential for improvement.  To learn more, please email mfisher@missionmeasurement.com.

Using Big Data For Social Good

Aug 27 2014

This article originally appeared in Forbes.  Read the full article here.

 

Every day, thousands of Americans apply for new credit cards, loans, and mortgages.  In the decision-making process, banks use one number to review a person’s financial history and assess their likelihood to pay off debt: a credit score. Similarly, other industries are striving to mimic this approach by using algorithm-based data to predict future outcomes in various settings. Enter Mission Measurement, the social impact consulting firm attempting to change the way corporations, government agencies, foundations, and non-profits invest in philanthropic causes by using data to forecast social impact program outcomes.

 

Mission Measurement CEO Jason Saul and the architect of Pandora ’s Music Genome Project, Nolan Gasser, have designed the Impact Genome Project, which seeks to assess philanthropic program outcomes based on the composition of a program’s design. The project was inspired by the work of Pandora and by the NIH’s Human Genome Project, the NIH project that mapped human genes to thoroughly understand the functionality of human genes and predict health outcomes.

The Customer Knows Best: Wall Street Needs to Prioritize Consumer Research

Aug 4 2014

In July’s edition of the Journal of Sustainable Finance & Banking John Hoeppner, Mission Measurement’s Head of Investment Research, outlines why a deep understanding of consumer decision-making is essential to investment analysis. The article features research on how companies can deliver social and traditional benefits to drive consumer purchase behavior.

 

PDF: The Customer Knows Best: Wall Street Needs to Prioritize Consume Research

Mission Measurement Trendletter – July 2014

Jul 24 2014

4 Tips for Measuring Outcomes (Not Activities)

 

One of the most common frustrations for nonprofits is determining how to make use of limited resources to conduct evaluations that show an organization’s impact and meet grant makers’ expectations. As your organization moves beyond collecting data about activities to measuring value, these suggestions will help.“4 Tips for Measuring Outcomes Instead of Activities,” Jason Saul, Chronicle of Philanthropy

 

Get More Bang for Your CSR Buck

 

In our consumer insights research for numerous Fortune 100 companies, we’ve found that there is a fundamental difference between socially conscious consumers and the consumers that respond to “Social Value Drivers”—economically-valued consumer benefits delivered through social impact. By providing these benefits, you can stop spending money on causes you hope your consumers care about and start investing in social impact that measurably drives consumer behavior. “Harnessing Social Impact to Drive Consumer Demand,” Perry Yeatman, The Holmes Report

 

Groupon Head of Social Innovation, Patty Morrissey, On Corporate Responsibility

 

How a business operates and the impact that it has on society and the environment is more important than what a company does philanthropically. If I were to tell you about a global platform designed to connect millions of customers with independently owned businesses, that generates local jobs, local wealth and local prosperity, you might think that I’m describing a B-corporation social enterprise, but that’s Groupon. Increasingly, companies are developing unique strategies around their core assets and capabilities. Mission Measurement does a lot of great work around this, helping organizations identify and maximize the social value drivers that are inextricably linked with their business practices.“How Does Groupon Think about Social Innovation?” Forbes  

 

Why Social Impact Funders Need Universal Outcomes

 

A universal outcomes framework will help funders to organize grants against priority outcomes and analyze the impact of their work on a portfolio level.  Investors in social impact can better measure the resources dedicated to priority outcomes, review the relative contribution of each program in a portfolio to its relevant outcome, and understand overall portfolio performance. “The Value of Universal Outcomes,” Jason Saul, GMNsight

Mission Measurement Announces $1 Million Investment in Social Impact Genome at White House Impact Investing Conference

Jun 25 2014

 

Washington, DC, June 25, 2014 — Today, at a White House roundtable on impact investing, Mission Measurement announced a $1 million investment in the Impact Genome Project™. This was one of more than 20 new private sector commitments designed to drive $1.5 Billion+ into impact investments that intentionally generate financial return and measureable social or environmental impact.

 
The Impact Genome Project™ is a groundbreaking initiative designed to codify the factors that lead to success in every major social program category. The work is being led by Jason Saul, CEO of Mission Measurement and Nolan Gasser, former Stanford Professor and famed architect of Pandora’s Music Genome Project™ and it currently being piloted by leading foundations and governments.

 
The project leverages Mission Measurement’s existing database of more than 78,000 outcome data points and analyzes social impact programs from thousands of organizations to form its initial evidence base.

 
“The greatest benefit of measurement for policymakers, legislators, social investors, and philanthropists is before funding decisions are made, not after. Traditional financial investors use predictive data to make more informed decisions and to forecast expected risks and returns. Social investors deserve the same. If indeed we want to create a true market for social finance, social investors need data that allows them to compare investment opportunities, benchmark returns and determine “value.” That’s why we’re so proud to participate in this roundtable today at the White House and to announce our solution to this critical challenge.” said Jason Saul, CEO of Mission Measurement.

 

Read more about this event here.

 
About Mission Measurement

 
Mission Measurement Corp is the world leader in measuring social outcomes. With offices in Chicago and Washington, DC, Mission Measurement helps companies, governments and nonprofits make better decisions and design better strategies. Our aim is to use our proprietary methodologies and data to help clients achieve twice the social impact for half the cost. For more information, visit us at www.missionmeasurement.com or follow us on Twitter at @missionmeasure, @jasonasaul, @perryyeatman.