By Chris Allan and Atalie Pestalozzi

You’ve probably seen them: those reports using outcome harvesting with beautiful graphs of the outcomes, like these two examples from recent reports.

Most Frequently Reported Outcomes by Program Graduates

How useful are these graphs, since Outcome Harvesting is primarily a qualitative method?

Counting different types of outcomes shows patterns of outcomes, which can be helpful for understanding where the program has been successful, which groups have been influenced the most, which areas are missing, and so on. And this can help construct narratives about what went right and what needs more attention. So, when nuanced narratives about these patterns explain the graphs and the work, then we have some solid evidence to go on.

But when we present this counting as we would quantitative results, there’s a risk that people will read more into them than is actually there. Here are some of the pitfalls:

1. Not all outcomes are the same. There are vast differences in significance across a set of outcomes. A recent evaluation we did produced one outcome where a major government policy changed, and another where an African community-based organization had received a small grant. The first we ranked as “high significance” due to its wider influence, the second “low,” but we counted both in the numbers of outcomes. A graph of all outcomes treats them with the same weight.
   
   
2. Readers understandably want to see a statistical analysis. Tallying up the number of outcomes says little about program influence. It is common to present outcomes in a graph like the one below. It implies that there were more than twice as many outcomes in changed discourse than there were in changed practices (51% vs. 20%).                                                                                                                                               Does this mean the program was twice as effective at changing discourse than in changing practices, as the total outcomes in each category would suggest? As a pattern yes, but as a statistical measure of effectiveness, the data just does not support that kind of conclusion.
   
   
3. Sometimes you lump, sometimes you split. Sometimes there are disagreements among an evaluation team about what changes to lump together, and which to report separately – “lumpers” and “splitters.” In a recent foundation evaluation, one grantee reported dozens of related outcomes for which the foundation’s contribution was minimal. If we took each individual outcome, it would have increased the outcomes we had for the whole program by about 25%, vastly overstating the grantee’s contribution to one small aspect of the program. By grouping that large number of related outcomes into a couple of outcome statements, we represented more accurately the contributions of that grantee, and were able to represent what the program had done more realistically.

4. Counting doesn’t capture the story. One of the most valuable things about Outcome Harvesting – and one that we hear regularly from clients – is how outcome statements help tell the story of what happened. Somewhat like a mini case study – it generally only takes a couple of typical outcome statements to show how change was achieved in each category. If you turn outcomes into a number, you risk losing the story.

   So what’s a harvester to do? See our next blog on how we address these issues.