Hello, I am Barbara Klugman, freelance evaluator, and initiator of the OH community’s blogs.
I’ve been grappling with the question of when rating significance of outcomes adds value to the users. When for my client, ‘significance’ correlates with ‘type of outcome’, there’s no added value in coding for significance. For example, if their highest goal is influencing a policy, then that change has the highest significance.
However, I had a client with policy changes as their high level goals, but whose priority that year was to get farmers to organize and speak for themselves. This was the critical lever for influencing policies appropriate to the farmers’ context. So any outcomes of farmer groups advocating were considered the most significant achievement.
Significance for each programme of another client, WIEGO, is quite different – for one it’s the establishment of international networks of informal workers; for another it’s national governments including informal workers in their employment statistics; for another it’s urban policy addressing the needs of informal workers. They generate many outcomes, and are frustrated by treating all outcomes as equally important. So, we are exploring categorising each outcome’s significance using Outcome Mapping’s concepts of ‘expect to see’, for outcomes just beyond the intervention’s sphere of control that should be influenced if the intervention is done well; ‘like to see’ – very significant, and ‘love to see’ – extremely significant, for outcomes far beyond their immediate influence because there are multiple actors and contributors. Each programme will use these in relation to their programme’s intended outcomes. This should allow recognition of most significant outcomes of very different types across the organisation.
I proposed the same approach to Gender at Work (G@W) to assess the support they were giving to research groups funded by IDRC to give greater attention to gender. For most of the research groups, success would be uptake of their research in academic journals or by others using the research findings for advocacy. But for G@W’s theory of change, the minimum ‘expect to see’ significance was that their support would influence research groups to strengthen their attention to gender in that project. What this looked like would differ depending on where they started. G@W’s ‘like to see’ was that the research group would apply gender-related insights gained while working on the specific project, to their overall approach to research. G@W’s ‘love to see’ / extremely significant was that research groups’ processes and findings related to gender were picked up and used in communities and influenced decision-makers. ‘Significance’ was used to look at what was considered significant in relation to G@W’s brief, irrespective of the intended outcomes of the research groups with their diversity of topics and objectives.
Do any of you categorise significance? If so, do you use ‘minor, medium, major’ and what criteria do you use to decide on these? Is it useful? Why? Any other approach? And if you have any other OH experiences to share or raise, please do write a blog – 500 words, and send it to me at bklugman@mweb.co.za.