Posted 1st August 2012
Not all differences are the same
JEREMY NICHOLLS, 01 AUGUST 2012
One of the most commons things I hear about SROI is that SROI is only one approach to measuring impact. Whilst clearly true, SROI and other approaches are asking the same question – How much difference have I made? (Which could also be how much difference do I think I will make).
The more interesting question is to consider where and why there are differences. I think there are, at least, three types, relating to accountability, risk and rigour.
Accountability
There is a lot of focus at the moment on outcomes, on being clear what you are trying to achieve, on focusing on your objectives and so on. These are all good management practice, which can be traced back to Peter Drucker setting out Management by Objectives in 1954.
But they are not the same as accountability. How accountable you want to be is a choice and one way of thinking about this is across a spectrum. At one end we want to be accountable for our performance compared to our objectives, at the other end we want to be accountable for the effect that our actions (or pursuit of our objectives) have on all those affected.
- Accountable for performance against objectives
- Accountable for intended positive outcomes experienced by target group (services user, customer or beneficiary)
- Accountable for intended and unintended positive outcomes experienced by target group
- Accountable for positive and negative outcomes experienced by target group
- Accountable for positive and negative outcomes experienced by all those effected
It is not possible to be accountable for all outcomes and so in all these, except for the first one, some way of deciding on the important outcomes is required.
The language for this decision in financial accounting, sustainability reporting and SROI is materiality. In some ways this is the most important issue as it this decision that determines what will be measured. It is not an issue for an organisation restricting accountability to its objectives as the decision of what is important or what should be included has already been included. Corporate sustainability reporting and SROI focus on the last one in the list, being accountable for material positive and negative outcomes experienced by those affected.
This doesn’t mean that an impact report may not end up only reporting on intended positive outcomes because none of the other outcomes turn out to be material – but the reader should want to know how this decision was reached.
So there are other approaches which take a different level of accountability as their starting point.
Risk
This is the risk that your analysis of your impact got it wrong. I often ask groups what questions they would want to ask in order to be confident that they know how much of a difference they have made or how much change they have created. Generally people come up with a very similar set of questions very quickly
· Who has changed?
· How have they changed?
· How much change was there which can be attributed to our actions?
And with more discussion
· Which changes are we going to be accountable for (see above)?
· Who answered these questions?
· What common measure across different changes
· What do we want to use our understanding for?
If any of these questions have not been answered there is an increased risk that your understanding of how much difference you have made will be off the mark.
If we miss out people who have changed, if the changes are not well understood, if the changes were caused by other factors, if we include unimportant changes but exclude important ones, of we don’t involve those experiencing the changes in making decisions – then the risk increases. And, of course, if we want to know how much change then we will have to quantify the amount of change somewhere in our understanding.
And different approaches will answer some or all of these questions. Those that answer fewer have more risk of being off the mark.
There is a common confusion in here when comparing SROI (or other frameworks) with methods of measuring specific changes. SROI is an approach that helps you decide what to measure. There are many appropriate tools to help you then measure. LM3 for example is a way of measuring local economic impact and is different to SROI because they are not in the same class of things.
At some point it becomes apparent that what the information is going to be used for has implications for how these questions are answered. There is a big difference between reporting to users, investors and the public on how much of an impact we have had and on using information to make decision to create more of an impact. Making decisions will mean making choices between different options, different services and different ways of designing services. And this means having a way of comparing amounts of different types of change – a single yardstick. Any organisation that makes choices is doing this, either transparently or implicitly.
Which brings is on to the most commonly perceived difference between SROI and some approaches to impact measurement (though not cost benefit analysis and financial accounting which are also both types of impact measurement) – the question of valuation.
Valuation helps the materiality decision by considering how important outcomes are compared to other outcomes. There are other approaches to this, for example Most Significant Change. But valuation also provides a single yardstick to frame a discussion about changing what an organisation does and make the discussion more transparent. Approaches to measuring impact which are designed for external reporting and communication do not need a single yardstick
Rigour
Linked to the question of purpose is the question of rigour. For some the examples of SROI that they read are not rigorous enough and for others they are too demanding. The reliance on available examples is understandable but misleading. The questions that need to be answered to assess impact can be answered at different levels of rigour depending on the purpose. For example considering what might have happened anyway (benchmarking, counterfactual) might range from asking your stakeholder to a randomised control trial. Producing a report which may be publically available is one purpose. Thinking about value in your organisation is another. There are approaches which set out their stall at a particular level of rigour. The SROI networks stall has been that:
· you decide what level of rigour you need for your purpose
· that there is a level of rigour required to become recognised as an accredited practitioner which is set out in an SROI analysis
Sometimes the level of rigour will be the same as approaches which have a predetermined level of rigour, sometimes less (SROI is too simple) sometimes more (SROI is too hard). Currently the SROI network is developing guidance on different levels of rigour for different purposes.
In conclusion
So yes there are different approaches but we should be clear about what type of difference we are talking about.
· How accountable is the approach trying to be
· How much risk is there as a result of the questions the approach is or is not answering
· How much rigour is required for the audience or purpose
Decisions on accountability do not preclude any organisation considering these basic questions. Whilst a small organisation will have less resource and capacity than a large organisation and this may affect the rigour with which these questions can be answered., those involved can still reflect on them when making decisions.
SROI turned these questions into a set of principles.
Understand change – that will be positive and negative changes for all those effected, what the changes are and how much of them occurred
Only include what is material – since the approach is aiming for wider accountability and needs a decision on what to include
Value what matters – to help with materiality, to help people make decisions (and to reveal the value of those whose value is not recognised but that’s another story)
Don’t over claim – to recognise that not all the change will be caused by our activities
Involve stakeholders – in order that judgements are informed by (not led by) stakeholders
Be transparent – since you will have made some judgements
Verify the result – because you are an optimist trying to make the world a better place
There is one big difference between SROI and other approaches to impact (although not with financial reporting) – the need for assurance. Both financial reporting and SROI recognise that a process of providing information that is good enough to be used to make decision and that has been based on the application of a set of principles will have required judgements. Human beings are optimists, and this is a good thing, but we need someone to reign in optimism at times – to give others the assurance that our judgements have been reasonable.