Posted 18th November 2015
Last month, Daniel Fujiwara from Simetrica published ‘The Seven Principle Problems with SROI’ (SPP). The paper runs through seven separate points regarding SROI. The points are based on existing research in economics and policy evaluation and draw in particular on many parallels between challenges that have faced cost benefit analysis (CBA) and those which exist for SROI today. The paper is a welcome contribution to the ongoing development and discussions around SROI, and covers a range of issues within SROI, borrowing from ethics, economics and statistics to do so. Despite what some may infer from the title of the paper, it is not a line-by-line critique of the Seven Principles of Social Value.
Throughout the paper, Social Value UK (formerly the SROI Network) has been referenced and cited. Below is our response to some of the points arising.
1. Context – rigour and purpose
One main issue of concern for us is the potential for confusion between the current Guide to SROI and the Seven Principles of Social Value. When the Guide was originally written there were two audiences in mind. One was an organisation, probably relatively small, seeking to start to understand and manage the value it was creating. The other audience was practitioners, to enable them to use the Principles recognising that they would need to be applied with different levels of complexity depending on the audience and purpose. However, the Guide does not include examples to highlight what this variance of audience and purpose would mean in practice.
As specific examples, point 6 in the paper is headed ‘Valuation theory and methodology in SROI are outdated and incomplete’ and in point 5: ‘Statistical methods for inferring causality are problematic in SROI’.
Whilst the points made in SPP may be relevant for some audiences in relation to what is included in the current Guide, the Principles ‘Value what matters’ and ‘Do not overclaim’ are not being fundamentally criticised here. The paper also does not recognise that there are examples of these Principles in practice which are up to date and, whilst not meeting Daniel’s required level of rigour, are more than satisfactory to be useful and helpful to organisations making resource allocation decisions. For example, it can be very helpful for an organisation to ask people ‘what would have happened anyway?’ as this may provide insights that allow more targeted services to specific groups, in essence thinking about customer segmentation as a way of increasing value gained by customers.
Our major concern would therefore be that an attempt to improve guidance creates an additional barrier to organisations trying to create more value in their work, increases the perception that SROI has to be expensive and complex, and doesn’t recognise the reality that most organisations currently make resource allocation decisions on imperfect information complemented with a healthy dose of ‘gut feeling’. Making decisions which are informed by perhaps equally imperfect information but now also on social value can only increase transparency in the decisions.
Social Value International (SVI) has recognised for some time that there is a need for the current Guide to cover:
- the use of Principles for different audiences and purposes (which would include both more and less rigour in their application)
- more in depth guidance in general
As a further example Professor Adrian Henriques has also written a recent piece on the Principle of overclaiming in SROI. In it, he argues that for business it is the gross impact rather than the net impact which is more important. This would mean that debates about the level of rigour applied in determining a counterfactual risk miss the point. This discussion raises interesting issues in relation to a specific audience and purpose and although we would still argue that both gross and net impact should be reported, we agree that the issue Adrian raises is important.
SVI maintains a Methodology Subcommittee which has developed a work plan in consultation with members. In particular, the Subcommittee is preparing guidance by working through the Seven Principles of Social Value. As a result SVI has produced supplements on stakeholder involvement and materiality and the draft supplement on outcomes is currently out for consultation with members. SVI recognises the need for more guidance on valuation and overclaiming and would agree with Daniel’s points that more guidance is required. In September, SVI hosted a three day meeting with the World Business Council for Sustainable Development which brought together 22 experts in valuation to start work on developing this further guidance.
However, it is important that this is not read as a critique of the Principles and that the processes that SVI has in place should be recognised. Daniel raises his issues in the context of public policy evaluation, predominantly in central government. The need for more guidance would be useful for any other audience requiring a similar level of rigour. However it should also not be forgotten that policy is sometimes made without recourse to evidence based on a high level of rigour, and so developing an argument based on our existing guidance may still prove useful.
So whilst SVI acknowledges that this high rigour, more specialist use of SROI would be appropriate for public policy evaluation, we also argue that this is only one of a number of audiences and purposes that SROI is suitable for. There are many organisations who do use SROI, often on a recurring basis, because they find it ‘good enough’ for their purpose and audience. This can be seen from the range of case studies on our website, most recently from BT (British Telecom) and FLUPP, a non profit foundation in Brazil.
2. Context – the alternatives
Following on from this our related concern would be that reading the paper as a critique of the Principles of Social Value without the context of considering other approaches for accounting for social value is dangerous. The purpose of these Principles should not be forgotten – to reduce inequality and environmental degradation by taking into account wider value. This wider value is recognised by giving a voice to those affected by an organisation’s work, to enable them to express what those effects are and how important they are, therefore increasing accountability.
There are many reasons why society does not require organisations to be fully accountable for what happens to people as a result of their activities. The consequence is that organisations are able to decide how accountable they should be and to choose their own approaches, if any, to measuring the wider value they create and destroy. Those methods are variable, they lack the consistency that a principle based approach provides and can include approaches where organisations decide how they have affected people, to only account for intended positive outcomes, not to consider any counterfactual and so on.
The danger again is not that practise improves but the voluntary nature of managing and reporting on social value allows organisations to fall back onto approaches that have a lower level of rigour than even the existing Guide to SROI, to a level which fails to be accountable and therefore does not result in any changes in resource allocation decisions.
This point – that SROI is often better than the current practice of most organisations and most alternatives – is especially pertinent for point 2 (‘SROI is silent on the issue of interpersonal comparisons and perversely places greater weight on the outcomes of the rich). Whilst it is acknowledged that interpersonal comparisons can be problematic, and certain valuation techniques can mean that outcomes and interests of richer people are weighted higher, in a world where organisations are not accountable enough to non-financial stakeholders, whose voices go unheard, even current guidance on the principle of ‘Valuing what matters’ has to be a step in the right direction.
3. The Normative basis for SROI
Whilst this may seem somewhat abstract to many people seeking to account for and manage the social impact of their organisations, we would agree that this is an important issue. There is a normative positon that underpins the Principles of Social Value and therefore SROI as an example of their application. This is:
“to reduce inequality and environmental degradation and improve wellbeing by incorporating social, environmental and economic costs and benefits.”1
We would agree with Daniel that this could usefully be developed so that the basis against which judgements are being made is clearer and we have started to explore this issue with the aim of a more detailed normative position. Some of the approaches that are discussed in Daniel’s paper such as the agency-centred approach and welfarism are relevant to this discussion.
4. Other issues
Some of the other points are more semantic. For example in point 3 (‘SROI’s views on stakeholders can be too narrow’), Daniel argues that ‘non-use value, defined as ‘a value that I might place on an outcome even though I get no direct use or impact out of it’ is not included. Whilst there may be no examples where this has been included, and we haven’t checked, the important point is that if value is material and experienced by stakeholders, it should be included in an account.
We have written various articles on the risks of bias in the ratio and the danger of the ratio without inclusion of supporting information, which provides transparency in how it was calculated.2
Given the ratio is a calculation using underlying information, transparency means it can be calculated and presented in different ways and is then also easy to interpret. However, Daniel is correct that the guidance is to include negative outcomes in the numerator. We think that the guidance is clear on this, both in the calculation and in the discussion on outcomes.
Finally in relation to point 7 about which costs are included in the calculation, Daniel is correct that the SROI guidance is to include the costs necessary for the activity being considered. However if there are material costs being met by other stakeholders as lost opportunities these could be included as outcomes. We agree that more clarity on this would be helpful.