One of the key types of projects I work on is the economic impact study, which looks at how government expenditure, regulatory changes, public policy or commercial decisions have changed or will change the welfare of the country. In order to carry out such an exercise, you must build a model which estimates economic activity (in terms of money spent and employment) under two scenarios – with and without the policy or commercial decision.
Building the model is not necessarily the hard part. Defining the two scenarios can take significant effort, in order to ensure that all potential impacts have been considered. This is particularly true when the exercise is designed to look at impacts in the future, rather than how things may have been different in the past.
It is usually relatively easy to define a scenario following the status quo, as most forecasts on demand, pricing and costs will have been carried out assuming no change in strategy. Even where no forecasts already exist, it can be relatively simple to trend existing data forward. However, understanding the counterfactual – the situation which does not exist – is more tricky. Assumptions must be made over how changes in policy or strategy will affect the market. These take the form of price elasticity of demand, consumer behaviour patterns, and assumptions over consequential actions so that a chain of impacts can be felt.
The BBC reported on a study carried out in 2012 examining the impact of removing tolls from the Severn Bridge, carried out by Arup. In this report, the counterfactual was driven by the idea that removing tolls would increase traffic, which would more than offset the reduction in revenue that the government would receive. However, there were a number of assumptions needed to come to this conclusion, including (but not limited to):
- Price elasticity of demand, to estimate the number of vehicles after the removal of tolls. This seems very high, with an increase of 12% after the policy change. Ordinarily, we would assume that road tolls like this were relatively inelastic – people do not usually have a choice as to whether they cross the bridge or not.
- A split of business and leisure traffic. This is important as the impact on the economy will be different for each group.
- A link between accessibility and productivity. Arup has used previous studies to estimate this link, and applied this to the increased business traffic.
- Estimates on the impact on business location and performance. It is unclear from the report exactly how these have been derived.
It is clear to see from this list that assumptions are the cornerstone of counterfactual analysis. Changes to assumptions can have significant impacts on the conclusions of a study – and as a result they can be the most effort-intensive part of such work.