Heterogeneous Impacts in Voluntary Agreements: A Changes-in-Changes Approach to the UK Climate Change Agreements

Abstract

The limited microeconometric evidence on the efficacy of environmental Negotiated Agreements (NAs) is an obstacle to both their introduction and effective design. We help fill this gap by providing evidence on the impact of the second Climate Change Agreements (CCAs) on business electricity consumption and employment. The CCAs are NAs offering a reduction on the Climate Change Levy (CCL), an energy consumption tax, in exchange for commitments to improve energy efficiency. We use the novel changes-in-changes method to account for heterogeneity in treatment effects. Our results indicate that the second CCAs yielded improved outcomes compared to the counterfactual of full CCL with an average reduction of − 4.81% in electricity consumption. They also reveal the importance of allowing for heterogeneity, as the impact on electricity consumption at the identified deciles varied between − 9.33 and 12.54%. This is a marked difference from the first CCAs which were found to increase consumption. The heterogeneity in treatment response is corroborated when extending the study to two large industrial sectors in the sample and when studying firms selecting differing target reporting methods. Confirming the findings from earlier studies of the first scheme, our results indicate a non-statistically significant reduction in employment, about − 4.6% on average, for the second CCAs.

1 Introduction

The use of voluntary agreements in environmental regulation has garnered substantial interest from academics and policy makers alike. When enacting regulation policymakers need to balance private and public welfare; overly stringent regulation could harm businesses whereas lax regulation would not adequately accomplish society’s environmental goals (Pearce 2006). Drafting the optimal policy is made difficult by information asymmetries between regulators and firms that are costly to remove and not necessarily in the interest of firms to resolve (e.g. Montero 2000; Bellassen and Shishlov 2016). Voluntary agreements (VAs) intend to achieve environmental policy aims without mandating participation. Participants are oftentimes given flexibility in how emissions abatement is achieved, letting them choose the path that minimises compliance costs (Segerson and Miceli 1998). Negotiated agreements (NAs) have a similar structure, as firms take part voluntarily, but involve negotiation of targets between the government and industry stakeholders with potential mandatory regulation as an alternative (Bailey and Rupp 2006; Fleckinger and Glachant 2011). Despite the prevalence of NAs in modern environmental policy making, their track record in terms of efficacy is mixed. We provide evidence of their efficacy by assessing the impact of the second UK Climate Change Agreements (CCAs).

The CCAs were introduced in 2001 with the Climate Change Levy (CCL), an energy tax levied on most non-domestic, i.e., non-household, energy consumption.¹ In order to avoid disadvantaging energy intensive sectors against international competition the CCAs reduce the CCL rates in exchange for a commitment to meet negotiated energy efficiency targets. Participation in the scheme is voluntary and policy targets are negotiated between the government and sector bodies, which also monitor the performance of participating firms. The first CCAs ended in December 2010 and were followed by the second CCAs in April 2013, which are planned to continue until 2025. A thorough discussion of the CCAs/CCL package is provided in Sect. 2.

This paper contributes to the existing literature by building on the quantitative evaluation of energy and environmental policy in three ways. First, this work measures the impact of the currently unstudied second phase of the CCAs² on electricity consumption and employment, where we find evidence supporting the efficacy of negotiated agreements. Second, we use a matched data set containing facility-level electricity consumption based on meter-level data for the second CCAs; hitherto available studies of the CCAs are restricted to plant level data, that can contain a multitude of facilities, of the first phase.³ Third, it conducts a quasi-experimental evaluation using the changes-in-changes (CIC) estimator developed in Athey and Imbens (2006), which estimates heterogeneous treatment effects non-parametrically across the quantiles of the outcome distribution. The CIC estimator has seen seldom prior application and, to the knowledge of the authors, none in evaluating energy policy.

Many structural features of the CCAs contribute to potential heterogeneity in its impact. Sector-wide efficiency targets were negotiated between the government and trade bodies, thus it is safe to assume that the stringency of different sector-wide agreements cannot be assumed identical. In a similar fashion, facilities in the policy are heterogeneous in size, which can affect feasible abatement options. Furthermore, participants are able to choose the type of targets⁴ used to assess compliance with the policy. Thus, understanding how different aspects of the policy affected participants is valuable in uncovering its real impact. The CIC estimator retains this nuance in the treatment effect by evaluating treatment effects across quantiles of the outcome distribution. Furthermore, the CIC requires only an outcome variable and a treatment indicator, which is beneficial here due to data matching reducing the observations available. These advantages of the method ultimately motivated the use of the CIC estimator.

We build on the statistical evaluation of the first phase of the CCAs conducted in Martin et al. (2014) by focusing on the heterogeneous impacts of the second CCAs. Our results indicate that the outcomes of the second CCAs have markedly improved compared to the first, as electricity consumption decreased by a statistically significant − 4.81% compared to a full CCL counterfactual. This result stands in strong contrast to the findings in Martin et al. (2014), which showed that the emissions and consumption targets in the first CCAs did not provide sufficiently strong incentives for firms to reduce electricity consumption [See also Ekins and Etheridge (2006)]. On the other hand, our results indicate no statistically significant impact of the second CCAs on average employment, mirroring the results in Martin et al. (2014). With regard to the heterogeneity of the impacts on electricity consumption across different sized participants, the estimated treatment effect ranges from a − 9.33% decrease to a 12.54% increase. In most deciles of the distribution, participation in the CCAs reduced or not significantly affected electricity consumption compared to the CCL. Therefore, the policy achieved similar policy outcomes to the levy without adding to the tax burden of participating firms. Similar results are found when estimating the what-if scenario of the CCL control group participating in the CCAs.⁵ This implies that CCAs, or a similarly structured NA, could be an attractive alternative to price-based instruments when the dominant objectives are to encourage reductions in emissions or energy consumption while minimising additional costs for firms. This might be particularly relevant in those instances where opposition to price-based instruments is strong, for example, due to concerns related to international competitiveness.

We also expand the analysis to different policy target types and individual sectors. The treatment effect within target types is mixed regarding heterogeneity, while outcomes between target types differ strongly. Results of the two sectors studied, henceforth Sector A and Sector B,⁶ provide an interesting contrast. Sector A, consisting of predominantly small and some large electricity consumers, mostly increased electricity consumption in the CCAs; however, some very large facilities decreased their consumption skewing the average impact of the policy downward to a 0.93% increase. Sector B, in comparison, are medium to large scale users that consistently decreased their electricity consumption in the CCAs by − 13.9% on average.

The remainder of the paper is structured as follows: Sect. 2 discusses the CCA/CCL policies, Sect. 3 frames the CCAs within the wider literature of VAs/NAs and provides select econometric evaluations of VAs/NAs, Sect. 4 elaborates on the changes-in-changes estimator, Sect. 5 describes the data and related matching procedures, Sect. 6 presents the results, and, lastly, Sect. 7 concludes the paper.

2 The Climate Change Levy (CCL) and the Climate Change Agreements (CCAs)

The Climate Change Levy (CCL) is a tax charged on energy consumed in industry, commerce, and the public sector introduced on April 1st, 2001.⁷ The aim of the policy was improving businesses’ energy usage and stimulating investment in energy efficiency (DETR 2000) by imposing a per unit of energy or weight basis tax that applies to electricity, gas, and other solid fuels. Levy rates were fixed from 2001 to 2006 and adjusted with inflation in the following years. In 2012 and 2014, the period used in this study, electricity rates were 0.51 and 0.54 pence/kWh, respectively.

Fearing a reduction in the competitiveness of energy intensive industries, the government introduced the Climate Change Agreements (CCAs) alongside the CCL.⁸ The CCAs are negotiated agreements between the government, sector bodies, and firms granting reduced CCL rates to participants when meeting energy efficiency targets. The first CCAs ended in December 2010 and were followed by the second CCAs, which are set to cover January 2013 to March 2025; the period between 2011 and 2012 served as a gap between the two policies that granted CCAs discounts without emissions or efficiency commitments. As the policy was revised between the two iterations, we will focus the discussion on the workings of the second CCAs and highlight the differences in Sect. 2.1.

Discount rates were adjusted throughout the policy; for example, the applicable rates on electricity in 2012 and 2014 were 65% and 90%, respectively. Instead of setting performance targets with individual firms, the Department for Business, Energy and Industrial Strategy (BEIS) negotiated targets with sector associations, which monitor and collect participating firms’ energy consumption and emissions. The negotiated targets were then included in an umbrella agreement⁹ between the Environment Agency and the sector associations (AEAT 2004).

Firms taking part in the CCAs need to agree to meet established targets in underlying agreements with the sector associations. Performance in the policy is evaluated for each target unit, which can be a single or a group of facilities based on vicinity and production criteria, rather than the firm in its entirety. Targets are established as a percentage decrease from a base year, which for the second CCAs is in most cases 2008, and can be set in three ways: (1) absolute; (2) relative; (3) novem. Absolute targets are based on total energy usage or emissions, for example, measured in kWh or tCO₂. Relative targets are set relative to some unit of output, e.g. kWh/$\text {m}^2$ of fabric. Lastly, novem targets are applicable in cases where a target unit produces more than one product with different energy intensities or output units.¹⁰ Performance against targets is evaluated in biennial target periods; failure to meet the goals can be remedied through accrued over-performance in earlier periods¹¹ or a buy-out mechanism (EA 2014).¹² If neither option is chosen access to the rebate is revoked and regular CCL rates apply (EA 2017).

Eligibility to the CCAs changed throughout the policy. Initially, firms holding a permit for the Integrated Pollution Prevention and Control (IPPC) regulation, which was later merged with the Waste Management Licensing (WML) legislation and renamed Environmental Permitting Regulation (EPR), were allowed to join. The rules were expanded in 2005 to allow in sectors fulfilling energy intensity and trade metrics.¹³ In contrast, production processes under the mineralogical and metallurgical processes category were made exempt from the CCL and thus the CCAs in 2014 to avoid distortions of competitiveness compared to other EU countries (HMRC 2015).

2.1 Changes from the First to Second CCAs

To illustrate why outcomes for the first and second CCAs could be different, we provide a brief overview of key changes implemented between the two. Scheme compliance was measured at the sector level during the first CCAs; access to the discount was granted collectively to the entire sector when sector targets were met, however, when targets were missed, DEFRA could request performance data of specific target units and revoke access to the CCAs rebate to individual firms. This implied that participants could still access the rebate without meeting their individual targets, which could incentivise shirking.¹⁴ In contrast, the second CCAs evaluate performance solely at the target unit level. In effect, this curbs free-riding incentives by holding each target unit accountable for continued access to the rebate regardless of sector performance. The percentage rebate on the CCL for electricity also varied throughout the policy, starting at an 80% reduction in 2001 that subsequently changed to 65% and 90% in April 2011 and April 2013, respectively (BEIS 2020). Additionally, in the first CCAs sectors were allowed to choose a base year to measure improvements against, which lead to base years ranging between 1990 and 2008 at the tail-end of the first policy (DEFRA 2007). Although this afforded flexibility to participating sectors, it could have incentivised them to choose years that lessen the stringency of targets. The second CCAs homogenised the base year to 2008 in order to reduce administrative burden, which also likely affected relative target stringency by preventing firms strategically choosing base years.¹⁵ Furthermore, the buy-out mechanism was simplified and uncoupled from the UK Emissions Trading Scheme allowance price, which was historically low ranging from 50 pence to $\pounds$4 per tonne of CO₂ (DECC 2012).¹⁶ Instead, prices were initially fixed at $\pounds$12 per tonne CO₂ in the second CCAs and were subject to review in following target periods (EA 2015a). By introducing a constant cost that is substantially higher than historic prices, non-compliance became significantly more expensive. Ultimately, it seems plausible that information gathered and learning accrued during the first CCAs has contributed to reducing information asymmetries between policy makers and firms, with the resulting changes made to the policy leading to more ambitious outcomes.

3 Literature Review

Voluntary agreements (VAs) and negotiated agreements (NAs) are a broad category of policy instruments encompassing a multitude of configurations reflected in the academic literature (Bailey and Rupp 2006). Many theoretical models have been developed to understand how different flavours of the two might affect adoption rates, economic performance, and other policy outcomes (Khanna 2001), but evidence of their success is rather sparse.¹⁷ In order to maintain focus on the quantitative evaluations of VAs/NAs, this section will feature two parts. First, it provides an overview of general criteria for a VA/NA to be successful and their applicability to the CCAs/CCL.¹⁸ Second, it will briefly discuss the mixed reputation of VAs/NAs and findings from econometric evaluations of the CCAs.

3.1 Theoretical Context of the CCAs and CCL Package

From a theoretical perspective, VAs/NAs are an interesting form of environmental regulation since affected subjects can—at least on paper—choose whether or not to participate in the scheme. In practice, however, participation is often negotiated under duress of alternative mandatory policies, influencing whether participation was truly voluntary (See Goodin 1986; Segerson and Miceli 1998; Croci 2005). However, voluntary measures can be advantageous for all parties involved as they encourage a pro-active cooperative approach that reduces conflicts, offers greater flexibility to find cost-effective solutions tailored to specific conditions, and grants the ability to quickly meet environmental targets due to decreased negotiation and implementation lags (Segerson and Miceli 1998). When setting targets, the government needs to strike a balance between achieving policy aims and ensuring participation. Asymmetric information about abatement potential could result in overly stringent policy targets that discourage participation, whereas lax targets could lead to not achieving social environmental goals.¹⁹

As there are many potential pitfalls that might render VAs/NAs ineffective, we draw on reviews of the literature to find common criteria that ensure that the policy meets its environmental objectives. First, participation incentives, both in the form of benefits and punishments, need to be sufficiently strong to encourage firms to engage with the policy. These can come in multiple forms, such as regulatory threats, subsidies, or establishing green credentials to policymakers and consumers (Segerson 2013). Regulatory threats in particular are considered to be effective if the likelihood of enacting legislation is high, i.e., the threat is credible (Segerson 2018). Second, monitoring of scheme performance needs to be transparent and verifiable. Without adequate monitoring of compliance it becomes difficult to determine when to grant and revoke access to privileges of the policy (Rezessy and Bertoldi 2011). Third, free-riding incentives need to be minimised and punishments need to be enforceable. In conjunction with the second criterion, free-riding becomes an attractive alternative if accurate measurements of individual performance are not obtainable. For example, a policy that only measures compliance at the sector level renders the danger of individual punishment moot and makes free-riding economically advantageous (de Vries et al. 2012).

How are the salient criteria explored in the aforementioned papers applicable to the CCAs and CCL package? The CCAs offered strong legislative participation incentives by granting a substantial discount on the CCL. Since the CCL and CCAs were introduced at the same time, there is no uncertainty about the regulatory threat. Thus missing targets in the CCAs becomes costly and unattractive. Additionally, the policy offered an opportunity to establish green credentials with consumers and policymakers. Surveys conducted among firms in the second CCAs indicate that reputation was a major motivator to participate, as 58% of respondents answered that demonstrating green credentials affected their decision, making it the third most frequently cited reason after a reduction in the CCL and the achievability of sector targets (BEIS 2020). Regarding the second and third criteria, while the first CCAs measured scheme compliance at the sectoral level,^20,21 thus not effectively prohibiting free-riding (Dijkstra and Rübbelke 2013), this short-coming was remedied in the second iteration when overall targets were negotiated at the sector level, passed down to individual target units, and scheme compliance was evaluated at the target unit level.

3.2 Statistical Policy Evaluation of VAs/NAs and CCAs/CCL

The available econometric studies on VAs/NAs have covered diverse configurations and geographic locations, finding mixed results regarding their efficacy (Segerson 2018). A prominent example is the United States’ Environmental Protection Agency’s 33/50 program, the evaluation of which has led to vigorous academic debate. The program was a voluntary agreement that asked participants to reduce emissions of 17 toxic gases by 33% by 1992 and 50% by 1995. Whether the policy succeeded in reducing emissions is debated; historically, studies have found evidence attributing reductions to the policy (Bi and Khanna 2012; Khanna and Damon 1999; Innes and Sam 2008) as well as disputing them (Vidovic and Khanna 2012; Gamper-Rabindran 2006). Results from more recent studies are similarly inconclusive. Research conducted in Zhou et al. (2020), which accounts for potential information spill-over effects between participants and non-participants, shows that the policy was effective in reducing emissions as non-participants learned from participants. On the other hand, estimates from Hoang et al. (2018) indicate that the policy did not succeed in lowering general emissions, although firms shifted away from chemicals most toxic to human health.

While the overall literature evaluating impacts of VAs/NAs is mixed, a policy sporting a similar structure to the CCL/CCAs packages has seen success. In 1993 a CO$_2$ tax was implemented in tandem with a possible rebate for energy intensive firms in Denmark. To qualify for the reduction, firms had to enter an agreement with the government to undertake negotiated energy saving activities and progress was evaluated by energy consultants. Results from Bjørner and Jensen (2002) indicate that the negotiated agreement was indeed successful in reducing carbon emissions, as energy consumption was found to substantially decrease. However, the authors caution that the sample size of eligible firms was rather small and related criteria, such as economic efficiency of the policy, were not considered making broader conclusions of the efficacy of NAs difficult. By using a large sample of firms covering diverse industries, this paper provides further evidence towards NAs successfully accomplishing their policy goals.

3.3 Econometric Evaluations of the CCAs and CCL

Focusing on the CCAs and CCL, two major econometric studies have investigated their effect, Adetutu and Stathopoulou (2020) and Martin et al. (2014). Adetutu and Stathopoulou (2020) studies the role of information asymmetry between regulators and firms in the first CCAs and CCL from 2001 to 2007. During target negotiations, the government is potentially at an information disadvantage regarding abatement opportunities available to specific firms. In such cases, firms might be able to negotiate lenient targets while claiming the CCAs rebate. The study first establishes the best-practices level of energy efficiency via stochastic frontier analysis and contrasts results to firm-specific energy efficiency levels. It then uses these results to evaluate how relative efficiency affects the received CCAs discount using pooled OLS, fixed effects, and system-GMM estimators.²² Results indicate that the average firm in the CCAs could reduce energy expenditure by an additional 19% when matching best practices available in the sample and that firms with higher energy efficiency received lower CCAs discounts. Thus, it appears that participants in the first CCAs left substantial improvement potential unrealised.

Martin et al. (2014) compares the impact of the CCL against the CCAs over the 2001 to 2004 period at the plant level.²³ Before presenting the results, it is important to highlight that carbon taxes, i.e., the CCL, is the instrument of interest in Martin et al. (2014), leading to firms affected by the CCAs being used as the control group. In contrast, as our focus lies in the efficacy of VAs/NAs, we treat CCL participants as the control group. While this complicates the comparison of our results with those in Martin et al. (2014), the different framing is made explicit when presenting and contrasting outcomes between the two studies. Estimation of the model in Martin et al. (2014) is conducted via two-stage IV regression and results show that plants subject to the CCL reduced electricity consumption by 22.6% and energy share in gross output (EGO), defined as energy expenditures over gross output, by 17.2% when compared to participation in the first CCAs, corroborating results of macroeconomic evaluations of the policies in Ekins and Etheridge (2006). Results for employment and $CO_2$ indicate that not participating in the CCAs leads to higher employment, although not statistically significant, and decreased emissions by between 9.6 and 22.6%. Overall, the findings indicate that the first CCAs delivered decreased environmental performance, that is, worsening energy efficiency and increasing emissions, while not delivering any statistically significant improvement in economic performance. The study posits that the poor performance of CCAs compared to the CCL may be related to low target stringency.

4 The Changes-in-Changes Estimator

The changes-in-changes (CIC) estimator used in this study was developed in Athey and Imbens (2006) as a generalisation of the popular differences-in-differences (DID) estimator. Instead of producing a singular summary statistic, such as the average treatment effect, the method preserves the heterogeneity in treatment response by estimating the treatment effects across the distribution of the outcome variable. Akin to the DID estimator, the CIC estimates the counterfactual distribution of the treated group in the absence of the policy post-treatment by combining the difference in the distributions of the control group before and after treatment, the time effect, with the distribution of the treated group before treatment. Furthermore, as the method estimates the treatment effect non-parametrically, no assumptions are made about the functional form of the equation generating the outcomes, Y. Outcomes without or prior to any treatment are posited to be generated by a shared function

$$\begin{aligned} Y_i^N = h(U_i,T_i) \end{aligned}$$

(1)

while outcomes after treatment are generated by

$$\begin{aligned} Y_i^I = h^I(U_i,T_i) \end{aligned}$$

(2)

with $U_i$ being an index aggregating all individual characteristics relevant to the outcome variable and $T_i$ being a treatment indicator for participant i. Given a realisation of U = u, the treatment effect, $\tau (u)$, can be computed as the difference in outcomes

$$\begin{aligned} \tau (u) = h^I(u,1) - h(u,1) \end{aligned}$$

(3)

Thus, the estimation of the treatment effect hinges on the unobservable U. To help conceptualise U, a real-life example of such an index would be the EPC rating used in the United Kingdom to evaluate a building’s energy efficiency, which aggregates housing characteristics such as window glazing and water heating system into a singular score; however, U does not need to be directly observable for the estimation given the identifying assumptions discussed in detail in the subsequent Sect. 4.1.

By finding matching levels of the unobservable, the method is robust to self-selection in the quantile treatment effects, as U encapsulates factors that could affect the treatment outcomes. However, since there are no restrictions on the differences in the distribution of U between the treated and control group, it is important to bear in mind that the estimated average impact is a treatment effect for the treated. As an example, suppose that a treatment shifts the distribution of U asymmetrically, for example, by a large margin in the treatment group and by a small margin in the control group if it had been subjected to treatment. In that case, the estimated average treatment effect would only be valid for the treatment group and not for the control group.

4.1 Identification

The focus of this section is to present the three key assumptions of the estimator, how they enable identification of the treatment effect, and a way to test the assumptions in an intuitive manner. For a detailed discussion of the CIC methodology we refer to the original (Athey and Imbens 2006) paper.

The first assumption for the CIC is that outcomes before treatment, regardless of later treatment status, are generated by the same function $h(U_i,T_i)$. This allows the comparison between the treated and control group before any treatment occurs.

The second assumption requires the function $h(U_i,T_i)$ to be strictly monotonic. As mentioned in the previous section, the key to the estimation in the CIC is the identification of similar levels of U. Although finding this matching level of U is complicated by not necessarily being directly observable, under monotonicity the relative position of U, i.e., the ranking in the distribution, can be found. Instead of using the exact values of U, one can use quantiles and therefore the ranking of each U to find similar individuals throughout the treatment and control group.

The last assumption requires time invariance of U within groups. If the passing of time substantially reshuffles the relative position of outcomes inside a group it becomes impossible to trace the same level of U across time. For example, for a value of U, if it is at the 60th percentile in the control group in the pre-treatment time period, the same U needs to be at the 60th percentile of the control group in the post-treatment period.²⁴ Through this assumption it becomes possible to recover the time effect by comparing the pre-treatment outcomes and post-treatment outcomes of the control group at the same percentile. Thus, the three assumptions can be written more formally as

• 1. Model: The outcome of an individual in the absence of treatment satisfies $Y_i^N = h(U_i,T_i)$

• 2. Strict Monotonicity: the function h(u,t), where h: $\mathbb {U}\times \{0,1\} \xrightarrow {} \mathbb {R}$ is strictly increasing in u for t$\in \{0,1\}$

• 3. Time Invariance within Groups: $U\perp T|G$

When combining the three assumptions, it becomes possible to identify the treatment effect $\tau _q$ for each quantile q via

$$\begin{aligned} \tau _q = F^{-1}_{Y,11}(q) - F^{-1}_{Y,01}(F_{Y,00}(F^{-1}_{Y,10}(q))) \end{aligned}$$

(4)

where $F_{Y,gt}$ is the cumulative density function (cdf) of the outcome variable Y with group membership g at time period t and $F^{-1}_{Ygt}$ is the inverse of the cdf. The first part of the right-hand side of the equation, $F^{-1}_{Y,11}(q)$, is the observed outcome of the treated (g = 1) after the treatment occurred (t = 1) at quantile q, while the second part, $F^{-1}_{Y,01}(F_{Y,00}(F^{-1}_{Y,10}(q)))$, is the counterfactual outcome. The estimation of the counterfactual and treatment effect is straightforward when broken down into steps and tackled from the innermost to the outermost operation. To further illustrate the procedure, we provide a visual example in Fig. 1 for some arbitrary outcome variable and quantile q = 0.65, where blue curves are the outcomes of the treated, red curves the outcomes of the control, the green curve the counterfactual, solid curves the outcomes at t = 0, dashed curves the outcomes at t = 1, and black lines serve as visual guides.

Fig. 1

Visualisation of the CIC estimator

For each quantile q, we first find the associated outcome value in the distribution of the treated at t = 0 (e.g. point A in Fig. 1). Since both groups’ outcomes are generated by the same equation h(U, T) before treatment (assumption 1), we can find the associated quantile of the outcome value in the control group at t = 0 (point B in Fig. 1) by locating the same U in the distribution of the control group before treatment. Then, via time invariance within groups (assumption 3), we recover the time effect from the control group by evaluating the same quantile in the distribution of the control group at t = 1 (point C in Fig. 1), which serves as the counterfactual value at q (point D).²⁵ By repeating this process for each q, the procedure allows us to trace out a counterfactual distribution by applying the appropriate time effect to each quantile in the outcome distribution of the treatment group at time 0. Lastly, the treatment effect is estimated by taking the difference in the observed outcome of the treated in t = 1 and the counterfactual at q ($\tau$ in Fig. 1).²⁶

A consequence of the method is that the range of estimable values for the counterfactual distribution is determined by two factors: 1. the overlap between the distributions of the treated and control at time 0; 2. the support of $Y_{01}$, $\mathbb {Y_{01}}$. Any $y_{10}\not \in \mathbb {Y_{00}}$ cannot be translated from the treatment group to the control group at t = 0, that is the second step of translating the outcome in the treatment group in Fig. 1 (moving from A to B) vertically to the control group is not possible as no reference value exists. Additionally, for any $y\not \in \mathbb {Y_{01}}$, the counterfactual is not identified, due to the counterfactual distribution being limited by the range of values existing in the control group at t = 1. Therefore, the distributions of the treatment and control group do not affect estimation results, but the range of estimable quantiles.

In order to verify the assumptions we use the test introduced by Melly and Santangelo (2015), which evaluates the following criterion

$$\begin{aligned} F_{Y,0-1}(F^{-1}_{Y,1-1}(q)) = F_{Y,00}(F^{-1}_{Y,10}(q)) \end{aligned}$$

(5)

for two time periods before treatment and all $q \in (0,1)$ using a two-sample Kolmogorov–Smirnov²⁷ test. Similar to tests of the common trend assumption in DID models it uses periods prior to treatment, labelled t = − 1 and t = 0, to check whether model assumptions are satisfied. The intuition of this test is that, before treatment, the translation of outcomes from the treatment to the control group at quantile q should consistently remain similar in ranking when all outcomes are generated by h(U, T). If this is not the case, it would indicate that either outcomes are not generated by a shared h(U, T) or that the distribution of U is not independent of time within each group. We provide a visual representation of the test in Fig. 2 using the same colour and curve styles as in Fig. 1.

Fig. 2

Visualisation of Melly and Santangelo, 2015

The left-hand side of the equation first evaluates the corresponding outcome of a quantile q at time period − 1 of the to-be-treated (A in Fig. 2). Then, it finds the quantile associated with the value, i.e., the translated ranking, among the control (C in Fig. 2) in the same period. The right-hand side does the same for another pre-treatment period 0 (B to D in Fig. 2). If these rankings between time 0 and − 1 vary, either the distribution of the unobservable, and by extension of the outcome variable, is unstable within groups over time or the outcomes are not generated by a shared function h(U, T). Referring to Fig. 2, if the vertical distance between C and D labelled ”discrepancy” were too big in the pre-treatment periods at any quantile, the KS-test would reject its null hypothesis of the two samples being drawn from the same distribution. Thus, given the assumptions of the method, a KS-test would fail to reject its null hypothesis.

5 Data

The data for this article was combined from a multitude of data sources provided by the Department for Business, Energy and Industrial Strategy (BEIS). Participants were identified using scheme data, while the control group was created using information declarers in the Carbon Reduction Commitment (CRC), which are defined in detail in Sect. 5.2. The two groups were matched to readings in BEIS’s meter dataset through text-based address matching. Both groups were matched to the Inter-Departmental Business Register (IDBR) to gain employment statistics and cross-referenced against major concurrent policies.

5.1 Treatment Group: CCA Participants

Information for the treatment group comes from scheme data collected by the Environment Agency (EA). The original dataset contains 14,366 facilities in 53 sectors that were in contact with the EA regarding participation in the second CCAs. Firms that were deemed non-eligible and those from 14 sectors related to mineralogical and metallurgical process are removed from the sample, the latter due to becoming exempted from the CCL on April 1st, 2014. The dataset used in this study includes 6849 facilities that entered the policy during the treatment year, 2013, and had an active CCA in the post treatment period, 2014. Table 1 shows the median, mean and standard deviation for electricity consumption and employment in five sectors, chosen to be large enough to be non-disclosive, and the total in 2012. The table illustrates the heterogeneity not only across but also within sector, indicated by the large standard deviations and discrepancy between the median and mean, corroborating our case to use the CIC estimator.

Table 1 Variable summary statistics by sector (2012)

5.2 The Control Group: CRC Information Declarers

The control group consists of facilities of information declarers during phase 1 of the Carbon Reduction Commitment (CRC) scheme. The CRC, later renamed to CRC Energy Efficiency Scheme, was introduced on April 1st, 2010, and concluded on March 31st, 2019. The CRC aimed to drive energy efficiency and reduce carbon emissions in large energy users in the public and private sector that did not fall under EU ETS or CCA (EA 2015b). Eligibility to CRC phase 1 was determined by two factors: (1) The organisation had at least one settled half-hourly meter; and (2) total qualifying supplies in 2008 matched the qualification threshold (6000 MWh) or more (DEFRA 2007). Information declarers are firms satisfying only criterion (1), and as such are not affected by the CRC. However, the presence of half-hourly meters is indicative of high electricity usage.²⁸ Overall, information declarers are attractive as a control group since they are the largest electricity consumers that have neither been offered participation in the CCAs nor the CRC while being subject to full CCL rates. They offer substantial overlap in the outcome distributions for the CIC due to their size²⁹ and they are not in the sectors covered by the CCAs, which alleviates concerns about potential spillover effects. The CRC data provided by BEIS contains 11,225 facilities for information declarers.

5.3 Data Matching

Members of the treatment and control groups are matched to electricity meter locations through the Levenshtein ratio, which evaluates how many edit operations are required to change one word to another relative to their lengths (Levenshtein 1966).³⁰ To ensure the quality of the data, only facility-meter pairs within a narrow range of postcodes with Levenshtein ratios above 80% are accepted, resulting in an overall 43.21% match rate for the treatment and 48.75% for the control group. Using a similar algorithm, facilities in both groups are matched to local units,³¹ the closest equivalent of facility level data in the Inter-Departmental Business Register (IDBR), to recover employment information. Match rates for employment are 60.39% in the treatment group and 55.54% in the control group.

5.4 Concurrent Policies

The second CCAs did not occur in a legislative vacuum and, as such, important concurrent policies exist. Besides the aforementioned CRC, two other policies are relevant to firms in the treatment and control group: the EU Emissions Trading Scheme (EU ETS) and Energy Savings Opportunity (ESOS). The EU ETS, introduced on January 1st, 2005, is a cap and trade system aiming to reduce $\text {CO}_2$ emissions from facilities with large combustion equipment. Energy covered by EU ETS is generally not eligible for the CCAs discount. However, under very specific circumstances both policies can apply at a given facility.³² ESOS is a mandatory assessment policy that came into force July 1st, 2014, which applies to large UK undertakings and their corporate groups.³³ Firms are tasked to identify assets and activities that constitute at least 90% of their energy consumption and undertake an energy audit including energy savings suggestions. However, these suggestions are non-binding and as such, improvements to energy efficiency are non-mandatory (EA and BEIS 2014).

To isolate the impact of the CCAs, this paper removes any facility in the treatment group participating in CRC phase 2, approximately 13% of the original sample, and EU ETS phase 3, approximately 2% of the original sample. Overlap with CRC phase 2 was checked using scheme lists provided by BEIS. EU ETS participant data was gathered by web scraping from the website³⁴ of the European Commission and matched to CCAs participants. Facilities in ESOS are not removed, due to the first compliance period, December 2015, being significantly beyond the treatment year, 2014, and only 0.15% of participating companies’ directors reviewing the recommendations contained in the policy audit in Q4 of 2014 (EA and BEIS 2018) and none before. After cleaning the data of erroneous readings, e.g., error codes, the following summary statistics for the sample used in the study are obtained (Tables 2, 3).

Table 2 Electricity summary statistics

Table 3 Employment summary statistics

6 Results

Before discussing the results it is important to evaluate the estimable range for each variable and verify the CIC assumptions. As discussed in Sect. 4, the range of identifiable quantiles depends on the overlap in the distributions of the outcome variables between the treatment and control group. Within the sample of this study, electricity consumption is identified up to the 93^rd percentile of the treated group and employment is fully identified. Additionally, we also find evidence that the distribution of unobservables, U, is time invariant within groups. We applied the test of Melly and Santangelo (2015) described in Sect. 4 on observations in 2011 and 2012, an interim period between the first and second CCAs that did not impose any policy targets while granting rebates (BEIS 2020). Without targets in place, there are fewer factors affecting facilities’ electricity consumption and employment, motivating our choice to verify estimator assumptions in those years. As the p values for the Kolmogorov–Smirnov test for electricity and employment are 0.12 and 0.06, respectively, we fail to reject the null hypothesis that the treatment and control groups are drawn from the same distribution when no treatment has occurred. Thus, the time invariance assumption seems to be verified in the pre-treatment period for electricity, while not as strongly supported for employment.

Going forward, the pre-treatment year is set to 2012 and the post-treatment year to 2014 to coincide with the end of the first CCAs and the end of the first target period of the second CCAs. To minimise excess description, we will use negative treatment effect as a shorthand to describe those instances when the observed level of the outcome variable in the treated group is smaller than the level in the counterfactual, that is, participation in the CCAs is associated with a smaller level than the counterfactual CCL outcome, whereas a positive treatment effect implies the opposite.

6.1 Electricity Consumption

Table 4 contains the treatment effect ($\tau$), standard error (SE) and significance level, outcome for the treated units ($Y_{11}$), counterfactual outcome ($Y^N_{11}$), and the percentage difference between the treatment and counterfactual of electricity consumption (% diff.) for each decile, q. The final row contains what we call the Identifiable Quantile Mean (IQM), which is the mean of the treatment effect within the identifiable range.

Table 4 CCAs impact on electricity consumption by decile—overall sample

Overall, the identified deciles indicate that the CCAs delivered electricity consumption reductions greater or about equal to the CCL, as most deciles are negative and the majority not statistically significant. This conclusion is supported by the IQM, which shows an average reduction in electricity consumption of − 4.8% significant at the 10% level. Interestingly, the positive treatment effects are located at the lower and upper end of the presented deciles. These findings mirror the qualitative analysis of the CCAs conducted in BEIS (2020), which found that firms at the extremes of energy consumption were less receptive to the incentives provided by the CCAs. For small firms meeting targets in the CCAs may not have been cost-effective, while very large firms were making substantial efforts to improve energy efficiency regardless. Thus, the CCAs appear to be most efficacious in medium sized electricity consumers. Figure 3 contains plots for each quartile of the results.

Fig. 3

Empirical and counterfactual distributions for electricity consumption by quartile

Each facility participating in the CCAs faces two opposing economic incentives compared to the CCL baseline. On one hand, CCAs participation lowers the cost of electricity by reducing the applicable CCL rate and therefore incentivises higher consumption. On the other hand, CCAs impose energy efficiency targets aimed at making up for the lower financial incentive to reduce energy consumption. Despite the majority of firms (82%) choosing relative targets that do not require a decrease in absolute levels of consumption, the CCAs reduced electricity consumption similar to what would have been delivered by the CCL. Thus, the pressure to increase efficiency included in the targets appears to have delivered a decrease of similar levels for electricity consumption when compared to the CCL. As the aim of CCAs was to mitigate the impact of the CCL on energy intensive industry and to deliver efficiency improvements broadly equivalent to what would be achieved by the application of the full rate of the CCL (DECC 2012), the policy appears to have met its intended objective. These findings contrast with the evaluation of the first CCAs conducted in Martin et al. (2014), which found a statistically significant reduction in electricity consumption delivered by full CCL rates compared to CCAs. They concluded that the combination of targets and discount provided by the first CCAs were insufficient to outweigh the price incentives of the full levy. The same cannot be said for the second CCAs.

An almost 5% average reduction of electricity consumption among a subset of business users harbours another interesting possibility, that is, whether lowered electricity demand could reduce electricity prices.³⁵ We investigate this with a back of the envelope exercise by multiplying the IQM with the number of identified facilities and find that the estimated reduction is approximately 0.21 tWh, which is 0.22% of industrial and 0.07% of total electricity consumption in 2014 (BEIS 2022). It therefore appears unlikely that the second CCAs affected the electricity market price, even when considering that not all facilities in the policy were able to be matched.

6.2 Employment

Before discussing the estimates for employment in Table 5 and Fig. 4, it is important to mention that the quantiles of the CIC analysis are with respect to the variable being assessed, so that the facility at the 90th percentile of electricity consumption is not necessarily related to the facility at the same quantile of employment.³⁶

Table 5 CCAs impact on employment by decile—overall sample

Fig. 4

Empirical and counterfactual distributions for employment by quartile—overall sample

CCAs participation appears to have a limited effect on employment. Contrary to expectations, employment in CCAs participating firms is predominantly lower than in the counterfactual across quantiles with the exception occurring at the upper tail-end of the distribution where the impact of the CCAs is positive at some percentiles. The IQM is negative, but, like any other estimate of the treatment effect in Table 5, statistically not significant. The graphs in Fig. 4 support these findings as the counterfactual distribution is mostly to the right of the treated distribution, implying a lower demand for labour under the CCAs compared to what would have been delivered by the full CCL rates. Similar results were found in Martin et al. (2014) for the first CCAs, as full CCL taxation was found to increase demand for labour, although the impact was not statistically significant.

Our results indicate that firms responded differently to the CCAs depending on the level of electricity consumption, while the same is not as evident for employment. To further investigate factors that could affect the treatment response in electricity consumption and employment, we conduct two additional analyses. First, we study the effect of different target types and, second, we choose two sectors for within and across sector analysis.³⁷

6.3 Target Type Analysis

As many economic considerations may influence a firm’s choice of target type, such as expected demand or rate of technological change, facilities with different target types are likely to respond to the policy differently. Participants in the second CCAs were allowed to choose between three types of targets: 1. relative, 2. novem, and 3. absolute.³⁸ The majority of participants, 82% in target period 1 of the second CCAs, adopted relative targets, while 13% used novem, and 6% absolute targets.³⁹ The small number of facilities adopting absolute targets⁴⁰ discourages the use of the CIC estimator within the sub-group, leading us to combine them with facilities choosing novem targets. While the joint group is still relatively small, the results provide an interesting contrast to the relative target group. Here it is important to reiterate that the percentiles in the tables below are not the same as those found in Table 4. Nevertheless, since relative targets constitute the majority of the sample, the percentiles in the overall and the relative target sample are fairly close. As the CIC requires overlap in the outcome distributions between the control and treatment group, we use the entirety of the CRC information declarer control group and verify the use via the pre-treatment KS-test for each subset of the data.

We also estimated the impact of the CCAs on employment within each sub-group, which closely resembled treatment effects in the overall sample, i.e., the policy resulted in a non-statistically significant reduction in employment. In a few deciles the treatment effect was significant at the 10% level, but due to the low significance and sporadic frequency our prior conclusion on the effect of the policy remains the same. This is particularly interesting from the sector analysis, as facilities in Sector A and B are highly heterogeneous in terms of their electricity consumption scale and capital intensity. Thus, there seems to be little evidence that the CCAs had a heterogeneous impact on employment regardless of the economic reality faced by facilities. For the sake of brevity, we included the estimation tables for employment in “Appendix CCA Employment Impacts by Targets and Sectors”. We do not report results of the 100th percentile due to data security concerns,⁴¹ however, none of the identified results at the percentile were statistically significant.

6.4 Relative Targets

For relative targets, 2078 and 2097 facilities are in the sample for the pre- and post-treatment periods, respectively.⁴² As the KS-test fails to reject the null hypothesis of identical distributions of the unobservable in the pre-treatment phase with a p value of 0.27, we proceed to the estimation using the CIC. Estimates of the treatment effect, identified up to the 94th percentile, are shown in Table 6 and Fig. 7 in the Appendix.

Table 6 CCAs impact on electricity consumption by decile—relative target sample

Our results for the units adopting relative targets very much resemble those of the overall sample. The 20th, 30th, and 90th percentile treatment effects are positive, implying higher electricity consumption under the CCAs than what would have been observed under the full CCL while the opposite occurs in all other cases. However, of the three, only the 20th percentile effect is statistically significant. This percentile, approximately 70,000 kWh, is situated between the 20th, 65,000 kWh, and 30th percentile, 100,000 kWh, in Table 4, corroborating the statistical significance observed in those cases. The 90th percentile in Table 6 is located between the 80th and 90th percentile in Table 4, which explains the small treatment effect reported in Table 6 due to proximity to the crossing-point between negative and positive treatment effects in the whole sample. This overall negative impact is confirmed by the IQM, which shows a reduction of − 5.7% significant at the 5% level.

6.5 Absolute and Novem Targets

Compared to the relative targets, much fewer facilities opted to use the other two target types.⁴³ Since the KS-test fails to reject the null hypothesis of similar distributions in the pre-treatment period with a p value of 0.93, we proceed with the estimation. Electricity consumption in the group is relatively long-tailed to the right, causing the maximum identifiable quantile of the treatment effect to be the 86th percentile. In contrast to the total and relative target samples, the facilities choosing absolute or novem targets have decreased their electricity consumption under CCAs compared to what would have been observed under full CCL taxation at all deciles. Standard errors of the estimates in Table 7 are generally quite large, leading to only two percentiles, the 20th and the 80th, having statistically significant treatment effects. Another consequence of the smaller sample size, see Fig. 8 in the Appendix, is that for high levels of electricity consumption both the observed and counterfactual distributions become less smooth. This results in the impact becoming more volatile at higher levels of electricity, confirming results observed throughout this study. Overall, based on the impact at the deciles and the value of the IQM, it appears that targets in the second CCAs for absolute and novem facilities were sufficiently stringent to lead to a decrease or no significant difference in energy consumption when compared to the CCL.

Table 7 CCAs impact on electricity consumption by decile—absolute & novem target sample

6.6 Sector Analysis

We now focus our attention on the impact of the policies on two CCAs sectors. Since the targets were negotiated for each sector by trade associations it is possible that target stringency differs across sectors. Additionally, electricity consumption within a CCAs sector spans a smaller range than in the whole sample, therefore allowing a more granular investigation into the impact of the policy. We study the treatment effect in two CCAs sectors that cover different ranges of the outcome distribution. To ensure data security for the CCAs participants, we opted to anonymise the names of the sectors. Sector A consists mostly of small-scale consumers, whose electricity consumption is below the median of the total sample, and few large-scale consumers. In comparison, facilities in Sector B are medium to large-scale consumers with their 10th percentile situated close to the median of the whole distribution. Therefore, our analysis illustrates the effect of the policy in different sectors but also across levels of electricity consumption.

Outcomes between the two sectors are markedly different, as the treatment effect in Sector A is mostly non-statistically significant, whereas firms in Sector B consistently decreased their electricity consumption compared to their counterfactual. Overall, the sector results corroborate the findings of increases in electricity consumption for facilities around the 20th to 30th percentile of the overall distribution and decreases for larger facilities.

6.7 Sector A

Sector A has a high number of treated units in the sample with 586 and 589 facilities in the pre- and post-treatment periods. Since the KS-test fails to reject the null hypothesis of similar distributions of the unobservable in the pre-treatment phase with a p value of 0.22, we proceed with the estimation. When looking at Table 8 there appears to be limited heterogeneity in the treatment response, since the effect is mostly positive with two exceptions at the 10th and 90th percentile of the distribution, neither of which is statistically significant. Two out of three statistically significant deciles, the 20th and 50th, largely coincide with the 20th and 30th percentiles in the total distribution (Table 4). The remaining statistically significant decile is the top-end of Sector A at the 100th percentile, which lies between the 70th and 80th percentiles in the overall distribution (4). Thus, from the results in Table 8 there seems to be some evidence that the CCAs increased electricity consumption in Sector A, but the average impact is rather small at a non-statistically significant 0.93% increase. This seems to be mainly driven by large facilities above the 95th percentile but below the 100th percentile, which show a negative treatment effect; due to the very long-tailed nature of electricity consumption in the sector, the larger facilities substantially impact the IQM. Thus, Sector A showcases the strength of the CIC estimator over more conventional methods, as the small average impact would lead to the conclusion that the effect of the CCAs is a minor increase in electricity consumption over the CCL in this sector, while a number of medium to large electricity consumers, relative to the overall sample, have decreased their consumption. Figure 5 corroborates these findings, as the counterfactual consumption is predominantly lower than the observed level except at the upper end of the distribution, where the treatment effect is more variable.

Fig. 5

Empirical and counterfactual distributions for electricity consumption by quartile—Sector A

Table 8 CCAs impact on electricity consumption by decile for the Sector A

6.8 Sector B

Sector B offers a similarly large sample size with 554 and 559 facilities in the pre- and the post-treatment period, respectively. As the KS-test fails to reject the null hypothesis of similar distributions in the pre-treatment period with a p value of 0.39, we continue with the estimation. Facilities in the sector are medium- to large-scale electricity consumers and their treatment effects differ significantly from the impact in Sector A. First and foremost, all deciles show a negative treatment effect, implying lower levels of electricity consumption than what would have been observed in the full CCL counterfactual. The effect, which largely lies between the 40th and the 80th percentile in Table 4, is significant in 7 out of the 9 deciles studied. In percentage terms, the treatment effect is quite stable, perhaps a reflection that facilities are homogeneous in their production function regardless of size. In line with the decile treatment effects, the IQM in the sector is significant at the 1% level and indicates an approximately − 14% decrease in electricity consumption attributable to the CCAs. The graphs in Fig. 9 support these findings as the counterfactual consumption is to the right of the treated consumption by a large margin. One would therefore conclude that there seems to be strong evidence of the CCAs reducing electricity consumption in Sector B when compared to the alternative of full CCL (Fig. 6, Table 9).

Fig. 6

Empirical and counterfactual distributions for electricity consumption by quartile—Sector B

Table 9 CCAs impact on electricity consumption by decile—Sector B

7 Conclusion

In this paper, we contribute to the academic discussion on voluntary and negotiated agreements by conducting a quasi-experimental evaluation of the second Climate Change Agreements. The policy offers a discount on the Climate Change Levy, a tax on energy consumed by most non-domestic, i.e., non-household, users in exchange for energy efficiency improvements. Our work contributes to the existing literature from three points of view. First, we analyse the impact of the so-far unstudied second CCAs on electricity consumption and employment, finding evidence towards the efficacy of negotiated agreements. Second, we exploit facility level data,⁴⁴ while currently available studies of the first CCAs are restricted to plant level data, which can include multiple facilities grouped together. Third, we apply the novel changes-in-changes estimator developed in Athey and Imbens (2006), which, in contrast to the more common differences-in-differences method, allows for the evaluation of the policy impact across the quantiles of the distribution of outcomes. Therefore, we explicitly account for heterogeneity at different levels of the variable of interest.

Outcomes of the second CCAs provide a marked contrast to the first. The first CCAs were criticised for insufficiently stringent targets that lead to worse policy outcomes when compared to full taxation under the CCL; estimates in Martin et al. (2014) attribute a − 22.6% reduction in electricity consumption to the CCL versus the first CCAs. This shortcoming seems to have been addressed via changes made in the second CCAs, which have largely delivered, as intended, consumption reductions at least equivalent to what would have been achieved by the application of the full CCL rates. Electricity consumption tends to be lower than the counterfactual in six of nine deciles with the reduction ranging from − 1.5 to − 9.3%, while it was higher in the remaining three deciles, ranging from a 9.3 to 12.5% increase. This results in an average reduction of − 4.8%. On the other hand, our findings on employment effects mirror those for the first CCAs in Martin et al. (2014), as the impact is not statistically significant and negative with an average − 4.6% decrease among the identified treatment effects.

Our analysis uncovered a tendency of the CCAs to deliver increased electricity consumption for small consumers compared to a full CCL counterfactual. The opposite tends to occur in the case of large consumers. Target types seem to have some influence on this pattern. 82% of the sample adopted relative targets, which follow the overarching trends established for the entirety of the sample. However, facilities choosing the other two targets, novem and absolute, consistently decrease their electricity consumption compared to the counterfactual. The pattern above is also related to the impact in specific CCAs sectors. We focus on two large sectors that occupy different portions of the electricity consumption distribution. On one hand, Sector A consists primarily of small-scale electricity consumers, which tend to increase electricity consumption under the CCAs compared to the full CCL. Only some very large facilities in the sector reduced their electricity consumption, skewing the average impact within the group downward to a statistically non-significant 0.93% increase. On the other hand, facilities in Sector B tend to be medium- to large-scale consumers that consistently decreased electricity consumption in the CCAs compared to the CCL by on average − 14.9%.

This study delivers important evidence that the second CCAs, an example of negotiated agreements, have delivered reduction in electricity consumption largely comparable to what one would have obtained if the CCL, an energy tax, was applied at the full rate. Our work shows that consumption behaviour was systematically different between smaller and larger electricity consumers participating in the CCAs, which is further supported by the impact in two sectors with large numbers of facilities. As is the case in many empirical studies, results could perhaps be more insightful by improving the quality of data. Our facility level consumption data is novel, but having to match different datasets to obtain the sample used in this study is not ideal, due to the compounding impact that failure to match across the datasets has on available sample size. Requiring scheme participants to disclose associated meter and company reference numbers could substantially improve the analysis by making data matching obsolete. Without heavy sample attrition, integrating the effects of other covariates⁴⁵ into the distributional analysis would be valuable to investigate how different economic factors contribute to the heterogeneity in treatment and might uncover interesting economic mechanisms. Possible variables of interest include investments, existing levels of capital, R &D expenditure, employment data by skill level/education, whether electricity is purchased centrally or facility by facility, characteristics of the demand for the main good produced, the market the firms operate in, strength of sector associations, and levels of international competition. Lastly, it would have also been interesting to assess the impact of the CCAs not only in terms of energy consumption, but also in terms of their impact on energy expenditure by studying how reductions in consumption interact with the effective electricity price paid by firms.

Appendix

1.1 CCA Employment Impacts by Targets and Sectors

The subsequent tables contain the impact of the CCAs on employment for the target type and sector analysis conducted in Sects. 6.3 and 6.6, respectively. Tables contain information on each decile (q), estimates of the impact measured in number of employees ($\tau$), the standard error (SE), headcounts under treatment ($Y_{11}$), headcounts under the counterfactual ($Y^N_{11}$), and percentage difference between the two cases (% diff.). The results within the four sub-groups studied here did not significantly diverge from the employment results found in Sect. 6.2, leading us to include them in the appendix (Figs. 7, 8).

Fig. 7

Empirical and counterfactual distributions for electricity consumption by quartile—relative sample

Fig. 8

Empirical and counterfactual distributions for electricity consumption by quartile—absolute & novem sample

1.2 Relative Targets

For the relative target group the entire counterfactual distribution is estimable and the p value of the pre-treatment KS-test is 0.06. We continue the estimation with the caveat that model assumptions in this subgroup are not satisfied at the highest (10%) level. Outcomes for the group in Table 10 only show minor differences to the overall results shown in Table 5, that is the treatment effect is predominantly negative and statistically non-significant. Only two deciles do not follow this trend, the 30th and 70th, which are negative and significant at the 10% level. As the results mostly mirror the outcomes in the main estimation, they seem to confirm that participation in the CCAs did not affect employment differently than the CCL.

Table 10 CCAs impact on employment by decile—relative targets

1.3 Novem and Absolute Targets

The entire distribution of the counterfactual is estimable and the p value of the pre-treatment KS-test is 0.99 in the novem and absolute targets group. We thus proceed with the estimation for which the results can be found in Table 11. The impact of the CCAs in the group strongly resemble the overall outcomes as none of the estimated treatment effects are significant. A key difference is that at three deciles of the distribution, the 10th, 20th, and 60th percentile, the treatment effect is positive while it was negative in similar sized facilities in the overall distribution (Table 5). However, as the results are not statistically significant the findings corroborate that the CCAs did not lead to different employment outcomes when compared to the CCL.

Table 11 CCAs impact on employment by decile—absolute and novem targets

1.4 Sector A

In Sector A the entire counterfactual distribution is estimable and the p value for the pre-treatment KS-test is 0.99, leading us to proceed with the estimation. Matching outcomes of the overall sample found in Tables 5, 12 shows that the treatment effect in Sector A is non-statistically significant and negative across the deciles studied. This result is interesting as Sector A covers a wide range of small- to large-scale electricity users, further indicating that the CCAs did not significantly affect employment when compared to the CCL.

Table 12 CCAs impact on employment by decile—Sector A

1.5 Sector B

Lastly, for Sector B, the counterfactual distribution is identified from the 1st to the 99th percentile and the p value of the pre-treatment KS-test is 0.12, allowing us to proceed with the estimation. Outcomes found in Table 13, strongly resemble the results of the overall distribution in Table 5, as the majority of results are negative and statistically non-significant. Only the 60th decile is negative and significant at the 10% percent level. In contrast to Sector A, Sector B consists of a more homogeneous group of medium- to large-scale electricity users. Thus, both sectors appear to confirm that, overall, the CCAs did not affect employment significantly when compared to the CCL even in light of different sectoral features.

Table 13 CCAs impact on employment by decile—Sector B

1.6 Hypothetical Impact of CCAs on CRC Information Declarers

While the CCAs were designed as an alternative to the CCL with the objective of encouraging similar performance without increasing the tax burden of energy intensive firms, it is interesting to see how firms with lower energy intensity would have fared in the policy. Therefore, as an additional robustness check, we estimate the treatment effect on the original control group, the CRC information declarers, for both electricity and employment. In this case, the CCAs participants become the control group and CRC information declarers paying full CCL rates are now the treatment group. Therefore, the counterfactual in this section is the hypothetical outcome if the CRC information declarers had received the CCAs instead. Since the treatment effect is estimated as the observed value of the treated minus the counterfactual and the roles have reversed, so has the interpretation of results. A positive treatment effect implies that the counterfactual, i.e., the hypothetical CCAs outcome, is smaller than the observed CCL outcome of CRC Information Declarers. Overall, this analysis confirms our conclusion that the CCAs performed similarly to the CCL with regard to electricity consumption and employment.

1.7 Electricity Impact

The p value on the pre-treatment KS-test is 0.23, thus giving us confidence in the time invariance assumption, and the entire distribution of electricity consumption is estimable. Results in Table 14 confirm the initial findings of the CCAs performing about as well as the CCL in reducing electricity consumption, as most treatment effects are not statistically significant. The only exception here is the 10th percentile showing a higher electricity consumption under the CCL than the CCAs, which is significant at the 1% level. In the case of the distribution of CCAs firms (see Table 4), electricity consumers of a similar size also showed consumption higher under the CCL compared to the CCAs. This provides some evidence that very small electricity consumers, at approximately the 10th percentile of the overall distribution, may react more strongly to efficiency targets than price incentives under the CCL.

Table 14 CCAs as counterfactual: impact on electricity by decile

1.8 Employment Impact

The distribution of employment is identified from the 1st to the 99th percentile and the p value on the pre-treatment KS-test is 0.99. Results for this scenario shown in Table 15 confirm that the CCAs do not appear to have an overall statistically significant impact on employment. One exception to this occurs at the 90th percentile, at which employment is reduced by 7.61% under the CCAs significant at the 5% level. The treatment effect of similar sized firms in the original estimation (see Table 5) showcase an effect of similar magnitude, however, the results were not significant.

Table 15 CCAs as counterfactual: impact on employment by decile