Proceed With Caution: Challenges In Embedding The Precautionary Principle In Paris-Aligned Benchmarks

By Malgorzata Olesiewicz, data scientist, SI research and Jaakko Kooroshy, global head of SI research

Climate regulations now require the precautionary principle to be respected. But what does it mean?

“Principle 15: Where there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent environmental degradation.”

This precautionary principle is meant to ensure that measures which could prevent serious environmental damage are not dismissed only because the damage itself is uncertain.¹ The principle was first put into practice in the 1970s within German environmental legislation² and later popularised through a number of international agreements³ in particular the United Nations 1992 Rio Declaration.⁴ Three decades on, it has been incorporated into a host of regulatory frameworks, overseeing everything from genetically modified crops, X-rays, and asbestos to the pharmaceutical approval process.⁵

In 2020, the European Commission introduced the principle into the sustainable finance regulatory framework including for climate indexes, as labelled ‘Paris-aligned’ or ‘climate transition’ benchmarks (PABs and CTBs)⁶. To apply these labels, among other criteria, PABs and CTBs Regulation prescribes that benchmark administrators “shall formalise, document and make public” how the precautionary principle has informed estimation methodologies for any input data used in index construction.⁷ In this blog, we attempt to shed some light on the complex questions faced by index providers in interpreting and implementing the principle in the context of benchmarks.

The precautionary principle in the context of corporate carbon emissions

The principle is particularly relevant for corporate emissions estimates because steady emissions reductions (-7% on an annual basis) are key to meeting the regulatory requirements of these climate benchmarks, and a significant share of corporates do not disclose emission data.⁸

But the EU legislative arsenal (e.g., Article 13 in the EU Commission Delegated Regulation) doesn’t specify what a reasonable application of the precautionary principle looks like in this context or for other regulatory use cases, leaving practitioners to devise their methodology and criteria.9 This ambiguity creates challenges: how can carbon emissions estimations reflect the precautionary principle and, more importantly, what are appropriate modelling assumptions to make?

Reducing underestimation and underreporting

A prominent approach suggested in the literature is the 95^th quantile estimation,¹⁰ which effectively assumes that the non-reporting companies have worst-in-class emissions. With disclosure currently still voluntary in most jurisdictions, gaps might not be random since companies themselves decide whether to disclose their data (i.e., self-select).^11,12 The approach, therefore, assigns the 95^th percentile of the carbon intensity distribution of reporting peer companies to non-reporting, rather than the usually preferred average or median (50^th percentile) to ensure that corporate emissions data is not underestimated. In other words, the estimation method aims to incentivize companies to report their emissions and to report them accurately.

This approach seems straightforward, but it implicitly assumes a relatively orderly distribution of the underlying data. In the case of corporate carbon emissions, however, investors must contend with large volumes of unreported data and very high variances within peer groups producing statistical distributions with very long tails. For example, for the FTSE All World, in 2020 the 95^th quantile of reporting companies by ICB Subsector was on average 10 times¹³ higher than the median.

In this setup, choosing the 95^th percentile replaces large volumes of unreported data with extreme values. While this does ensure that values are rarely underestimated, it can come with unintended consequences, including on average significantly overestimating emissions which often end up being an order of magnitude higher than actual emissions or peer group averages.

Moreover, to provide more reliable estimates, carbon estimation models often have to trade off larger sample sizes for more homogenous peer groups with much smaller sample sizes (choosing the Sector rather than Subsector for example creates a larger, but also much more heterogenous peer group with the 95^th quantile on average being pushed to 21 times the median). More granular peer groups indeed help the accuracy of the estimates but also make calculations of the 95^th percentile (and the estimates based on it) very volatile year-on-year as they are often driven by disclosures from just one or two firms.

In a nutshell, the 95^th percentile approach turns out to be impractical for corporate emissions estimates, given a large number of non-reporting companies, high degrees of heterogeneity in reported data, and sample size limitations, resulting in unrealistically high estimates for non-reporting companies that tend to be volatile year-on-year. This may be especially problematic for several investment use cases where total emissions of the universe are of interest or where the ranking of the companies depends on the distribution of the entire universe.

An alternative, but ultimately similar approach suggests using - in addition to the 95^th percentile - a range of other percentiles to generate a distribution of estimated values.¹⁴ The authors of this alternative approach suggest that the mean of the distribution should be considered the most accurate estimate but offer investors to choose higher quantiles of the distribution to comply with the precautionary principle at their discretion.

While this approach may be preferable to a more simplistic 95^th percentile approach, it ultimately side-steps the question of what an appropriate implementation of the precautionary principle looks like. Effectively this forces data users, rather than data providers to determine how the precautionary principle should be implemented.

Addressing some of these challenges, FTSE Russell’s approach for implementing the precautionary principle for emissions estimates relies on three premises.

Firstly, the principle is implemented at each step of the dataset curation. This means that its application should not be limited only to the actual estimation methods, but also the sourcing and screening of the input data and guard not only against under-estimating of non-reported data but also against underreporting by disclosing companies. In practice, we detect underreporting by first checking if all the required data has been disclosed (i.e., materiality) and then by checking their quality using proxy data such as production data as a benchmark. Finally, we replace the outliers with respective quantiles of the carbon intensities distribution within respective peer groups. An overview of our approach is presented in Figure 2.

Secondly, we rely on a multi-model approach for emissions estimation. Limited trust should be placed in any individual estimation model or data source. The approach uses two models based on reported data (median and interpolation or regression model) and one model based on macro-level data (input-output model), which reduces the dependency on the accuracy and representativeness of reported carbon data (for details see our latest report on carbon emissions). In this approach, the biases of one estimation technique can be attenuated by the introduction of alternative methods, such as different source data, peer group classifications, and statistical methods. Moreover, the aggregation of multiple models stabilises prediction by reducing annual variation, which is crucial for quantitative investment strategies.

Finally, we aim for a pragmatic and balanced implementation of the precautionary principle. In practice, this means minimizing the risk of underestimating data while at the same time avoiding excessive volatility or distortion in the overall data distribution, which would make this data unsuitable to use in sustainable investment solutions.

¹ Report of the United Nations Conference on Environment and Development A/CONF.151/26, United Nations General Assembly, (1992).

² Bourguignon, D., The precautionary principal Definition, applications, and governance, European Parliamentary Research Service, (2015).

³ e.g., Vienna Convention for the Protection of the Ozone Layer (1987)

⁴ [ibid.] Report of the United Nations Conference on Environment and Development A/CONF.151/26, United Nations General Assembly, (1992).

⁵ [ibid.] Bourguignon, D., The precautionary principal Definition, applications, and governance, European Parliamentary Research Service, (2015).

⁶ The principle has been also discussed in the recent taxonomy regulation, for details see: https://finance.ec.europa.eu/system/files/2022-10/221011-sustainable-finance-platform-finance-report-usability_en_1.pdf

⁷ See Article 13 of the EU Commission Delegated Regulation for minimum standards for EU Climate Transition and Paris-Aligned benchmarks, 2020/1818 of 17 July 2020; EUR-Lex - 32020R1818 - EN - EUR-Lex

⁸ Simmons J., Kooroshy J., Bourne E., Jain M., Clements L., Mind the gaps: clarifying corporate carbon, FTSE Russell, (2022).

⁹ Bourguignon, Didier. The precautionary principal Definition, applications, and governance (European Parliamentary Research Service, 2015).

¹⁰ Hoepner, A., and Joeri R., Emissions estimations should embed a precautionary principle, Nature Climate Change 11.8 (2021): 638-640.

¹¹ Olesiewicz et al., (2022), Navigating the corporate disclosure gap: Modelling of Missing Not at Random Carbon Data.

¹² Allen, E., Lyons, K., Tavares, R., The application of machine learning to sustainable finance. Journal of Environmental Investing 8, (2017):115–129.

¹³ For 2019 (FTSE All World), 95th percentile was on average 8 times higher than median on a subsector level and 19 times higher on sector level.

¹⁴ Han et al., (2021), Estimation of Corporate Greenhouse Gas Emissions via Machine Learning, Bloomberg Quant Research.

© 2023 London Stock Exchange Group plc and its applicable group undertakings (the “LSE Group”). The LSE Group includes (1) FTSE International Limited (“FTSE”), (2) Frank Russell Company (“Russell”), (3) FTSE Global Debt Capital Markets Inc. and FTSE Global Debt Capital Markets Limited (together, “FTSE Canada”), (4) FTSE Fixed Income Europe Limited (“FTSE FI Europe”), (5) FTSE Fixed Income LLC (“FTSE FI”), (6) The Yield Book Inc (“YB”) and (7) Beyond Ratings S.A.S. (“BR”). All rights reserved.

FTSE Russell® is a trading name of FTSE, Russell, FTSE Canada, FTSE FI, FTSE FI Europe, YB and BR. “FTSE®”, “Russell®”, “FTSE Russell®”, “FTSE4Good®”, “ICB®”, “The Yield Book®”, “Beyond Ratings®” and all other trademarks and service marks used herein (whether registered or unregistered) are trademarks and/or service marks owned or licensed by the applicable member of the LSE Group or their respective licensors and are owned, or used under licence, by FTSE, Russell, FTSE Canada, FTSE FI, FTSE FI Europe, YB or BR. FTSE International Limited is authorised and regulated by the Financial Conduct Authority as a benchmark administrator.

All information is provided for information purposes only. All information and data contained in this publication is obtained by the LSE Group, from sources believed by it to be accurate and reliable. Because of the possibility of human and mechanical error as well as other factors, however, such information and data is provided "as is" without warranty of any kind. No member of the LSE Group nor their respective directors, officers, employees, partners or licensors make any claim, prediction, warranty or representation whatsoever, expressly or impliedly, either as to the accuracy, timeliness, completeness, merchantability of any information or of results to be obtained from the use of FTSE Russell products, including but not limited to indexes, data and analytics, or the fitness or suitability of the FTSE Russell products for any particular purpose to which they might be put. Any representation of historical data accessible through FTSE Russell products is provided for information purposes only and is not a reliable indicator of future performance.

No responsibility or liability can be accepted by any member of the LSE Group nor their respective directors, officers, employees, partners or licensors for (a) any loss or damage in whole or in part caused by, resulting from, or relating to any error (negligent or otherwise) or other circumstance involved in procuring, collecting, compiling, interpreting, analysing, editing, transcribing, transmitting, communicating or delivering any such information or data or from use of this document or links to this document or (b) any direct, indirect, special, consequential or incidental damages whatsoever, even if any member of the LSE Group is advised in advance of the possibility of such damages, resulting from the use of, or inability to use, such information.

No member of the LSE Group nor their respective directors, officers, employees, partners or licensors provide investment advice and nothing in this document should be taken as constituting financial or investment advice. No member of the LSE Group nor their respective directors, officers, employees, partners or licensors make any representation regarding the advisability of investing in any asset or whether such investment creates any legal or compliance risks for the investor. A decision to invest in any such asset should not be made in reliance on any information herein. Indexes cannot be invested in directly. Inclusion of an asset in an index is not a recommendation to buy, sell or hold that asset nor confirmation that any particular investor may lawfully buy, sell or hold the asset or an index containing the asset. The general information contained in this publication should not be acted upon without obtaining specific legal, tax, and investment advice from a licensed professional.

Past performance is no guarantee of future results. Charts and graphs are provided for illustrative purposes only. Index returns shown may not represent the results of the actual trading of investable assets. Certain returns shown may reflect back-tested performance. All performance presented prior to the index inception date is back-tested performance. Back-tested performance is not actual performance, but is hypothetical. The back-test calculations are based on the same methodology that was in effect when the index was officially launched. However, back-tested data may reflect the application of the index methodology with the benefit of hindsight, and the historic calculations of an index may change from month to month based on revisions to the underlying economic data used in the calculation of the index.

This document may contain forward-looking assessments. These are based upon a number of assumptions concerning future conditions that ultimately may prove to be inaccurate. Such forward-looking assessments are subject to risks and uncertainties and may be affected by various factors that may cause actual results to differ materially. No member of the LSE Group nor their licensors assume any duty to and do not undertake to update forward-looking assessments.

No part of this information may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without prior written permission of the applicable member of the LSE Group. Use and distribution of the LSE Group data requires a licence from FTSE, Russell, FTSE Canada, FTSE FI, FTSE FI Europe, YB, BR and/or their respective licensors.

Original Post

Editor's Note: The summary bullets for this article were chosen by Seeking Alpha editors.