As global temperatures continue to rise, climate mitigation strategies such as stratospheric aerosol injections (SAI) are increasingly discussed, but the downstream effects of these strategies are not well understood. As such, there is interest in developing statistical methods to quantify the evolution of climate variable relationships during the time period surrounding an SAI. Feature importance applied to echo state network (ESN) models has been proposed as a way to understand the effects of SAI using a data-driven model. This approach depends on the ESN fitting the data well. If not, the feature importance may place importance on features that are not representative of the underlying relationships. Typically, time series prediction models such as ESNs are assessed using out-of-sample performance metrics that divide the times series into separate training and testing sets. However, this model assessment approach is geared towards forecasting applications and not scenarios such as the motivating SAI example where the objective is using a data driven model to capture variable relationships. In this paper, we demonstrate a novel use of climate model replicates to investigate the applicability of the commonly used repeated hold-out model assessment approach for the SAI application. Simulations of an SAI are generated using a simplified climate model, and different initialization conditions are used to provide independent training and testing sets containing the same SAI event. The climate model replicates enable out-of-sample measures of model performance, which are compared to the single time series hold-out validation approach. For our case study, it is found that the repeated hold-out sample performance is comparable, but conservative, to the replicate out-of-sample performance when the training set contains enough time after the aerosol injection.

随着全球气温持续上升，人们越来越多地讨论平流层气溶胶注入（SAI）等气候缓解策略，但这些策略的下游影响尚不清楚。因此，人们有兴趣开发统计方法来量化 SAI 期间气候变量关系的演变。已提出将特征重要性应用于回声状态网络 (ESN) 模型，作为使用数据驱动模型了解 SAI 效果的一种方法。这种方法依赖于ESN对数据的良好拟合。如果不是，特征重要性可能会将重要性放在不代表底层关系的特征上。通常，时间序列预测模型（例如 ESN）是使用样本外性能指标进行评估的，这些指标将时间序列划分为单独的训练集和测试集。然而，这种模型评估方法适用于预测应用程序，而不是诸如激励 SAI 示例之类的场景，其中目标是使用数据驱动模型来捕获变量关系。在本文中，我们展示了气候模型重复的一种新颖用途，以研究 SAI 应用中常用的重复保留模型评估方法的适用性。 SAI 的模拟是使用简化的气候模型生成的，并且使用不同的初始化条件来提供包含相同 SAI 事件的独立训练和测试集。气候模型复制可以对模型性能进行样本外测量，并与单一时间序列保留验证方法进行比较。对于我们的案例研究，发现当训练集在气溶胶注射后包含足够的时间时，重复保留样本的性能与重复样本外的性能相当，但比较保守。