Machine Learning Techniques for Anomaly Detection in High-Frequency Time Series of Wind Speed and Greenhouse Gas Concentration Measurements

Abstract

Fluxes of greenhouse gases (GHG) may be assessed in situ using the eddy covariance method through processing high-frequency measurements of gas concentration and wind speed acquired at certain sites, e.g., carbon measurement test areas of the pilot project of the Ministry of Education and Science of Russia. The measurements commonly come with noise, anomalies, and gaps of various natures. These anomalies result in biased GHG flux estimates. There are a number of empirical and heuristic approaches for filtering noise and anomalies, as well as for gap-filling. These approaches are characterized by many tuning parameters that are commonly adjusted by an expert, which is a limiting factor for large-scale deployment of GHG monitoring stations. In this study, we propose an alternative approach for anomaly detection in high-frequency measurements of GHG concentration and wind speed. Our approach is based on machine learning techniques. This approach is characterized by a lower number of tuning parameters. The goal of our study is to develop a fully automated data preprocessing routine based on machine learning algorithms. We collected the dataset of high-frequency GHG concentration and wind speed measurements from one of the carbon measurement test areas. In order to compare anomaly detection algorithms, we labeled anomalies in a subset of this dataset. We present two approaches for anomaly detection, namely: (a) identification of outliers based on the error magnitude in time series statistical forecasts performed by a machine learning (ML) algorithm; and (b) classification of anomalies using an ML model trained on the labeled dataset of outliers we mentioned above. We compared the approaches and algorithms based on the F1-score metric assessed with respect to an expert-labeled subset of anomalies in GHG concentration and wind speed time series. Within the forecast-error based approach, we trained several ML models: the ARIMA autoregression method, the CatBoost model for autoregression, the CatBoost model for forecasting employing additional features, and the LSTM artificial neural network. Within the supervised classification approach, we tested the CatBoost classification model. We demonstrate that ML models for forecasting deliver a high quality of time series prediction within the autoregression approach. We also show that the anomaly identification method based on the autoregression approach delivers the best quality with the F1-score reaching $0.812$.