A semi-supervised interactive algorithm for change point detection

Abstract

The goal of change point detection (CPD) is to identify abrupt changes in the statistics of signals or time series that reflect transitions in the underlying system’s properties or states. While many statistical and learning-based approaches have been proposed to address this task, most state-of-the-art methods still treat this problem in an unsupervised setting. As a result, there is often a large gap between the algorithm-detected results and the expected outcomes of the user. To bridge this gap, we propose an active-learning strategy for the CPD problem that combines with the one-class support vector machine (OCSVM) model, resulting in an interactive CPD algorithm that improves itself by querying the end-user. This approach enables us to focus on detecting the desired change points and ignore false-positives or irrelevant change points. We demonstrate that the interactive OCSVM model can be combined with various unsupervised CPD models to function in a semi-supervised setting, resulting in improved detection accuracy. Our experimental results on various simulated and real-life datasets demonstrate a significant improvement in detection performance on both single- and multi-channel time series, even with a limited number of queries.

Appendix: Limitation of AUROC in evaluating CPD algorithms

We discuss the limitations of using AUROC to evaluate CPD algorithms via a toy example. Before presenting the example, we will first introduce the definition of the AUROC metric in the context of CPD.

Since imbalanced data is common in CPD tasks, many papers in the CPD literature (e.g., KLCPD Chang et al. 2019, RuLSIF Liu et al. 2013, ABD Lee et al. 2018, and TIRE De Ryck et al. 2021) define the true positive rate (TPR) and false positive rate (FPR) as:

$$\begin{aligned} TPR = \dfrac{N_{TP}}{N_{GT}} \end{aligned}$$

(10)

and

$$\begin{aligned} FPR = \dfrac{N_{FP}}{N_{TP}+N_{FP}}. \end{aligned}$$

(11)

The ROC curve is then obtained by varying a detection threshold ($\upsilon$) from high to low. Point $(FPR, TPR) = (1.0, 1.0)$ is manually added to the ROC curve to ensure that a perfect performance corresponds to an AUROC of 1 (De Ryck et al. 2021).

Based on these definitions, we show an illustrative toy example in Fig. 7. Here, we show two possible detection results, e.g., by two hypothetical CPD algorithms. In Table 4, we summarize the number of samples in these two detection results and compute the f1-scores.

Fig. 7

Toy example for showing a limitation of the AUROC metric. In the left column, the two sub-figures simulate two detection results obtained from two hypothetical CPD algorithms on the same time series. We mark the locations of ground truth change points with the vertical lines in red. The black curve denotes the dissimilarity measurement produced by the CPD algorithm (a high value corresponds to a potential CP). Green points and black points represent true positive and false positive samples, respectively. Three colored flat dashed lines represent three values of the detection threshold $\upsilon$ during generating of the ROC curve. In the right column, the corresponding ROC curves are plotted. Three colored points on the ROC curves correspond to the $\upsilon$ values in the left sub-figures in the same colors

Table 4 Summary of detection results in toy example

As shown in Table 4, Result 2 detects one more change point and one less flase negative thereby producing a better f1-score than Result 1. However, the AUROC achieved by Result 2 is much worse than that of Result 1. This is because CPD algorithm 2 assigns a higher dissimilarity value to the false positive sample located at time step 375, causing the corresponding ROC curve to start from point (1.0, 0.0), resulting in an awkward shape. In real-world applications, the dissimilarity produced by a CPD algorithm is determined locally and is usually very sensitive to outlier data samples and initial model parameters. This makes the start of the ROC curve very sensitive to stochastic effects, often leading to awkward shapes as in the second example of Fig. 7. The same CPD algorithm can even result in very different AUROC values on the same dataset due to different initial weights. This is why we chose to evaluate the CPD approaches in this study based on the f1-score.