A detailed study of resampling algorithms for cyberattack classification in engineering applications

Dataset

The UNSW-NB15 dataset was created by the Cyber Range Lab at UNSW Canberra to simulate a heterogeneous environment of legitimate traffic and real attack traffic (Moustafa & Slay, 2015). The original dataset consists of 2,540,044 tuples and exhibits a high-class imbalance, as only the distribution of Normal type tuples accounts for more than 87% of the total samples. However, as previously indicated, the authors have published a row and feature reduced dataset (Moustafa, 2015). This latter set is already divided into test and trial sets, where an imbalance between the number of samples from different traffic classes is also maintained. The experiments in this research have been performed with the reduced dataset. Table 1 shows the 44 features and their type, of which 40 are numerical and four categorical (proto, service, state and attack_cat). Distribution of classes for the training and test sets is specified in Moustafa & Slay (2015).

The rationale behind choosing this dataset is that in comparison to the previously mentioned datasets, not only does UNSW-NB15 have the most realistic network traffic, but it also contains the highest variety of cyberattacks compared to a more recent IDS dataset, CIC-IDS2018 (Sharafaldin, Lashkari & Ghorbani, 2018), which only captures five different types of attacks.

Researchers who created the UNSW-NB15 group dataset feature in five groups depending on the information kept.

• Flow features. They represent the flow features extracted from the UNSW-NB15 dataset, including source and destination IP addresses and port numbers. Although these features are fundamental, they are not included in the reduced dataset to focus on more analytically valuable features. This table sets the groundwork for understanding the basic structure of network traffic data being analyzed.

• Basic features. They detail the basic features of network transactions captured in the UNSW-NB15 dataset. These features include the state of the transaction, duration, bytes transmitted between source and destination, packet counts, and service type. This diverse set of features is crucial for initial analysis and detection models, offering insights into the general behavior of network traffic and potential anomalies.

• Content features. They focus on content features, which include TCP window sizes, sequence numbers, packet sizes, transaction depths, and response body lengths. These features dive deeper into the network traffic content, providing a richer context for anomaly detection by examining the specifics of the transmitted data and the characteristics of individual packets.

• Time features. They introduce time-based features, such as source and destination jitter, packet inter-arrival times, and TCP connection setup times. Time features are vital for identifying anomalies that manifest through unusual time patterns, such as delays or rapid sequences of activities, which might indicate malicious behavior or network issues.

• Generated features. They list generated features, which are derived from the analysis of network traffic, including binary flags for specific conditions (e.g., FTP logins), counters for connection patterns, and transaction statuses. These features represent higher-level abstractions and summaries of the traffic, focusing on behavioral patterns that help distinguish between normal and anomalous activities.

Resampling techniques

A synthetically generated dataset was used to visualize the effect of each of the methods applied in the research. It can be visualized in Fig. 1. It consists of a three-class dataset with a total o 479 samples. One of them being the majority class with 379 samples. The remaining two are composed of 50 samples. To understand each resampling technique, Fig. 2 shows the original dataset and the effect of the resampling algorithms on it. Depending on the applied approach, each class’s final number of samples is highlighted in red and green when oversampling or undersampling, respectively.

From a cybersecurity perspective, resampling actual traffic results in synthetic traffic, which is helpful for two main reasons: on the one hand, it allows dealing with imbalance by generating traffic similar to that observed in real networks. On the other hand, it enables CPS attack simulations that exploit previously unknown hardware, firmware, or software vulnerabilities, commonly known as 0-day attacks. CPS systems containing many heterogeneous devices are more likely to suffer these attacks.

• Oversampling. Oversampling methods seek to generate more information from minority classes in a balanced dataset to increase the number of samples and facilitate identifying decision boundaries between classes.

  • –

    Synthetic Minority Oversampling Technique (SMOTE). This method is one of the most popular when working with unbalanced datasets. This algorithm is based on generating synthetic samples of the minority classes from the nearest neighbor samples. Two factors are involved when generating synthetic samples using SMOTE: the number of neighbors to be considered during generation and the sampling factor that will govern the number of samples generated in the minority class. The detailed procedure is described in Chawla et al. (2002) and is summarized as follows: for each sample in the minority class, its k-nearest neighbors are computed and, depending on the sampling factor, a subset of neighbors is randomly selected. For each selected neighbor, a synthetic sample is generated from the difference of feature vectors between the sample to be treated and its neighbor.

  • –

    SMOTE-Borderline1 (BLINE-1). In classification problems, the calculation of the decision boundaries of a class consists of estimating a function that separates the samples belonging to that class from the rest. Given the complexity and quantity of available data, finding the optimal decision boundaries is a complex task that requires a high computational time. In addition, the existence of unbalanced datasets or borderline samples must be added. The SMOTE-Borderline1 method is a variation of the original SMOTE method aimed at identifying class boundaries by generating synthetic samples. The algorithm proposed by Han, Wang & Mao, 2005 and the basic idea consists of the generation of synthetic samples of the minority class that are surrounded to a greater extent by samples of the majority class among their k neighbors.

  • –

    Adaptive synthetic (ADASYN). Another alternative is proposed in He et al. (2008) when analyzing highly imbalanced datasets. This sampling technique increases the number of samples in the minority class by shifting the decision boundary toward those samples that are more difficult to learn. In this way, the ability to classify minority samples is improved. The algorithm works as follows: the first step determines the number of total synthetic samples to be generated. Then, the calculated number of synthetic samples for each of the minority class samples is generated. This number is obtained by the number of samples belonging to the majority class among its k-nearest neighbors. Finally, for each minority sample, a neighbor belonging to the minority class is chosen and a synthetic sample will be generated from the difference between their feature vectors.

• Undersampling. These methods aim to reduce the number of samples in the training set while maintaining the classification accuracy of the final model.

  • –

    Random undersampling (RU). The random undersampling method is the most basic technique for balancing a dataset. Specifically, this method randomly removes samples from the majority class until the classes are balanced. It is important to note that this method only works to decrease the number of samples in the majority classes and does not introduce any data variety after removing samples.

  • –

    Cluster centroids (CC). This algorithm is based on generating a new set of samples from the class to be resampled. Unlike the other undersampling methods, no selection criterion is applied to decide which samples become part of the final training set. Nevertheless, the undersampling will be performed by generating fewer new samples. The cluster centroids technique uses the unsupervised KMeans algorithm to estimate the clusters of the original set and, once identified, replace the original samples with the centroids of the class to which they belong.

  • –

    NearMiss (NM). The literature describes three heuristics for sample selection based on Nearest Neighbors algorithm. These are presented in Mani & Zhang (2003). In NearMiss-1 (NM1), a subset of samples from the majority class is selected such that the average distance between each majority sample and the nearest minority neighbors is the smallest. In NearMiss-2 (NM2), a subset of samples from the majority class is selected such that the average distance between the processed sample and the farthest minority neighbors is the smallest. Finally, the NearMiss-3 (NM3) heuristic consists of two steps: the first step is to search for a specified number of majority class neighbors for each minority class sample. In the second step, the selected majority of samples are the farthest away from their neighboring samples. Both NearMiss-1 and NearMiss-2 can be influenced by noise, but NearMiss-3 maintains close majority samples due to the first-stage selection, where it is ensured that several samples of the minority class surround each majority sample. For this reason, the experiments of this research employ the latter heuristic (NM-3).

  • –

    Repeated edited nearest neighbour (RENN). This undersampling method is based on the repeated application of the edited nearest neighbours (Wilson, 1972) algorithm. It can be summed up as follows: for each majority sample, its K nearest neighbors are calculated, and a selection criterion will be applied to determine whether the sample is kept. There are two selection criteria: the first one considers that the majority sample should be removed in case the majority of its k-nearest neighbors do not belong to the same class; the second, more restrictive, implies that all k-neighbors must belong to the class of the processed sample; otherwise, it will be removed. In the case of RENN, the undersampling strategy applied in this research, the repetition of the algorithm implies further sample elimination.

  • –

    Tomek Links (TKL). This technique is based on the concept of Tomek’s link. The explanation of this concept and the detailed implementation of the algorithm are revised in Tomek (1976). A pair of samples meeting two criteria in a dataset is called a Tomek link: (i) the samples belong to different classes and (ii) they are mutually nearest neighbors. Applying undersampling based on Tomek’s links may be performed by eliminating both samples or the one belonging to the majority class. For experiments carried out in this contribution, a second criterion is applied.

Table 2 shows the configuration used for each resampling method. Resampling is performed using Python 3.8 with Pandas, Numpy and Imbalanced-learn (Lemaître, Nogueira & Aridas, 2017) libraries. To decide the resampling strategy, the approach described in “Experimental Design” has been applied when the algorithm is capable of controlled resampling. Otherwise, the $auto$ value of the sampling_strategy parameter was chosen. Following algorithms do not allow to set a custom resampling factor: ADASYN, repeated edited nearest neighbors and Tomek Links.

Classification algorithms

For the evaluation of resampling algorithms, classical machine learning models have been trained. Each algorithm was selected based on its distinct characteristics and proven effectiveness in handling various aspects of cyberattack detection. SVM (Hearst et al., 1998) is renowned for its robustness in high-dimensional spaces, making it ideal for complex attack patterns (Ahmad et al., 2018). Decision trees (Quinlan, 1986) provide interpretability, which is crucial for understanding the decision-making process behind classifying types of cyberattacks (Ferrag et al., 2020). KNN (Guo et al., 2003) was chosen for its simplicity and efficacy in instances where attacks form identifiable clusters (Alharbi et al., 2021). Lastly, Naive Bayes (Zhang, 2004) offers fast prediction capabilities, especially useful for large datasets typical in cybersecurity (Gu & Lu, 2021). Later, these models are combined to build a more robust solution, which will be described in-depth in “Experimental Design”.

Evaluation metrics

Our study evaluates intrusion detection models’ performance using several metrics. Among the metrics used are accuracy, precision, recall/detection rate (DR), F1-score, and geometric mean (G-mean), which are calculated based on true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Each metric is described in Mogollón-Gutiérrez et al. (2023) together with its calculation using confusion matrix.

Dataset preproccesing

As usual, the data must be prepared prior to applying classification algorithms. This task has been meticulously performed using Python 3.8, alongside Pandas and Scikit-Learn for efficient data handling and preprocessing (Pedregosa et al., 2011). To ensure the integrity and quality of our dataset, we implemented thorough validation and cleaning steps, including outlier detection and handling missing values. Numerical features were normalized to a standard score, achieving a Gaussian distribution with mean zero and unit variance, thereby enhancing model comparability and performance. Categorical features underwent ordinal encoding, carefully chosen to maintain the inherent structure and significance of the data, thereby avoiding the introduction of ordinal assumptions where inappropriate. We confirmed the absence of redundant samples, as highlighted in Moustafa & Slay (2016), further ensuring the robustness of our dataset. To prevent data leakage, a rigorous approach was adopted where preprocessing steps were applied within each fold of the cross-validation process, ensuring that our model evaluation remains unbiased and reflective of true predictive performance. Figure 4 graphically details our preprocessing workflow, underlining our commitment to data integrity and the reproducibility of our research findings.

Generation of classification models

Due to the nature of the UNSW-NB15 dataset, there is a clear imbalance with a majority presence of legitimate behavior. This fact may have a negative impact on the performance of a CPS traffic classifier in a production environment. Our proposal addresses this issue by using a binary model system that allows for the classification of network traffic among the ten classes considered. For this purpose, ten binary classifications are built to distinguish the traffic of one category from the rest, following a one-vs-rest approach. These models are generated in a balanced way considering previous resampling algorithms. Thus, the goal of this step is the generation of 10 binary models for each classification algorithm. This is repeated for each pair of resampling techniques (3 × 5 resampling combinations) and without resampling the dataset, resulting in 16 grouped experiments. In the following step, these models will be combined to generate two-step ensemble models.

To better understand the model generation process, Fig. 5 shows a step-by-step diagram of the tasks that must be performed to generate a specialized binary model associated with a traffic category.

Starting from the UNSW-NB15 dataset, generating a specialized binary model is as follows: the first step is to create a binary dataset composed of samples of the target category and the rest. The number of samples of the target category in the new dataset is the number of samples in the original dataset. The number of samples in the target category is required ( $n\mathrm{\_}samples$) to determine the number of samples for the rest of the categories. The value obtained is divided by the number of categories minus one ( $n\mathrm{\_}samples\mathrm{\_}c$). Since UNSW-NB15 consists of ten categories, it is always divided by nine. $n\mathrm{\_}samples\mathrm{\_}c$ will serve as a threshold to decide the resampling type for each category. In case $n\mathrm{\_}samples>n\mathrm{\_}samples\mathrm{\_}c$ means there are more samples than expected, the undersampling method will be applied. When needing more samples, $n\_samples<n\_samples\_c$, the preprocessed category is oversampled. Once all categories are resampled, together with the samples of the initially selected category, a new resampled binary dataset is generated. As can be appreciated, one oversampling technique and one undersampling technique are used.

The second step consists of the generation of the binary model. For the four classification algorithms reviewed above, a hyperparameter tuning, together with preprocessing, is performed to generate, for each algorithm, the model that best fits the dataset. This optimization uses Grid Search and five-fold cross validation methodology, taking the macro-F1 metric as an evaluation criterion during the hyperparameter fitting process. The reason for considering macro-F1 is that the new dataset has two main classes; therefore, both classes are considered equally important. Finally, after having trained four models, we proceed to select the best model based on the evaluation metrics explained in “Evaluation Metrics”. At this point, a specialized binary model for one category is generated. This process must be repeated for each category in dataset and each pair of resampling algorithms.

To sum up, 640 experiments are required. Since three oversampling techniques and five undersampling techniques have been studied, a total of fifteen different combinations have been tested. In addition, the possibility of not using any resampling technique is also considered, resulting in sixteen possible combinations of experiments. These combinations train the classifiers for each of the 10 types of network traffic considered. Furthermore, the process is repeated for each of the four classification algorithms, generating 640 classification models. Among all of them, and according to the performance metrics considered, the final system includes the best model for each network traffic class: the final system includes ten classification models.

The experiments generate balanced training datasets by applying oversampling and undersampling techniques. Some resampling methods can be configured by including the cardinality of the resulting datasets. That is, the number of samples desired after applying the resampling techniques can be configured in some resampling algorithms but not others. Then, the class distribution of the resulting datasets is computed as follows:

• For the resampling methods where the final cardinality can be defined, the datasets used to generate the binary models are generated in a balanced way. Thus, for each model, half of the samples belong to one category, and the other half is composed of traffic samples from the other nine categories. This second half is configured to contain the same number of samples from each of the remaining nine categories, where all categories have the same weight (same cardinality) in the second set of samples. For example, according to dataset distribution, the Backdoor category consists of 1,746 samples, so the resulting training set for the generation of the binary model consists of 3,492 tuples: half of the samples of the “backdoor” type (1,746), and the other half (1,746) of “not backdoor” type. This second half is built with the same cardinality (194) for each of the nine categories. To reach the 194 samples, oversampling or undersampling tasks must be performed on the category processed. The result of this process is shown in Fig. 6 for the “backdoor” category.

• For the resampling algorithms where the resulting number of samples cannot be controlled, the decision to oversample or undersample is based on the same idea. In this case, the number of samples in each category is calculated, and then oversampling is applied to categories with fewer samples and undersampling to those with more samples than the number obtained.

Algorithm proposed in Mogollón-Gutiérrez et al. (2023) shows the process of generating training sets for all dataset categories. In Fig. 5, the activities that are part of the highlighted section should be repeated for each category of the original dataset. This results in a specific dataset for each traffic category generated from an oversampling and an undersampling technique.

Once the specialized datasets are generated, hyperparameter tuning for binary models is carried out using GridSearch and five-fold cross-validation using Python 3.8 and Pandas, Numpy and Scikit-Learn (Pedregosa et al., 2011) libraries. This implies that, for each algorithm, all possible combinations of parameters will be tested over five partitions on the original training set, thus, estimating the optimal values for each parameter. Table 3 shows the hyperparameter search space for each of the applied algorithms.

Two-stage classification model

With 640 models already generated, the next step is to group binary models by the pair of resampling techniques applied for its generation. Thus, 40 models are generated for each one. These binary models, in turn, are divided into 10 categories. In addition, for each category, four classification algorithms are trained.

In order to evaluate the performance of each resampling method, a criterion must be established to select the binary model (out of four) that performs the best in each category. This choice is based on the evaluation metrics obtained individually, specifically macro-F1. As previously explained, this metric considers both classes equally and does not assume the number of samples that make up each class. The selected model becomes part of the two-stage model for a pair of resampling algorithms. In total, 15 approaches will have to be evaluated and compared with the baseline approach. This baseline performance is obtained through the combination of non-resampled binary models.

Following the same idea, an improved solution is proposed. It is based on selecting the best binary model for each category among 16 approaches. Thus, each model is selected with the resampling techniques that yield the best results. This new approach aims to compare its performance with the baseline solution and find the resampling strategies that best fit each kind of traffic.

The construction of the system for both approaches is the same, and it is represented graphically in Fig. 7. It is based on a two-step classification model organized into two phases, as described in Mogollón-Gutiérrez et al. (2023). In this framework, the first classification level aims to detect suspicious traffic. Then, the second classification level is intended to classify the cyberattack among labeled categories.

Binary models results

Table 5 shows macro-F1 after generating balanced binary models with the different groups of resampling techniques using NB classification algorithm. To evaluate each model, the original test set has been binarized into two classes: one for the category the model seeks to detect and another class for the rest of the traffic. When applying the classical NB classification algorithm, only the Worms class yields better results when no resampling technique is applied. This may be due to the fact that the set of samples labelled as Worms is small and, therefore, the generation of synthetic samples is not a clear improvement. Detecting individual attacks is not very good, producing macro-F1 values around 0.5 for Analysis, Backdoor, Reconnaissance, Shellcode and Worms. Slightly better results are achieved for DoS and Exploits, which reach 0.6128 and 0.6640 for macro-F1, respectively. Regarding legitimate traffic detection, macro-F1 reaches 0.7692 after resampling the training set with Borderline1 and RENN for oversampling and undersampling, respectively.

Table 6 considers macro-F1 as an evaluation metric for the binary models obtained from the training of the KNN classification algorithm. As in the previous experiment, this metric aims to evaluate the classification ability of the specialized model between one type of network traffic and the rest. Analogous to the previous models, Worms obtains macro-F1 when the imbalance is not solved. The same applies in the case of Exploits, Reconnaissance and Shellcode classes. However, for these three classes, there are resampling methods that achieve similar performance. When training KNN for traffic detection collected in UNSW-NB15, the models increase the generalization capability for Backdoor, DoS and Fuzzers when using ADASYN+RENN. Regarding attack detection, the Normal model achieves 0.8731 for macro-F1 using SMOTE+TKL.

Table 7 evaluates the detection capability between a class and the rest, considering macro-F1 as the evaluation metric. It shows the results after training models with the SVM algorithm. The optimal hyperplane search obtains the best evaluation using a raw dataset for generic attack traffic. However, it should be noted that performance metrics for this class are very high, and the differences between the different resampling techniques are minor. Furthermore, an interesting finding is that the best-performing resampling method for SVM models is RENN. for majority sample reduction. Regarding suspicious network traffic detection, Borderline1 + RENN yields the best macro-F1 with 0.8528.

Table 8 groups the models built after training DT. Following the same criterion, macro-F1 is considered for estimating the most accurate model. Again, Fuzzers, Generic and Worms are more easily distinguishable from the other samples when the imbalance problem is not addressed. Nevertheless, it should be noted that most classes achieve the best performance when the DT algorithm is considered. ADASYN + TKL are the best combination of oversampling and undersampling techniques for Exploits, Reconnaissance and Shellcode, achieving macro-F1 of 80.14, 92.07 and 73.24, respectively. The Normal model, responsible for distinguishing between legitimate and illegitimate traffic, achieves 91.14 for macro-F1 when ADASYN+CC is performed.

Approach 1: evaluation over resampling methods

Following the model generation explained in “Generation of Classification Models”, Fig. 8 first approach displays how binary model selection is performed when oversampling and undersampling techniques are used to generate a resampled dataset. Table 9 collects the evaluation metrics calculated for the 16 experiments carried out. These 16 experiments are the result of combining the three oversampling methods (SMOTE, Borderline-1 SMOTE and ADASYN) with the five undersampling methods (random undersampling, cluster centroids, Near-Miss 3, repeated edited nearest neighbors and Tomek Links). Evaluation in imbalanced domains has some peculiarities. Metrics such as accuracy can lead to errors in the interpretation of results. Precision, recall, F1 and G-mean metrics have been collected to solve this problem. To test each of the 16 experiments, the test set described in Moustafa & Slay (2015) is used.

Unsurprisingly, undersampling techniques lead to a more significant impact on the classifier’s performance than oversampling algorithms. As shown in Table 4, undersampling is carried out more frequently. Our experiments demonstrate that each undersampling technique produces similar evaluation metrics among the three oversampling algorithms. In this experiment, oversampling techniques have a slight impact on classification tasks.

When oversampling is applied, either SMOTE, Borderline-1 or ADASYN, the best performing undersampling method is RENN, obtaining an F1 around 0.77 and a G-mean around 0.84. When using RENN with the oversampling methods, SMOTE + RENN shows the best results with 0.7708, 0.7918, 0.7708, 0.7760, 0.8428 for accuracy, precision, recall, F1 and G-mean, respectively. The rationale behind RENN’s performance is that it works by removing traffic samples whose category differs from a majority subset. Removal of these potential outliers outperforms the rest of the methods.

G-mean indicates true negative rate (specificity) and true positive rate (sensitivity). Although many research works focus on evaluating their methods using accuracy, G-mean allows considering class imbalance when measuring the performance of a classifier. Similarly to F1, G-mean achieves higher values when using RENN with any of the studied oversampling algorithms. In these experiments, this metric ranges from 0.8389 to 0.8428, indicating a good balance between individual accuracies among ten traffic categories. Regarding NM3 experiments, G-mean reaches 0.4686, which is the best case for this undersampling method when combined with ADASYN. These metrics evidence severe misclassification in some classes.

Figure 9 graphically compares the 15 experiments performed and, in turn, how their performance ranks against not applying any balancing technique. Note that this baseline performance has been obtained through the same experimental design but without using resampled datasets for each traffic category. Baseline model achieves 0.7207, 0.8071, 0.7207, 0.7537 and 0.8242 for accuracy, precision, recall, F1-score and G-mean, respectively. It can be seen that nine of the 15 experiments outperform the system built from unbalanced datasets (baseline performance). These nine experiments correspond to SMOTE, Borderline1 and ADASYN techniques with RU, RENN and TKL. Therefore, these three undersampling methods improve the base performance in combination with any of the three oversampling methods.

The improvement achieved by deleting Tomek links from the majority classes demonstrates that UNSW-NB15 contains noisy samples for several traffic categories. As discussed, this method reduces noise by removing adjacent samples from different classes. Contrary to expectations, this study demonstrates that the application of RU supposes a significant improvement in terms of F1 compared to baseline performance. Another interesting finding is that NM3 undersampling is not recommended when working with imbalanced problems in the network traffic domain. Low results may be explained by the fact that since these techniques remove relevant traffic data during the undersampling process, classifiers cannot establish well-defined borders between different cyberattack samples.

Approach 2: proposed classification model

A second approach is proposed to improve the performance of the previous experiments. It consists of the combination of the best models for each traffic category collected in UNSW-NB15. Unlike the experiments discussed above, where an oversampling and an undersampling technique were used to generate all the specialized models and create the two-step model, this second experiment selects the best-performing model. It compares their performance for each pair of resampling operations. This improvement supposed this approach to be considered as the proposal of this study. Since this is a two-phase model, the first step will be to evaluate the performance of the binary models in charge of attack detection (Normal class), i.e., the one able to distinguish between legitimate and illegitimate traffic. For this purpose, based on the macro F1-score evaluation metric, the best model is chosen. This metric, which relates precision and recall, aims to minimize the number of samples incorrectly classified as legitimate traffic so that the remaining nine models can reevaluate these at the second classification level. Figure 10 contains the evaluation for the first step of the model. In particular, generating the binary model for legitimate traffic detection from ADASYN + CC results in the system with the best global classification capacity, reaching a value slightly above 0.91.

Once the model has been selected for the first phase, the second classification level is composed of the sampling models that achieve the best performance individually for each category. Thus, the best-performing algorithm and sampling techniques are selected for each type of traffic. Since the construction of these models aims to distinguish between one class and the rest, the evaluation methodology used has assigned the same weight to both classes (macro-F1), i.e., the evaluation is not affected by the number of samples of each class.

For each pair of resampling methods, Table 10 shows macro-F1 for the best algorithm (among DT, NB, KNN and SVM) for each kind of attack present in UNSW-NB15. In most cases, resampling yields slightly better results than original data without balancing. Unlike previous experiments, these results showed that RENN is not the best choice for undersampling imbalanced data. Depending on data distribution, it is necessary to find the pair of resampling methods that fit better for training a model able to distinguish one traffic from the rest. Generic, Normal and Reconnaissance are the most easily identifiable categories, obtaining 0.9879, 0.9114 and 0.9207 for macro-F1, respectively. Generic traffic achieves the best result without resampling. An explanation for this might be that there are enough samples in the train set and no fuzzy borders between this class and the rest. Regarding Normal and Generic traffic, generating a balanced train set using ADASYN produces better results than raw one-vs-rest train sets. Analysis and Backdoor OvR specialized models do not produce good results, but a performance increase is found when SMOTE + RENN is used in comparison with no resampling approach.

Table 11 lists detailed information on the models that make up the final system. The resampling techniques used for sample generation in constructing each specialized binary model and the best-performing algorithm are indicated. For the first classification phase, the Normal model will be in charge of detecting legitimate behavior. For this purpose, ADASYN + CC is applied, which yields the best classification results along with the decision trees algorithm. The best-performing algorithms at the second classification level were SVM and decision trees. The tuned hyperparameters used for generating each model are indicated to facilitate the replication of the experiments. Notice that the environment should be configured Python 3.8 and Scikit-Learn.

A statistical procedure has been carried out to determine if the performance of the second approach can be considered statistically significant. As a reminder, the baseline consists of a hybrid two-stage ensemble without dealing with class imbalance. In addition, four new non-hybrid two-stage ensembles are built employing classification algorithms widely used in the literature (NB, KNN, SVM, DT). The main difference with the baseline approach is that each of these four classifiers is built using only one algorithm per traffic category. Thus, the NB-based model consists of 10 binary models obtained through the training of the Naïve Bayes algorithm. The same occurs for the three remaining models. This process aims to verify if the obtained results are statistically significant compared to six more classifiers. A 10-fold cross validation is executed for each classifier to estimate the average F1.

The statistical analysis was conducted for seven experiments with 10 paired samples as shown in Table 12. The tests’ family-wise significance level (p-value) is set to 0.05. The first step is to conduct the Shapiro & Wilk (1965) test to check normality. It has failed to reject the null hypothesis that the population is normal for all populations (minimal observed p-value = 0.241). Therefore, it is assumed that all populations are normal. Bartlett’s test is applied (Arsham & Lovric, 2011) for homogeneity, and it failed to reject the null hypothesis (p = 0.761) that the data is homoscedastic. This implies that data is homoscedastic, so that assumption is considered. There are more than two populations and all are normal and homoscedastic so repeated measures ANOVA (Kaufmann & Schering, 2014) are used to determine if there are any significant differences between the mean values of the populations. If the results of the ANOVA test are significant, post-hoc Tukey HSD tests (Tukey, 1949) are needed to infer which differences are significant. Each population’s mean value (M) and standard deviation (SD) are reported. Populations are significantly different if their confidence intervals are not overlapping. The null hypothesis is rejected ( $p=1\ast 10e^{-7}$) for repeated measures ANOVA that there is a difference between the mean values of the populations NB (M = 0.595 + −0.013, SD = 0.022), SVM (M = 0.710 + −0.013, SD = 0.016), KNN (M = 0.724 + −0.013, SD = 0.023), DT (M = 0.746 + −0.013, SD = 0.023), Baseline (M = 0.749 + −0.013, SD = 0.018), Approach 1 (M = 0.780 + −0.013, SD = 0.014), and proposed model (M = 0.797 + −0.013, SD = 0.018). Therefore, it is assumed that there is a statistically significant difference between the mean values of the populations. Based on the post-hoc Tukey HSD test, the following groups are considered to have no significant differences: DT and Baseline. These relationships may be explained by the fact that the Baseline two-step ensemble classifier contains several DT binary models also included in the DT-Based approach. All other differences are significant. This implies that present results are statistically significant to be compared with state-of-art approaches.

Once the proposal demonstrates to be statistically significant, Table 13 contains the confusion matrix obtained from the two-phase model built from binary classifiers shown in Table 11 using the test set. Attending the matrix’s main diagonal, the model performs well for the categories Exploits, Generic, Normal, Reconnaissance and Shellcode, with many correctly classified samples. In an intrusion detection system, attack detection is critical. In this regard, it is important to emphasize the number of samples correctly classified as Normal, which is 33,005 out of 37,000. Regarding the second level classification, in charge of categorising cyberattacks, the matrix shows how much of the traffic generated from other attacks is classified as DoS. Two reasons could explain this behavior. On the one hand, the environment where this traffic has been generated has suffered few changes during the attacks. Therefore, the samples used to generate the DoS lack helpful information to distinguish these attacks from the rest. On the other hand, the combination of ADASYN + RENN may not be optimal. While ADASYN tries to generate new synthetic samples that are hard to learn (near borderline) for minority attacks, RENN removes the majority of samples targeted with a label different from the surrounding ones. If tested samples belonging to undersampled classes are located near the borderline, the classification task is more prone to failure. Regarding traffic classification labelled as Analysis or Backdoor, the task is challenging due to data. These findings are also reported in other studies (Khan et al., 2019; Zhang et al., 2020; Manimurugan, 2021).

From the confusion matrix shown in Table 13, the proposed model’s performance can be estimated by calculating the aforementioned evaluation metrics.

Table 14 shows the comparison between (1) a non-hybrid two-stage ensemble employing four classification algorithms widely used in the literature (NB, KNN, SVM, DT) without resampling, (2) a hybrid two-stage ensemble without dealing with class imbalance (Baseline) and (3) Proposal. These three sets of results are compared against some existing state-of-the-art works. Since binary classification corresponds to the first classification step, only one model for benign traffic needs to be tested. For this reason, the DT-based model is chosen as the baseline for binary classification.

In the current study, when the model is made up of unbalanced models, the two-step ensemble does not produce the best results compared to other proposals in the literature. However, after selecting the correct balanced models and following the two-stage schema, an improvement is observed in both steps. For binary classification, taking baseline as a reference, a significant improvement is observed in the first stage, around 5% for F1-score and 7% for G-mean, which aims to distinguish between attack and legitimate behaviour. In the second stage, where suspicious traffic is classified among the nine remaining categories, results lead to similar conclusions improving over 4% for F1-score and 3% for G-mean compared to baseline. Compared with other state-of-art works, the proposed contribution delivers significantly better results in terms of F1. These results prove that choosing the suitable resampling operations and classification algorithm for a binary OvR dataset can improve performance when developing a two-step multi-class classification.