Published online Oct 18, 2023.
https://doi.org/10.3802/jgo.2024.35.e17
A machine learning-based prediction model of pelvic lymph node metastasis in women with early-stage cervical cancer
Abstract
Objective
To develop a novel machine learning-based preoperative prediction model for pelvic lymph node metastasis (PLNM) in early-stage cervical cancer by combining the clinical findings and preoperative computerized tomography (CT) of the whole abdomen and pelvis.
Methods
Patients diagnosed with International Federation of Gynecology and Obstetrics stage IA2-IIA1 squamous cell carcinoma, adenocarcinoma, and adenosquamous carcinoma of the cervix who had primary radical surgery with bilateral pelvic lymphadenectomy from January 1, 2003 to December 31, 2020, were included. Seven supervised machine learning algorithms, including logistic regression, random forest, support vector machine, adaptive boosting, gradient boosting, extreme gradient boosting, and category boosting, were used to evaluate the risk of PLNM.
Results
PLNM was found in 199 (23.9%) of 832 patients included. Younger age, larger tumor size, higher stage, no prior conization, tumor appearance, adenosquamous histology, and vaginal metastasis as well as the CT findings of larger tumor size, parametrial metastasis, pelvic lymph node enlargement, and vaginal metastasis, were significantly associated with PLNM. The models’ predictive performance, including accuracy (89.1%–90.6%), area under the receiver operating characteristics curve (86.9%–91.0%), sensitivity (77.4%–82.4%), specificity (92.1%–94.3%), positive predictive value (77.0%–81.7%), and negative predictive value (93.0%–94.4%), appeared satisfactory and comparable among all the algorithms. After optimizing the model’s decision threshold to enhance the sensitivity to at least 95%, the ‘highly sensitive’ model was obtained with a 2.5%–4.4% false-negative rate of PLNM prediction.
Conclusion
We developed prediction models for PLNM in early-stage cervical cancer with promising prediction performance in our setting. Further external validation in other populations is needed with potential clinical applications.
Synopsis
We developed a novel machine learning-based preoperative prediction model with good predictive performance for pelvic lymph node metastasis in early-stage cervical cancer by combining the clinical findings and preoperative computerized tomography of the whole abdomen and pelvis. There are potential applications in two distinct clinical scenarios.
INTRODUCTION
Cervical cancer remains a major cause of significant long-term morbidity and mortality in women worldwide. According to the updated report of global cancer statistics 2020 examining GLOBOCAN estimates of incidence and mortality for 36 cancers in 185 countries, cervical cancer represents the fourth most common and the fourth leading cause of death from cancer in women. The public health impact of cervical cancer is even more pronounced in less-developed countries, with higher incidence (18.8 vs. 11.3 per 100,000) and higher mortality (12.4 vs. 5.2 per 100,000) compared to more-developed countries [1].
The spread of cervical cancer usually follows a stepwise pattern from the primary site at the cervix to adjacent structures, including the vagina, paracervical soft tissue, urinary bladder, and rectum. The tumor cells can also metastasize to regional lymph nodes and distant organs. Pelvic lymph node metastasis (PLNM) is associated with paracervical tissue spread, a higher risk of cancer recurrence, and poorer survival outcomes [2, 3, 4]. In the revised International Federation of Gynecology and Obstetrics (FIGO) staging system for cervical cancer in 2018, the assessment of retroperitoneal lymph node status either by imaging or pathological evaluation is incorporated, given the wider availability of advanced imaging technology (such as magnetic resonance imaging [MRI] and positron emission tomography [PET] scan) and minimally invasive surgery. This applies to all disease stages. Of note, the staging guideline recommends that pathological findings of surgical specimens supersede imaging and clinical findings [5]. However, these modern applications are frequently not feasible in low-resourced regions where cervical cancer is most prevalent.
For women with early-stage cervical cancer, specifically those having FIGO stage IA1 with lymph-vascular space invasion (LVSI), IA2, IB1/2, and IIA1, radical hysterectomy (RH) with pelvic lymphadenectomy is generally an effective primary treatment with overall survival of 70%–90% [6, 7, 8]. The lymphadenectomy provided an opportunity to pathologically assess and remove the diseased lymph nodes. However, the reported prevalence of lymph node metastasis in early-stage cervical cancer stands between 15%–25% [9, 10]. This means that the majority of patients might not have cancer in the removed lymph nodes and may face unnecessary complications from lymphadenectomy, which include significant intraoperative complications like vascular injury and long-term morbidities such as lower limb lymphedema (LLL) and pelvic lymphocele [11, 12, 13, 14]. Patients with evidence of PLNM are classified as high-risk stage IIIC1, requiring concurrent chemotherapy and pelvic radiotherapy, which further increases the risk of LLL [15]. Sentinel lymph node (SLN) biopsy has been proposed and accepted as an alternative procedure for traditional pelvic lymphadenectomy in this setting in an attempt to minimize lymphadenectomy-related morbidities, particularly LLL [7, 8]. Nevertheless, the instruments and expertise required to complete the surgical procedure, as well as the ultrastaging, limit the application of the SLN biopsy in the low-resourced setting.
In this scenario, a simple and non-invasive tool that reliably predicts PLNM would be valuable. Various prediction models have been developed for cervical cancer over the past decade for different outcomes of interest, including overall survival, progression-free survival, recurrence or distant metastasis, treatment response, and toxicity or quality of life [16]. There have been few prediction models developed for PLNM in early-stage cervical cancer [16]. Kim et al. [9] developed a preoperative nomogram based on 304 early cervical cancer patients who received hysterectomy and pelvic/para-aortic lymphadenectomy using multivariable logistic regression with age, tumor size assessed by MRI, and lymph node metastasis assessed by PET/computerized tomography (CT) as input parameters. In a validation cohort of 189 patients, with 38% identified as low risk, the discrimination accuracy was 0.825, and the negative predictive value (NPV) was 95.8% [9]. Wang et al. [17] developed an MRI-based radiomics nomogram constructed based on the radiomics signature and clinicopathologic risk factors of 67 early-stage cervical cancer patients by employing multivariable logistic regression to predict PLNM preoperatively. In a validation cohort of 29 patients, the C-index was 0.922 [17]. Of note, these previously proposed models need further external validation in other larger populations. Also, both models used the information derived from MRI and/or PET/CT. The availability of this information is clearly limited in the low-resource settings.
We aimed to develop a novel machine learning-based preoperative prediction model for PLNM in early-stage cervical cancer by combining the preoperative information generally available in most settings, including clinicopathological findings and preoperative CT of the whole abdomen and pelvis. This approach of prediction model development has been used increasingly in oncology with proven predictive performance [16, 18, 19]. The use of machine learning algorithms provides extended capabilities beyond conventional regression methods for examining the association between risk factors and PLNM, and for classifying patients with low risk of PLNM who may not need pelvic lymphadenectomy.
MATERIALS AND METHODS
1. Study population
Patients initially diagnosed with FIGO stage IA2-IIA1 squamous cell carcinoma, adenocarcinoma, and adenosquamous carcinoma of the cervix who had primary radical surgery with bilateral pelvic lymphadenectomy at our institution from January 1, 2003 to December 31, 2020, were eligible. Patients who received prior treatment with chemotherapy and/or radiotherapy, patients with neuroendocrine carcinoma, and those who did not have preoperative whole abdominal and pelvic CT results were excluded. In addition, patients with missing data on predicting factors and outcome of interest were excluded. This study was conducted under the approval of the Faculty of Medicine Research Ethics Committee (approval number 317/2021, study code OBG-2564-08305).
2. Predictive and outcome variables
The outcome variable of interest is PLNM, defined as histopathological evidence of metastatic cancer to the pelvic lymph nodes, including external iliac, internal iliac, obturator, and common iliac lymph nodes, derived at the time of primary RH with bilateral pelvic lymphadenectomy, regardless of the number of involved lymph nodes. We examined the association between PLNM and preoperatively recognized clinical-pathological variables (age, menopausal status, parity, underlying disease, FIGO stage, tumor size at initial diagnosis, tumor appearance, prior conization, histology, and vaginal metastasis) as well as preoperative CT findings (maximum tumor size, parametrial metastasis, pelvic lymph node enlargement, and vaginal involvement).
3. Statistical analysis
For predicting factors selection (feature selection), univariable analysis was performed using Stata® program version 15 (StataCorp LP, College Station, TX, USA). The development and validation of machine learning models were performed using Python programming language (version 3.9.13; Python Software Foundation, Wilmington, DE, USA). For continuous variables with normal distribution, the results were expressed as mean and standard deviation. The categorical variables were presented as numerical values and proportions. Hypothesis testing of the association between PLNM and the predictive variables was performed using χ2 tests, Student’s t-test, or Mann-Whitney U tests. The p-value of <0.05 was considered statistically significant.
4. Machine learning-based prediction model development
Seven supervised machine learning algorithms, including logistic regression (LR), random forest (RF), support vector machine (SVM), adaptive boosting (AdaBoost, ADA), gradient boosting (GB), extreme gradient boosting (XGB), and category boosting (CB), were used to evaluate the risk of PLNM. The above-mentioned predictive variables of interest were considered in developing machine-learning models for the preoperative prediction of PLNM. The predictive variables associated with PLNM in the univariable analysis with a p-value of <0.05 were utilized for machine-learning model development (Table 1).
Table 1
Characteristics between participants with and without PLNM (n=832)
For data preprocessing, we encoded categorical variables using one-hot encoding and standardized all numerical variables. Moreover, the synthetic minority oversampling technique (SMOTE) was employed in the model developing (training) cohort to address the imbalance between PLNM and non-PLNM groups [20]. Additionally, grid search was used to find the optimal hyperparameters of the seven classification classifiers, with the hyperparameter set that maximized the area under the receiver operating characteristic [ROC] curve (AUC) being chosen and fitted to the whole training dataset.
Repeated nested cross-validation was used to improve the robustness and obtain an unbiased estimate of the generalization performance of these machine-learning models. This method consisted of 2 cross-validation cycles: outer loop and inner loop. The outer loop stratified 10-fold cross-validation where the dataset was to estimate the average performance of the machine learning models. At the same time, the inner loop stratified 10-fold cross-validation to tune the hyperparameters and training models. We repeated these 2 cross-validation cycles ten times to reduce the sampling bias (Fig. S1).
The predictive performance of the developed models in predicting PLNM was evaluated from the accuracy, AUC, sensitivity, specificity, positive predictive value (PPV), and NPV.
During the prediction model development and validation, the decision threshold, which was the probability threshold for predicting the presence of PLNM for an individual case, was initially set at 0.5 by default to maximize the classification concordance (accuracy), which yielded the ‘balanced model.’ Subsequently, to obtain the ‘highly sensitive’ model, we adjusted the decision threshold to enhance the sensitivity of the prediction model and explored the possibility of using it as a screening tool. This simple adjustment improves the sensitivity of the model (with the expected adverse effect on specificity) without a significant effect on the concordance in an interval near the maximum concordance [21]. The optimal decision threshold was determined by the threshold that optimized the sensitivity to 0.95.
All machine learning models were conducted on a Mac mini with an 8-core CPU Apple M1 chip using 16 GB of RAM.
5. Model explanation
This study employed SHapley Additive exPlanations (SHAP), a model-agnostic framework based on game theory, to develop explainer models for each machine learning model [22]. These models are used to construct predictor importance rankings and prediction contributions for each subject, enabling the identification of the variables that contributed most significantly to the PLNM prediction.
RESULTS
1. Patients’ characteristics
Of 1,370 eligible patients, 538 did not have a preoperative CT done and were excluded. PLNM was found in 199 (23.9%) of 832 patients included in this study. In univariable analysis, younger age, larger tumor size, higher stage, no prior conization, tumor appearance, adenosquamous histology, and vaginal metastasis, as well as the CT findings of larger tumor size, parametrial metastasis, pelvic lymph node enlargement, and vaginal metastasis, were significantly associated with PLNM (Table 1).
2. Machine learning-based prediction models performance
Table 2 shows the overall performance of the seven machine-learning algorithms with a decision threshold of 0.5 in predicting PLNM. The parameters representing predictive performance, including accuracy (89.1%–90.6%), AUC (86.9%–91.0%), sensitivity (77.4%–82.4%), specificity (92.1%–94.3%), PPV (77.0%–81.7%), and NPV (93.0%–94.4%) appeared satisfactory and comparable among all the algorithms (Fig. 1A). Of note, the specificity and PPV were relatively high. SVM was the most sensitive algorithm with highest NPV, and GB was the most specific model with the highest PPV.
Table 2
Performance of the seven machine-learning algorithms with the decision threshold of 0.5 in predicting pelvic lymph node metastasis
Fig. 1
ADA, adaptive boosting; CB, category boosting; GB, gradient boosting; LR, logistic regression; PLND, pelvic lymph node dissection; PPV, positive predictive value; RF, random forest; RH, radical hysterectomy; ROC, receiver operating characteristic; SVM, support vector machine; XGB, extreme gradient boosting.
Predictive performance of the developed models.
(A) ROC curve of the seven machine-learning algorithms. (B) Precision-recall curve of the adaptive boosting algorithm. The performance of the ADA algorithm was assessed using precision-recall-threshold analysis. As the decision threshold increases, the sensitivity (recall) decreases, whereas the PPV (precision) increases. The optimal decision threshold that provides maximum concordance (accuracy) at 95% sensitivity was 0.44 in this study cohort. The proportion of patients with positive prediction result, who would therefore require surgical procedure for the evaluation of the pelvic lymph node status is represented by the RH + PLND rate line on this diagram.
Table 3 demonstrates the performance of the seven machine-learning algorithms in predicting PLNM after adjusting the decision threshold to enhance the sensitivity of the prediction models to 95%. As expected, while the sensitivity of each algorithm was enhanced, the NPV proportionally increased, and the PPV suffered. Among the 7 machine learning models evaluated, the ADA demonstrated the highest specificity and PPV, coupled with a high NPV. This finding was associated with a significant reduction in the proportion of the patients with positive prediction results requiring surgical procedures for the evaluation of the pelvic lymph node status, specifically SLN procedure or lymphadenectomy, to only 60.2%.
Table 3
Performance of the seven machine-learning algorithms in predicting pelvic lymph node metastasis after adjusting the decision threshold to enhance the sensitivity of the prediction models to 95%
The performance of the ADA algorithm was assessed using precision-recall-threshold analysis, as presented in Fig. 1B. The results indicate that as the decision threshold increases, the sensitivity (recall) decreases, whereas the PPV (precision) increases. The optimal decision threshold that provides maximum concordance (accuracy) at 95% sensitivity was 0.44 in this study cohort. The proportion of patients with positive prediction results, who would therefore require surgical procedures for the evaluation of the pelvic lymph node status is represented by the RH + pelvic lymph node dissection (PLND) rate line on the diagram. Fig. 2 displays the predictor importance rankings for the ADA algorithm in predicting PLNM. These predictors are presented in descending order of importance, with the top three being pelvic lymph node enlargement detected through CT scan, parametrial invasion detected through CT scan, and histology.
Fig. 2
Predictor importance of the adaptive boosting algorithm.
CT, computerized tomography; LN, lymph node; PMI, parametrial invasion; SHAP, SHapley Additive exPlanations.
DISCUSSION
Traditional management of early-stage cervical cancer involves systematic pelvic lymphadenectomy during a RH procedure. In this study, PLNM was documented in approximately 24% of the included patients. This prevalence is consistent with those reported for this condition in the literature [10]. Therefore, 76% of the patients underwent full pelvic lymphadenectomy unnecessarily and risk having surgery-related morbidities without a clear benefit. An accurate system for predicting PLNM could be helpful in decision-making. Ideally, this system should be widely accessible in most regions and settings.
The SLN biopsy procedure has been proposed and adopted by many institutions as a less invasive alternative to the standard full lymphadenectomy [8]. The procedure is performed by injecting a dye or radiocolloid technetium-99 directly into the cervix and observing the dye staining or radioactivity of the SLNs that represent each side of the retroperitoneal pelvic and/or para-aortic lymph nodes. The detected SLNs are then removed and sent for pathological evaluation by conducting ultrastaging and immunohistochemical staining, which provides a higher chance for the detection of micrometastasis [23, 24, 25]. The key indicators for the success of the SLN biopsy procedure are the side-specific SLN detection rate and the sensitivity in detecting SLN metastasis. These parameters varied among studies and were affected by tumor size, SLN mapping approach (blue dye/indocyanine green alone, radiocolloid alone, or combined), neoadjuvant chemotherapy, and pathological assessment technique. Recent meta-analyses of pooled data from SLN in cervical cancer studies demonstrated a pooled detection rate of 85% to 89% and a pooled sensitivity of 88% to 90% [26, 27]. Side-specific standard lymphadenectomy should be performed in case of failed SLN detection. Also, any grossly enlarged or suspicious lymph nodes should be removed [28]. Preoperative imaging techniques, including CT, MRI, and PET, represent non-invasive approaches for the evaluation of retroperitoneal lymph node metastasis in cervical cancer. In this regard, PET (sensitivity 75%, specificity 98%) appears superior to MRI (sensitivity 56%, specificity 93%) and CT (sensitivity 58%, specificity 92%) in the prediction of lymph node status in women with cervical cancer [29]. The availability of the most recent advanced imaging technology is clearly limited in under-resourced regions.
In this study, we have developed prediction models for PLNM in early-stage cervical cancer using machine-learning algorithms. Given the high prevalence of cervical cancer in low-resourced regions, we chose the preoperative clinicopathological and CT findings readily available in most centers that provide care for women with gynecological cancers, regardless of the settings, as the inputs for the model. As the association between PLNM and its predictive variables is commonly non-linear, conventional statistical modeling methods may have significant limitations, and supervised machine-learning models could fit the data better and more suitably represent the association. This has been recently demonstrated in similar settings [17, 30, 31]. The predictive performance of the developed models was satisfactory and comparable among the seven algorithms examined in this study. By setting the decision threshold at the conventional level of 0.5, the ‘balanced’ model with a sensitivity of 77.4%–82.4% and specificity of 92.1%–94.3% was derived. After adjusting the decision threshold to enhance the model’s sensitivity to at least 95%, the ‘highly sensitive’ model was obtained. This simple modification of the standard algorithm by optimizing the decision threshold for allocating class (PLNM or no PLNM) provides a tradeoff between the number of true positive and true negative predictions with limited effects on the concordance (accuracy) [21]. These two models have potential applications in two distinct clinical scenarios. The first clinical scenario that requires attention is whether patients at high risk of PLNM should undergo primary surgery or receive primary chemoradiation instead. The second dilemma pertains to patients with a very low risk of PLNM, and whether full pelvic lymphadenectomy can be avoided in their case.
For the first clinical scenario, patients with FIGO stage IA2-IIA1 squamous/adeno/adenosquamous cervical carcinoma with preoperative abdominal/pelvic CT results are evaluated with the ‘balanced’ model with a decision threshold of 0.5. Based on our data, patients predicted by the model as “having PLNM” would have a 77.0%–81.7% chance of truly having PLNM (Table 2, PPV). Therefore, it would be reasonable to seriously consider primary chemoradiation in this situation instead of proceeding with primary radical surgery to minimize the risk of morbidity associated with combined surgery and concurrent chemoradiation. For the second scenario involving early-stage cervical cancer patients undergoing primary radical surgery, the ‘highly sensitive’ model with enhanced model sensitivity can be applied. This model is characterized by a sensitivity of at least 95% with a high NPV of 95.6%–97.5% (for all algorithms except for SVM). Patients predicted by this model as “no PLNM” would have a 2.5%–4.4% chance of having PLNM (false-negative prediction). This false-negative rate is not inferior to that of the SLN biopsy procedure. Therefore, omitting pelvic node procedures, including SLN biopsy or full lymphadenectomy, might be reasonable for these patients. Patients with a ‘highly sensitive’ model prediction result of “having PLNM” should undergo further pelvic lymph node assessment either by SLN procedure or full lymphadenectomy due to the low PPV of this ‘highly sensitive’ model. We estimated that 60.2%–85.2% of all patients evaluated by this ‘highly sensitive’ model would fall into this category, so up to 40% of the initial candidates for lymph node staging could have the procedure omitted (Table 3).
As part of our research, we investigated the performance of various machine learning algorithms, which included LR, RF, SVM, ADA, GB, XGB, and CB. Our goal was to determine which algorithm is most effective in addressing a specific problem. Upon analyzing the results, we found that the differences in performance among these algorithms were not as significant as we had anticipated. This similarity in performance can be attributed to the shared traits of many of these models. The majority of the algorithms we used were tree-based models that employed boosting techniques. In such situations, when dealing with relatively small datasets, the algorithms may demonstrate comparable performance.
The strengths of this study included the well-maintained data from a single institution with the use of a prudent risk factor selection method, standard machine-learning algorithms, and effective cross-validation techniques. However, the sample size was relatively small, leading to the concern about model overfitting. In addition, our study cohort consisted of patients who had undergone abdominal or pelvic CT scans, and a considerable number of early-stage cervical cancer patients without CT scan results were excluded. This exclusion might have led to a selection bias in our models, as we only analyzed those patients whom clinicians are more likely to investigate using such diagnostic methods. Furthermore, patients with missing data on predicting factors and the outcome of interest were excluded. This might affect the representativeness of our cohort data. Also, given the study’s retrospective nature, some variables’ inaccuracies may exist. The CT findings were extracted from the official radiologists’ reports stored in our hospital database, and there was no central review of the CT findings by designated radiologists. There may be a discrepancy among the radiologists, especially on the size of the tumor and the lymph node. The model proposed in this study should be considered experimental. Further external validation of the model in different populations is required.
In conclusion, we developed a novel prediction model for PLNM in early-stage cervical cancer with promising prediction performance in our setting. However, further external validation in other populations with possible model modification and update is needed.
SUPPLEMENTARY MATERIAL
Repeated nested cross-validation process.Fig. S1
Presentation:Parts of this study were presented at the Korean Society of Gynecologic Oncology (KSGO) 2023 meeting at Gyeongju Hwabaek International Convention Center (HICO), Gyeongju, South Korea on April 28, 2023
Conflict of Interest:No potential conflict of interest relevant to this article was reported.
Author Contributions:
Conceptualization: M.K., B.C., C.K.
Data curation: B.C., M.T.
Formal analysis: B.C., M.T.
Investigation: M.K., B.C., M.T., C.K.
Methodology: B.C., C.K.
Project administration: C.K.
Resources: C.K., B.C.
Software: B.C. M.T.
Supervision: C.K.
Validation: M.K., C.K.
Visualization: B.C.
Writing - original draft: M.K., B.C., C.K.
Writing - review & editing: M.K., B.C., M.T., C.K.
References
-
Guimarães YM, Godoy LR, Longatto-Filho A, Reis RD. Management of early-stage cervical cancer: a literature review. Cancers (Basel) 2022;14:575.
-
-
Charoenkwan K, Kietpeerakool C. Retroperitoneal drainage versus no drainage after pelvic lymphadenectomy for the prevention of lymphocyst formation in women with gynaecological malignancies. Cochrane Database Syst Rev 2017;6:CD007387
-
-
Peters WA 3rd, Liu PY, Barrett RJ 2nd, Stock RJ, Monk BJ, Berek JS, et al. Concurrent chemotherapy and pelvic radiation therapy compared with pelvic radiation therapy alone as adjuvant therapy after radical surgery in high-risk early-stage cancer of the cervix. J Clin Oncol 2000;18:1606–1613.
-
-
Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic minority over-sampling technique. J Artif Intell Res 2002;16:321–357.
-
-
Lundberg SM, Lee SI. A unified approach to interpreting model predictions. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, et al., editors. Advances in neural information processing systems 30. Red Hook (NY): Curran Associates, Inc.; 2017. pp. 4765-4774.
-
MeSH Terms
Figures
1 / 2
Tables
1 / 3
