Transfer learning based approach for lung and colon cancer detection using local binary pattern features and explainable artificial intelligence (AI) techniques

Dataset

This study utilized a recently published histopathological image dataset called the LC25000 dataset (Larxel, 2020), which was introduced in 2020. It consists of 25,000 colored images representing five distinct types of lung and colon tissues. These variants are colon adenocarcinoma, benign colonic tissue, lung adenocarcinoma, benign lung tissue, and lung squamous cell carcinoma.

• Colon adenocarcinoma, the most prevalent form of colon cancer, accounts for over 95% of all colon cancer cases. It arises from the development of a particular form of tissue proliferation known as adenoma in the large intestine, which subsequently progresses into cancer.

• Benign colonic tissue does not metastasize to different regions of the body.

• Lung adenocarcinoma, making up nearly 40% of all lung cancer cases and frequently affecting women, typically originates in glandular cells and extends towards the lung’s alveoli.

• Benign lung tissue, do not metastasize to different regions of the body Although benign tumors are generally not life-threatening, they require surgical removal and evaluation through biopsy to confirm the absence of cancer.

• Lung squamous cell carcinoma is a form of small-cell cancer that emerges in the air passages or bronchi of the lungs. It stands as the second most prevalent form of lung cancer, constituting approximately 30% of all cases.

The authors collected 1,250 original photos of cancer tissues from pathology glass slides (250 for each tissue type). They applied image augmentation to rotate and flip the actual images under various situations to enrich the dataset, resulting in an enhanced dataset of 25,000 images (5,000 images for each class). Before applying the augmentation procedures, the source photos were cropped to 768 by 768 pixels to guarantee a square shape. All photos in the collection have been verified and comply with the Health Insurance Portability and Accountability Act (HIPAA) rules. Illustrations of images from the dataset are presented in Fig. 2.

Local binary pattern

Local binary pattern (LBP) is a texture descriptor that is commonly used in computer vision for texture analysis and categorization. LBP works by examining the associations between pixels and their neighbours. Instead of processing the full image, LBP concentrates on specific areas of the image. A distinct neighbourhood of surrounding pixels is considered for each pixel in the picture. Based on its intensity levels in the neighbouring setup, each pixel is then compared to the central pixel. If the value of a neighbouring pixel is equal to or greater than the value of the centre pixel, it is assigned a value of one; otherwise, it is assigned a value of zero. This procedure produces a binary pattern, which is then transformed into a decimal number that represents the texture information in that particular area. A histogram of these LBP values is then created, providing a simple representation of the texture information in the image.

LBP is quite beneficial in a variety of applications like texture classification, face recognition, and object identification. Its resistance to changes in lighting conditions increases its value in circumstances where illumination varies. There are also several extensions and modifications of LBP, such as uniform LBP, rotation-invariant LBP, and other sorts of multi-scale and multi-block LBP, which broadens its application across a wide range of circumstances. It is critical to recognise that, while LBP is a powerful texture descriptor, its usefulness is dependent on the individual application and used dataset.

Deep and transfer learning models

CNNs are a key component of current deep learning, especially in computer vision (Alturki et al., 2023). These neural networks are built to replicate the human visual system, making them ideal for tasks such as object identification, image categorization and segmentation. CNNs are distinguished by their hierarchical architecture, which consists of numerous layers of convolutional, fully connected, pooling and convolutional layers. The network learns to recognize and extract essential information from input pictures in the convolutional layers by using filters that glide over the image, finding patterns and structures. Pooling layers help to minimize computing complexity by reducing the spatial dimensions of the data. CNNs are characterized by their ability to autonomously acquire knowledge and represent complicated hierarchical features from raw data, making them ideal for a variety of visual applications. These networks have transformed sectors like as medical imaging, self-driving vehicles, and face recognition, and their flexibility to multiple domains continues to drive artificial intelligence innovation and research.

VGG16, short for Visual Geometry Group 16, is used in computer vision and deep learning tasks. VGG16, created by the University of Oxford’s Visual Geometry Group, is renowned for its ease of use and outstanding performance. It comprises 16 weight layers, 13 of which are convolutional and three of which are completely linked, allowing it to learn complicated characteristics from input pictures. VGG16’s deep architecture and compact 3 × 3 convolutional filters let it catch delicate features in pictures, making it particularly well-suited for image classification and object identification. Its simple structure has made it a popular candidate for transfer learning, where pre-trained VGG16 models are fine-tuned for multiple applications. VGG16’s excellent accuracy, simplicity, and availability of pre-trained models have established it as a significant tool in the deep learning arsenal, particularly for image-based tasks such as image categorization, object recognition, and feature extraction.

MobileNet is a significant improvement in deep learning, aimed at overcoming the issues of implementing deep neural networks on resource-constrained devices like mobile phones and embedded systems. Google’s MobileNet is distinguished by its efficiency and small architecture. It employs depth-wise separable convolutions, which considerably lower computational burden while keeping the capacity to capture critical characteristics. This architectural approach enables MobileNet to achieve high levels of accuracy in tasks like image categorization, object identification, and others, even on devices with limited processing power and memory. Its small weight makes it ideal for real-time applications like mobile image recognition and edge computing. MobileNet has evolved into an important tool in the field of computer vision, paving the way for the deployment of deep learning models in a broad range of edge devices, therefore contributing to improvements in domains such as mobile app development and the Internet of Things (IoT).

ResNet, short for residual network, is a ground-breaking deep learning architecture that has revolutionized the training of very deep neural networks (Wang et al., 2019). ResNet, developed by Microsoft Research, established the notion of residual connections, allowing the network to efficiently learn from and relay information across many layers. These leftover connections alleviate the vanishing gradient problem, allowing for the training of incredibly deep networks with hundreds of layers. ResNet architectures have become a cornerstone in picture classification, object recognition, and other computer vision applications, routinely producing robust results on benchmark datasets. ResNet’s skip connections are a major breakthrough, allowing for the training of networks with remarkable depth, and they have since inspired the design of many other deep-learning models. ResNet’s influence goes beyond computer vision research, affecting research in natural language processing and reinforcement learning, and it continues to be a fundamental architecture in the deep learning environment.

EfficientNetB4 is a deep learning model of the EfficientNet family that is noted for its remarkable efficiency and performance (Zulfiqar, Bajwa & Mehmood, 2023). These models were created by Google AI and are intended to maximize the balance between model size, accuracy, and computing efficiency. EfficientNetB4 is one of the bigger variations, providing greater accuracy without significantly increasing computing needs when compared to lesser ones. It accomplishes this by employing a compound scaling mechanism that simultaneously scales the model’s depth, breadth, and resolution. This method makes EfficientNetB4 well-suited for a variety of computer vision tasks, such as picture categorization and object recognition, where it regularly produces cutting-edge results. The EfficientNet architecture has become a notorious alternative for transfer learning, allowing practitioners to fine-tune pre-trained models for specific applications with reduced computational effort. It exemplifies the ongoing effort in deep learning to develop models that are both efficient and effective, making it a valuable asset for various applications in computer vision.

Google Research’s Xception (Salim et al., 2023) is a deep learning architecture and it is known for its novel approach. The use of depthwise separable convolutions, which drastically reduce the parameters and computational complexity while retaining good accuracy, is the essential breakthrough of Xception. This design is based on the Inception model’s concept of utilizing different filter sizes inside a single layer, but takes it a step further by employing depthwise separable convolutions across the network. Xception gets outstanding performance in picture classification and object detection tasks and is noted for its ability to generalize effectively even with less training data. This makes it useful for transfer learning and fine-tuning diverse computer vision tasks. Xception is a key step toward more efficient and scalable deep learning models by eliminating duplicate calculations and parameters. It impacted the construction of succeeding neural network designs and continues to be an important model in the fields of computer vision and deep learning.

Inception-ResNetV2 is a deep learning architecture that incorporates parts from the Inception and ResNet models (Mujahid et al., 2022). This hybrid architecture, developed by Google, is intended to use the characteristics of both networks in order to achieve high accuracy in picture categorization and other computer vision applications. Inception-ResNetV2 combines the Inception modules for efficient feature extraction with the ResNet residual connections to overcome the vanishing gradient problem and enable the training of very deep networks. The model is well-known for its remarkable performance on picture recognition tasks, with top scores on a variety of benchmark datasets. Inception-ResNetV2 is especially useful for applications requiring high accuracy and the capacity to interpret complicated visual input. It strikes a compromise between model size and computational performance, making it a flexible deep-learning tool. The inclusion of the ideas of Inception-ResNetV2 has affected the development of succeeding neural network designs, and it remains a noteworthy model in the field of computer vision and deep learning.

Results of the transfer learning models with full feature set

This section presents the results of transfer learning models with a full feature set of cancer image dataset. Table 1 shows the classification results of the models using all feature sets. The highest results have been achieved by the InceptionResNetV2 model with 98.96% accuracy, 94.65% precision, and 93.98% F1 score. The Xception model has also shown remarkable results with 94.18% accuracy, 94.65% precision, 93.38% F1 score and the highest recall score of 93.66%. The MobileNet and EfficientNetB4 have shown almost similar results with 89.69% and 88.67% respectively. The VGG16 has shown the lowest results with 8.65% accuracy, 82.57% precision, 81.99% recall, and 81.59% F1 score.

Results of the transfer learning models with LBP features

For the detection of colon and lung cancer from histopathological images, the transfer learning models are employed on LBP features. The results shown in Table 2 reveal that the CNN model and transfer learning models have shown significant improvement in results when trained on LBP features. VGG16 has shown improved results with LBP features as compared to the results obtained with the full feature set. ResNet, MobileNet and EfficientNetB4 have shown significant improvement of 7% in accuracy results. However, IncpetionResNetV2 outperformed other models with 99.98% accuracy, 99.99% precision, recall and F1 score. The class-wise F1 score accuracy of the proposed model is shown in Table 3.

Results of MCC using both feature set

To provide an in-depth analysis of the four metrics utilized in this research work MCC is used. The results of both feature sets are shown in Table 4. Table 4 demonstrates the superiority of the proposed model using the LBP feature in all scenarios.

Results of K-fold cross-validation

To demonstrate that the models performed optimally, K-fold cross-validation is utilised. Table 5 displays the results of a five-fold cross-validation, demonstrating that the proposed approach outperforms other models in terms of F1 score, accuracy, recall, and precision. This thorough validation technique underlines the superiority of the proposed method, highlighting its robustness and dependability in generating remarkable outcomes for colon cancer detection.

Shapley additive explanations

SHAP (SHapley Additive exPlanations) (Lundberg & Lee, 2017) image data analysis is a strong approach for analysing the contributions of individual pixels or characteristics inside pictures to the predictions generated by machine learning models. It gives insights into the decision-making process of image classification algorithms, making it easier to comprehend and trust their conclusions. Deep learning models have demonstrated exceptional performance in image categorization challenges. However, because of their intricacy, they are sometimes referred to as “black boxes.” SHAP analysis assists in demystifying these models by offering a clear explanation with interpretation. SHAP analysis applies the Shapley values from the concept of cooperative game theory to images. The contribution of each pixel (feature) in the image to the model’s output is represented by Shapley values. They calculate how much each pixel influences the forecast. Positive Shapley values suggest factors that influence the forecast upward, whereas negative values indicate features that influence the prediction downward.

SHAP analysis finds the top contributing features or pixels in an image for a specific prediction. This data assists in determining which regions or characteristics of a picture had the most effect on the model’s judgement. SHAP includes visualisations to make the explanations more understandable. Making borders or overlays on images allows users to see which parts are crucial for the model’s prediction. SHAP analysis of image-based data is useful in a variety of disciplines, including medical imaging used for disease detection and analysis. Image categorization systems that use SHAP analysis can be more robust, accurate, and ethical. It enables academics and practitioners to put their faith in and develop their models, resulting in better and more trustworthy image-based predictions.

This study employed the SHAP library to explain the predictions of a transfer learning model, especially for image data. The explanation variable contains the output of the explanation procedure offering insight into which sections of the image contributed to the model’s predictions for the top five labels for lungs and colon cancer detection. The image plots created from the analysis provide insights into the model’s behaviour and decision-making process. They visualize the impact that individual pixels have on the model’s predictions for each image.

SHAP explanations show how individual features influence a specific instance’s prediction. The sum of these contributions, along with the bias term, mirrors the model’s initial prediction, capturing the forecast before the inverse link function is applied. Figure 3 presents the SHAP image plot for colon and lung cancer images from the LC25000 image dataset which offers a visual representation of the model’s decision-making process, highlighting the influential pixels and contributing to the interpretability and transparency of the model’s predictions for cancer classification. Pixels of darker regions contribute to a decrease in the model’s output and pixels of lighter regions contribute to an increase in the model’s output. High SHAP values in localized regions play a crucial role in the model’s decision-making for colon or lung cancer classification. Visual differences in SHAP plots for colon and lung cancer images may reveal distinct patterns or features that the model relies on to differentiate between the two types of cancer.

Discussion

In the presented results, two different sets of experiments were conducted to evaluate the performance of transfer learning models on histopathological images for the detection of colon and lung cancer. The experiments used a full feature set of the cancer image dataset and LBP features as inputs. The results from both sets of experiments are discussed below:

In the first set of experiments, the transfer learning models were applied to the full feature set of the cancer image dataset. The highest accuracy of 98.96% was achieved by the InceptionResNetV2 model, which also exhibited strong precision and F1 score as shown in Fig. 4. These results illustrate the varying performance of transfer learning models when applied to the complete feature set, with InceptionResNetV2 emerging as the top-performing model. In the second set of experiments, the transfer learning models were trained on LBP features, which are different feature representations. The results showed that most models achieved substantial improvements in accuracy when compared to using the full feature set as shown in Fig. 5. In particular, InceptionResNetV2 excelled with an exceptional accuracy of 99.98% along with high precision, recall, and F1 score. This indicates the effectiveness of LBP features in enhancing the performance of the transfer learning models for cancer detection in histopathological images.

These results highlight the impact of feature selection on the performance of transfer learning models. While the full feature set provided strong results for some models, the use of LBP features led to significant improvements, especially for InceptionResNetV2, which achieved near-perfect accuracy. This demonstrates the importance of selecting the right features for specific tasks, and it also underscores the effectiveness of transfer learning models for histopathological image analysis. InceptionResNetV2 is a deep CNN that has been pre-trained on a massive dataset of images, such as ImageNet. This means that it has already learned to recognize a wide range of generic image features, such as edges, textures, and shapes. When trained on a specific task, such as detecting cancer, InceptionResNetV2 can leverage this prior knowledge to better recognize the key patterns and characteristics that are indicative of the disease. This can lead to improved performance on the task compared to a model that has not been pre-trained. LBP features are widely recognized for their ability to effectively capture texture information in images. By employing LBP features as inputs to InceptionResNetV2, the model can adeptly utilize these texture descriptors to enhance its comprehension of histopathological images.

This synergistic integration of deep learning and texture-based features is likely a contributing factor to the model’s remarkable performance. The study not only provides insights into the models’ performance but also shows the potential for improving cancer detection using image analysis techniques. Additionally, the use of SHAP for model interpretation adds transparency to the predictions, helping to understand the models’ decision-making process and the influence of individual features or pixels on the results.

The major reasons for misclassified images in the deep Inception-ResNet-v2 model with local binary patterns (LBP) features include (1) Variability in image characteristics: Images with varying lighting conditions, orientations, scales, or angles are not adequately represented by the extracted LBP features. This variability can lead to misclassifications as the model struggles to generalize across different image characteristics. (2) Limited discriminative power of LBP features: While LBP features are effective for capturing local texture patterns, they do not always provide sufficient discriminative power to differentiate between classes, especially in cases where classes exhibit subtle differences or overlap in feature space.