Skip to main content
SpringerLink
Log in

A sentiment analysis model based on dynamic pre-training and stacked involutions

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Sentiment analysis is one of the core tasks in natural language processing, and its main goal is to identify and classify the sentiment tendencies contained in texts. Traditional sentiment analysis methods and shallow models often fail to capture the rich semantic information and contextual relationships in text, while increasing the network depth is prone to problems such as network degradation, which has some limitations in terms of accuracy and performance. Based on this foundation, a sentiment analysis model called BERT-Invos (Bidirectional Encoder Representations from Transformers (BERT)-based stacked involutions) is introduced. The model utilizes the dynamic pre-training language model BERT to encode text, enabling rich semantic features and contextual understanding. In addition, the model employs stacked involutions with varying dimensions to extract features and perceive local information, gradually learning different scales and hierarchical representations of the input text. Furthermore, the proposed method incorporates nonlinear activation functions such as ReLU6 and H-Swish to enhance the model’s expression capability and performance, ultimately delivering classification results. In the experiment, a financial news sentiment dataset was utilized for model validation and comparison against other models. The results revealed that the model achieved an accuracy of 96.34% in sentiment analysis tasks, with precision, recall, and F1 score reaching 96.37%, 96.34%, and 96.34%, respectively. Additionally, the loss value could be minimized to 0.07 with stable convergence, thereby enhancing the accuracy of sentiment classification and reducing loss rates. This improvement facilitates better capturing of local patterns in text and addresses the issue of degradation in deep neural networks. Regarding the deep architecture of the proposed model, future work will focus on further exploring optimization techniques for model compression and deployment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

The emotional analysis datasets that are used in this paper are available on the following public link: https://github.com/wwwxmu/Dataset-of-financial-news-sentiment-classification.

References

  1. Kalashami MP, Pedram MM, Sadr H (2022) EEG feature extraction and data augmentation in emotion recognition. Comput Intell Neurosci 2022(1):7028517–7028517

    Google Scholar 

  2. McCallum A, Freitag D, Pereira FC et al (2000) Maximum entropy Markov models for information extraction and segmentation. Icml 17(2000):591–598

    Google Scholar 

  3. Wallach HM (2004) Conditional random fields: an introduction. Tech Rep (CIS) 24(2):22–31

    Google Scholar 

  4. Hidayat THJ, Ruldeviyani Y, Aditama AR et al (2022) Sentiment analysis of twitter data related to Rinca Island development using Doc2vec and SVM and logistic regression as classifier. Procedia Comput Sci 197(1):660–667

    Article  Google Scholar 

  5. Rahman H, Tariq J et al (2022) Multi-tier sentiment analysis of social media text using supervised machine learning. Comput Mater Continua 74(3):5527–5543

    Article  Google Scholar 

  6. Palmer M, Roeder J, Muntermann J (2022) Induction of a sentiment dictionary for financial analyst communication: a data-driven approach balancing machine learning and human intuition. J Bus Anal 5(1):8–28

    Article  Google Scholar 

  7. Ojeda-Hernández M, López-Rodríguez D, Mora Á (2023) Lexicon-based sentiment analysis in texts using formal concept analysis. Int J Approx Reason 155(1):104–112

    Article  MathSciNet  Google Scholar 

  8. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  9. Ce P, Tie B (2020) An analysis method for interpretability of CNN text classification model. Future Internet 12(12):228–242

    Article  Google Scholar 

  10. Liu LW, Yu X (2019) Circulating neural network (rnn) and its application. IEEE J Sci Technol World 32(6):54–55

    Google Scholar 

  11. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30(1):1–11

    Google Scholar 

  12. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 32(1): 580–584

  13. Mohades Deilami F, Sadr H, Tarkhan M (2022) Contextualized Multidimensional Personality Recognition using Combination of Deep Neural Network and Ensemble Learning. Neural Process Lett 54(5):3811–3828

    Article  Google Scholar 

  14. Deng J, Cheng L, Wang Z (2021) Attention-based BiLSTM fused CNN with gating mechanism model for Chinese long text classification. Comput Speech Lang 68(1):101182–101194

    Article  Google Scholar 

  15. Sadr H, Nazari Soleimandarabi M (2022) Acnn-tl: attention-based convolutional neural network coupling with transfer learning and contextualized word representation for enhancing the performance of sentiment classification. J Supercomput 78(7):10149–10175

    Article  Google Scholar 

  16. Chen Y, Dai X, Liu M, Chen D, Yuan L, Liu Z (2020) Dynamic convolution: attention over convolution kernels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 38(1): 11030–11039

  17. Li D, Hu J, Wang C, Li X, She Q, Zhu L, Zhang T, Chen Q (2021) Involution: inverting the inherence of convolution for visual recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 39(1): 12321–12330

  18. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 34(1): 770–778

  19. Devlin J, Chang MW, Lee K, Toutanova K (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Stroudsburg, PA, 22(1): 4171–4186

  20. Yin D, Meng T, Chang K-W (2020) SentiBERT: a transferable transformer-based architecture for compositional sentiment semantics. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 58(1): 3695–3706

  21. Wan C-X, Li B (2021) Financial causal sentence recognition based on BERT-CNN text classification. J Supercomput 78(5):1–25

    Google Scholar 

  22. Zhang M, Wang J (2021) Automatic extraction of flooding control knowledge from rich literature texts using deep learning. Appl Sci 78(5):1–25

    MathSciNet  Google Scholar 

  23. Yuan S, Wang Q (2022) Imbalanced traffic accident text classification based on Bert-RCNN. J Phys Conf Ser 2170(5):1–9

    Google Scholar 

  24. Xu G, Zhang Z, Zhang T, Yu S, Meng Y, Chen S (2022) Aspect-level sentiment classification based on attention-BiLSTM model and transfer learning. Knowl Based Syst 245(5):1–9

    Google Scholar 

  25. Howard A, Sandler M, Chu G et al (2019) Searching for mobilenetv3. CoRR 67(1):1–10

    Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China (No.62103350) and Shandong Provincial Natural Science Foundation (ZR2020QF046).

Author information

Authors and Affiliations

  1. School of Computer and Control Engineering, Yantai University, Qingquan Road, Yantai, 264006, Shandong, China

    Shiyu Liu & Qicheng Liu

Authors
  1. Shiyu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qicheng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Idea conception, design of the model and computational framework, simulation of experiments and analysis of experimental results were carried out by SL. The first draft of the manuscript was written by SL. A previous version of the manuscript was commented on by QL. Comments on the manuscript were provided by all authors. Qicheng Liu was responsible for the overall direction and planning.

Corresponding author

Correspondence to Qicheng Liu.

Ethics declarations

Conflict of interest

The authors declared that they do not have any commercial or associative interest that represents a Conflict of interest in connection with the work submitted.

Ethics approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, S., Liu, Q. A sentiment analysis model based on dynamic pre-training and stacked involutions. J Supercomput (2024). https://doi.org/10.1007/s11227-024-06052-6

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11227-024-06052-6

Keywords

Search

Navigation