Skip to main content
SpringerLink
Log in

Training with One2MultiSeq: CopyBART for social media keyphrase generation

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Keyphrase generation, which can help people obtain key information from a long document (social media posts or scientific articles) in a short time, has made significant progress in recent years, especially for training by concatenating keyphrases with a predefined order. However, when using beam search for keyphrase generation, models tend to repeatedly generate the highest priority keyphrase type in each beam branch, which causes the model to weaken the generation performance on the underdog keyphrase type. To tackle this, we introduce the One2MultiSeq paradigm, which allows the model to train with two sets of keyphrases that have completely opposite connection orders. Moreover, given that social media content is often colloquial, informal, and multimodal (comprising not just text but also images), these properties necessitate the incorporation of a priori knowledge for models to effectively process such information. However, contemporary models lack this requisite capacity, thereby limiting their ability to proficiently handle these discrete data elements. To overcome this, we incorporate the pretrained model BART as our backbone architecture and employ a copy mechanism to further augment its keyphrase generation capabilities. Experimental results show that our method outperformed relatively advanced models, with gains of 3.51, 1.55, and 2.47 percentage points in F1@1, F1@3, and MAP@5, respectively, on the unimodal Twitter dataset; 3.23, 2.68, and 4.07 on the multimodal Tweet dataset; and increases of 4.32, 0.32, and 7.07 in F1@3, F1@5, and MAP@5, respectively, on the StackExchange dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Availability of data and materials

Our program code is publicly available and can be found in https://github.com/Mint-hfut/One2MultiSeq. The unimodal StackExchange and multimodal Tweet datasets are also publicly available, and they can be downloaded via the link below https://drive.google.com/file/d/12f2HOl6uOvsnCfiofuoB19vxWEEiGy00/view.

References

  1. Ferrara F, Pudota N, Tasso C (2011) A keyphrase-based paper recommender system. In: Agosti M, Esposito F, Meghini C, Orio N (eds) Digital libraries and archives. Springer, Berlin, pp 14–25

    Chapter  Google Scholar 

  2. Yang H, Sanner S, Wu G, Zhou JP (2021) Bayesian preference elicitation with keyphrase-item coembeddings for interactive recommendation. In: Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization. UMAP’21. Association for Computing Machinery, New York, pp 55–64. https://doi.org/10.1145/3450613.3456814

  3. Meng X, Wei F, Liu X, Zhou M, Li S, Wang H (2012) Entity-centric topic-oriented opinion summarization in twitter. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD’12. Association for Computing Machinery, New York, pp 379–387. https://doi.org/10.1145/2339530.2339592

  4. Hua X, Hu Z, Wang L (2019) Argument generation with retrieval, planning, and realization. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, pp 2661–2672. https://doi.org/10.18653/v1/P19-1255

  5. Kang D, Hovy E (2020) Plan ahead: self-supervised text planning for paragraph completion task. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, , pp 6533–6543. https://doi.org/10.18653/v1/2020.emnlp-main.529

  6. Meng R, Zhao S, Han S, He D, Brusilovsky P, Chi Y (2017) Deep keyphrase generation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, pp 582–592. https://doi.org/10.18653/v1/P17-1054

  7. Gu J, Lu Z, Li H, Li VOK (2016) Incorporating copying mechanism in sequence-to-sequence learning. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, pp 1631–1640. https://doi.org/10.18653/v1/P16-1154

  8. Yuan X, Wang T, Meng R, Thaker K, Brusilovsky P, He D, Trischler A (2020) One size does not fit all: generating and evaluating variable number of keyphrases. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp 7961–7975. https://doi.org/10.18653/v1/2020.acl-main.710

  9. Meng R, Yuan X, Wang T, Zhao S, Trischler A, He D (2021) An empirical study on neural keyphrase generation. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pp 4985–5007. https://doi.org/10.18653/v1/2021.naacl-main.396

  10. Ye J, Gui T, Luo Y, Xu Y, Zhang Q (2021) One2Set: Generating diverse keyphrases as a set. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, pp 4598–4608. https://doi.org/10.18653/v1/2021.acl-long.354

  11. Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, Stoyanov V, Zettlemoyer L (2020) BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp 7871–7880. https://doi.org/10.18653/v1/2020.acl-main.703

  12. Medelyan O, Frank E, Witten IH (2009) Human-competitive tagging using automatic keyphrase extraction. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Singapore, pp 1318–1327. https://aclanthology.org/D09-1137

  13. Wang M, Zhao B, Huang Y (2016) PTR: Phrase-based topical ranking for automatic keyphrase extraction in scientific publications. In: Hirose A, Ozawa S, Doya K, Ikeda K, Lee M, Liu D (eds) Neural Information Processing. Springer, Cham, pp 120–128

    Chapter  Google Scholar 

  14. Liu Z, Chen X, Zheng Y, Sun M (2011) Automatic keyphrase extraction by bridging vocabulary gap. In: Proceedings of the Fifteenth Conference on Computational Natural Language Learning. Association for Computational Linguistics, Portland, Oregon, USA, pp 135–144. https://aclanthology.org/W11-0316

  15. Liao S, Yang Z, Liao Q, Zheng Z (2023) Topiclprank: a keyphrase extraction method based on improved topicrank. J Supercomput 79(8):9073–9092. https://doi.org/10.1007/s11227-022-05022-0

    Article  Google Scholar 

  16. Nguyen TD, Kan M-Y (2007) Keyphrase extraction in scientific publications. In: Goh DH-L, Cao TH, Sølvberg IT, Rasmussen E (eds) Asian Digital Libraries. Looking Back 10 Years and Forging New Frontiers. Springer, Berlin, pp 317–326

  17. Mihalcea R, Tarau P (2004) TextRank: Bringing order into text. In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Barcelona, pp 404–411. https://aclanthology.org/W04-3252

  18. Merrouni ZA, Frikh B, Ouhbi B (2022) Hake: an unsupervised approach to automatic keyphrase extraction for multiple domains. Cogn Comput 14:852–874. https://doi.org/10.1007/s11192-021-04230-4

    Article  Google Scholar 

  19. Zhang C, Zhao L, Zhao M, Zhang Y (2022) Enhancing keyphrase extraction from academic articles with their reference information. Scientometrics 127:703–731. https://doi.org/10.1007/s11192-021-04230-4

    Article  Google Scholar 

  20. Zhang Q, Wang Y, Gong Y, Huang X (2016) Keyphrase extraction using deep recurrent neural networks on Twitter. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, pp 836–845. https://doi.org/10.18653/v1/D16-1080

  21. Zhang Y, Li J, Song Y, Zhang C (2018) Encoding conversation context for neural keyphrase extraction from microblog posts. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, New Orleans, pp 1676–1686. https://doi.org/10.18653/v1/N18-1151

  22. Zhang Y, Zhang C, Li J (2020) Joint modeling of characters, words, and conversation contexts for microblog keyphrase extraction. J Assoc Inf Sci Technol 71(5):553–567. https://doi.org/10.1002/asi.24279

    Article  Google Scholar 

  23. Zhang Y, Zhang C (2021) Enhancing keyphrase extraction from microblogs using human reading time. J Assoc Inf Sci Technol 72(5):611–626. https://doi.org/10.1002/asi.24430

    Article  Google Scholar 

  24. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Ghahramani Z, Welling M, Cortes C, Lawrence N, Weinberger KQ (eds) Advances in Neural Information Processing Systems, vol 27. https://proceedings.neurips.cc/paper/2014/file/a14ac55a4f27472c5d894ec1c3c743d2-Paper.pdf

  25. Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations, ICLR 2015; Conference date: 07-05-2015 Through 09-05-2015

  26. Chen J, Zhang X, Wu Y, Yan Z, Li Z (2018) Keyphrase generation with correlation constraints. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, pp 4057–4066. https://doi.org/10.18653/v1/D18-1439

  27. Zhao J, Zhang Y (2019) Incorporating linguistic constraints into keyphrase generation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, pp 5224–5233. https://doi.org/10.18653/v1/P19-1515

  28. Chen W, Gao Y, Zhang J, King I, Lyu MR (2019) Title-guided encoding for keyphrase generation. Proceedings of the AAAI Conference on Artificial Intelligence 33(01):6268–6275. https://doi.org/10.1609/aaai.v33i01.33016268

    Article  Google Scholar 

  29. Zhang Y, Xiao W (2018) Keyphrase generation based on deep seq2seq model. IEEE Access 6:46047–46057. https://doi.org/10.1109/ACCESS.2018.2865589

    Article  Google Scholar 

  30. Chan HP, Chen W, Wang L, King I (2019) Neural keyphrase generation via reinforcement learning with adaptive rewards. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, pp 2163–2174. https://doi.org/10.18653/v1/P19-1208

  31. Chen W, Chan HP, Li P, King I (2020) Exclusive hierarchical decoding for deep keyphrase generation. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp 1095–1105. https://doi.org/10.18653/v1/2020.acl-main.103

  32. Luo Y, Xu Y, Ye J, Qiu X, Zhang Q (2021) Keyphrase generation with fine-grained evaluation-guided reinforcement learning. In: Findings of the Association for Computational Linguistics: EMNLP 2021. Association for Computational Linguistics, Punta Cana, pp 497–507. https://doi.org/10.18653/v1/2021.findings-emnlp.45

  33. Ye J, Cai R, Gui T, Zhang Q (2021) Heterogeneous graph neural networks for keyphrase generation. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Punta Cana, pp 2705–2715. https://doi.org/10.18653/v1/2021.emnlp-main.213

  34. Wang Y, Li J, Chan HP, King I, Lyu MR, Shi S (2019) Topic-aware neural keyphrase generation for social media language. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, pp 2516–2526. https://doi.org/10.18653/v1/P19-1240

  35. Yu X, Chen X, Huang Z, Dou Y, Hu B (2022) Topic and reference guided keyphrase generation from social media. In: Memmi G, Yang B, Kong L, Zhang T, Qiu M (eds) Knowledge Science, Engineering and Management. Springer, Cham, pp 140–154

    Chapter  Google Scholar 

  36. Yang P, Ge Y, Yao Y, Yang Y (2022) GCN-based document representation for keyphrase generation enhanced by maximizing mutual information. Knowl Based Syst 243:108488. https://doi.org/10.1016/j.knosys.2022.108488

    Article  Google Scholar 

  37. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations . https://openreview.net/forum?id=SJU4ayYgl

  38. Wang Y, Li J, Lyu M, King I (2020) Cross-media keyphrase prediction: A unified framework with multi-modality multi-head attention and image wordings. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pp 3311–3324. https://doi.org/10.18653/v1/2020.emnlp-main.268

  39. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A.N, Kaiser L.u, Polosukhin I (2017) Attention is all you need. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in Neural Information Processing Systems, vol 30. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

  40. Devlin J, Chang M-W, Lee K, Toutanova K (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, pp 4171–4186. https://doi.org/10.18653/v1/N19-1423

  41. Radford A, Narasimhan K, Salimans T, Sutskever I. Improving language understanding by generative pre-training

  42. Ju Y, Iwaihara M (2022) Unsupervised keyphrase generation by utilizing masked words prediction and pseudo-label BART finetuning. In: Tseng Y-H, Katsurai M, Nguyen HN (eds) From Born-Physical to Born-Virtual: Augmenting Intelligence in Digital Libraries. Springer, Cham, pp 21–34

    Chapter  Google Scholar 

  43. Gulcehre C, Ahn S, Nallapati R, Zhou B, Bengio Y (2016) Pointing the unknown words. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, pp 140–149. https://doi.org/10.18653/v1/P16-1014

  44. Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, Davison J, Shleifer S, von Platen P, Ma C, Jernite Y, Plu J, Xu C, Le Scao T, Gugger S, Drame M, Lhoest Q, Rush A(2020) Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, pp 38–45. https://doi.org/10.18653/v1/2020.emnlp-demos.6

  45. Yao Y, Yang P, Zhao G, Ge Y, Yang Y (2023) Probabilistic keyphrase generation from copy and generating spaces. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2023.3290789

    Article  Google Scholar 

  46. Zeng J, Li J, Song Y, Gao C, Lyu MR, King I (2018) Topic memory networks for short text classification. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, pp 3120–3131. https://doi.org/10.18653/v1/D18-1351

  47. Dong Y, Wu S, Meng F, Zhou J, Wang X, Lin J, Su J (2023) Towards better multi-modal keyphrase generation via visual entity enhancement and multi-granularity image noise filtering. In: Proceedings of the 31st ACM International Conference on Multimedia. MM’23. Association for Computing Machinery, New York, pp 3897–3907. https://doi.org/10.1145/3581783.3612413

  48. Kim J-H, Jun J, Zhang B-T (2018) Bilinear attention networks. In: Bengio S, Wallach H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R (eds) Advances in Neural Information Processing Systems, vol 31. https://proceedings.neurips.cc/paper/2018/file/96ea64f3a1aa2fd00c72faacf0cb8ac9-Paper.pdf

  49. Zhang Q, Wang J, Huang H, Huang X, Gong Y (2017) Hashtag recommendation for multimodal microblog using co-attention network. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. IJCAI’17, pp 3420–3426

  50. Hulth A (2003) Improved automatic keyword extraction given more linguistic knowledge. In: Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pp 216–223. https://aclanthology.org/W03-1028

  51. Nguyen TD, Kan M-Y (2007) Keyphrase extraction in scientific publications. In: Goh DH-L, Cao TH, Sølvberg IT, Rasmussen E (eds) Asian Digital Libraries. Looking Back 10 Years and Forging New Frontiers. Springer, Berlin, pp 317–326

  52. Krapivin M, Autaeu A, Marchese M (2009) Large dataset for keyphrases extraction

  53. Kim SN, Medelyan O, Kan M-Y, Baldwin T (2010) SemEval-2010 task 5: automatic keyphrase extraction from scientific articles. In: Proceedings of the 5th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Uppsala, pp 21–26. https://aclanthology.org/S10-1004

  54. Wu H, Liu W, Li L, Nie D, Chen T, Zhang F, Wang D (2021) UniKeyphrase: A unified extraction and generation framework for keyphrase prediction. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, pp 825–835. https://doi.org/10.18653/v1/2021.findings-acl.73

  55. Wu H, Ma B, Liu W, Chen T, Nie D (2022) Fast and constrained absent keyphrase generation by prompt-based learning. Proceedings of the AAAI Conference on Artificial Intelligence 36(10):11495–11503. https://doi.org/10.1609/aaai.v36i10.21402

Download references

Funding

This work was supported by the National Key Research and Development Program of China (No.2019YFE0110300) and the National Natural Science Foundation of China (NSFC) (No.72071061).

Author information

Authors and Affiliations

  1. School of Management, Hefei University of Technology, Tunxi Road, Hefei, 230009, Anhui, China

    Bengong Yu, Chunyang Gao & Shuwen Zhang

  2. Key Laboratory of Process Optimization and Intelligent Decision-Making, Hefei University of Technology, Tunxi Road, Hefei, 230009, Anhui, China

    Bengong Yu

Authors
  1. Bengong Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chunyang Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuwen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Bengong Yu provided useful direction to the contribution points. Gaochunyang wrote the main manuscript text and program code. Shuwen Zhang prepared Figs. 1–10. All authors reviewed the manuscript.

Corresponding author

Correspondence to Bengong Yu.

Ethics declarations

Conflict of interest

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, B., Gao, C. & Zhang, S. Training with One2MultiSeq: CopyBART for social media keyphrase generation. J Supercomput (2024). https://doi.org/10.1007/s11227-024-06050-8

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11227-024-06050-8

Keywords

Search

Navigation