Academic Journal of Computing & Information Science, 2024, 7(10); doi: 10.25236/AJCIS.2024.071004.
Junhong Chen1,2, Kaihui Peng3
1School of Software Engineering, South China University of Technology, Guangzhou, China
2LeiHuo studio, NetEase, Hangzhou, China
3Faculty of Business and Economics, University of Malaya, Kuala Lumpur, Malaysia
This paper proposes an enhanced text matching model with augmented recurrent attention that utilizes interactive attention mechanisms. During vector encoding, the proposed model employs attention to interact between two input texts. Following the interaction, it leverages Bi-LSTM to re-encode the sequence at a more advanced level, enabling the model to comprehensively learn global information. Additionally, an attention mechanism is incorporated to emphasize the importance of high-weights words. Furthermore, a fusion layer is added to better integrate the two text segments into a single result, which facilitates subsequent text similarity computations. The model demonstrates a high accuracy in text similarity calculations.
Text similarity calculation; Attention mechanism; Pre-trained models
Junhong Chen, Kaihui Peng. Research on Text Similarity Algorithms Based on Interactive Attention. Academic Journal of Computing & Information Science (2024), Vol. 7, Issue 10: 24-30. https://doi.org/10.25236/AJCIS.2024.071004.
[1] Koch G, Zemel R, Salakhutdinov R. Siamese neural networks for one-shot image recognition[C]//ICML deep learning workshop. 2015, 2(1): 1-30.
[2] Xia P, Zhang L, Li F. Learning similarity with cosine similarity ensemble[J]. Information sciences, 2015, 307: 39-52.
[3] Bojanowski P, Grave E, Joulin A, et al. Enriching word vectors with subword information[J]. Transactions of the association for computational linguistics, 2017, 5: 135-146.
[4] Vaswani A. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017.
[5] Cui Y, Che W, Liu T, et al. Pre-training with whole word masking for chinese bert[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3504-3514.
[6] Yu Y, Si X, Hu C, et al. A review of recurrent neural networks: LSTM cells and network architectures[J]. Neural computation, 2019, 31(7): 1235-1270.
[7] Seo M, Kembhavi A, Farhadi A, et al. Bidirectional attention flow for machine comprehension[J]. arXiv preprint arXiv:1611.01603, 2016.
[8] Galassi A, Lippi M, Torroni P. Attention in natural language processing[J]. IEEE transactions on neural networks and learning systems, 2020, 32(10): 4291-4308.
[9] Liu X, Chen Q, Deng C, et al. Lcqmc: A large-scale chinese question matching corpus[C]//Proceedings of the 27th International Conference on Computational Linguistics. 2018: 1952-1962.
[10] Chen J, Chen Q, Liu X, et al. The bq corpus: A large-scale domain-specific chinese corpus for sentence semantic equivalence identification[C]//Proceedings of the 2018 conference on empirical methods in natural language processing. 2018: 4946-4951.
[11] Chen Q, Zhu X, Ling Z, et al. Enhanced LSTM for natural language inference[J]. arXiv preprint arXiv:1609.06038, 2016.
[12] Wang Z, Hamza W, Florian R. Bilateral multi-perspective matching for natural language sentences[J]. arXiv preprint arXiv:1702.03814, 2017.
[13] Yin W, Schütze H, Xiang B, et al. Abcnn: Attention-based convolutional neural network for modeling sentence pairs[J]. Transactions of the Association for computational linguistics, 2016, 4: 259-272.
[14] Varior R R, Shuai B, Lu J, et al. A siamese long short-term memory architecture for human re-identification[C]//Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14. Springer International Publishing, 2016: 135-153.