当前位置: X-MOL 学术Concurr. Comput. Pract. Exp. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Text classification using improved bidirectional transformer
Concurrency and Computation: Practice and Experience ( IF 2 ) Pub Date : 2021-07-18 , DOI: 10.1002/cpe.6486
Murat Tezgider 1 , Beytullah Yildiz 2 , Galip Aydin 1
Affiliation  

Text data have an important place in our daily life. A huge amount of text data is generated everyday. As a result, automation becomes necessary to handle these large text data. Recently, we are witnessing important developments with the adaptation of new approaches in text processing. Attention mechanisms and transformers are emerging as methods with significant potential for text processing. In this study, we introduced a bidirectional transformer (BiTransformer) constructed using two transformer encoder blocks that utilize bidirectional position encoding to take into account the forward and backward position information of text data. We also created models to evaluate the contribution of attention mechanisms to the classification process. Four models, including long short term memory, attention, transformer, and BiTransformer, were used to conduct experiments on a large Turkish text dataset consisting of 30 categories. The effect of using pretrained embedding on models was also investigated. Experimental results show that the classification models using transformer and attention give promising results compared with classical deep learning models. We observed that the BiTransformer we proposed showed superior performance in text classification.

中文翻译:

使用改进的双向转换器进行文本分类

文本数据在我们的日常生活中占有重要地位。每天都会产生大量的文本数据。因此,处理这些大型文本数据需要自动化。最近,我们见证了在文本处理中采用新方法的重要进展。注意力机制和转换器正在成为具有巨大文本处理潜力的方法。在这项研究中,我们介绍了一个双向转换器(BiTransformer),它使用两个转换器编码器块构建,利用双向位置编码来考虑文本数据的前向和后向位置信息。我们还创建了模型来评估注意力机制对分类过程的贡献。四种模型,包括long short term memory、attention、transformer和BiTransformer,用于对包含 30 个类别的大型土耳其文本数据集进行实验。还研究了在模型上使用预训练嵌入的效果。实验结果表明,与经典深度学习模型相比,使用 Transformer 和注意力的分类模型给出了有希望的结果。我们观察到我们提出的 BiTransformer 在文本分类方面表现出卓越的性能。
更新日期:2021-07-18
down
wechat
bug