Skip to main content
SpringerLink
Log in

Sentence pair modeling based on semantic feature map for human interaction with IoT devices

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

The rapid development of Internet of Things (IoT) brings an urgent requirement on intelligent human–device interactions using natural language, which are critical for facilitating people to use IoT devices. The efficient interactive approaches depend on various natural language understanding technologies. Among them, sentence pair modeling (SPM) is essential, where neural networks have achieved great success in SPM area due to their powerful abilities in feature extraction and representation. However, as sentences are one-dimensional (1D) texts, the available neural networks are usually limited to 1D sequential models, which prevents the performance improvement of SPM task. To address this gap, in this paper, we propose a novel neural architecture for sentence pair modeling, which utilizes 1D sentences to construct multi-dimensional feature maps similar to images containing multiple color channels. Based on the feature maps, more kinds of neural models become applicable on SPM task, including 2D CNN. In the proposed model, first, the sentence on a specific granularity is encoded with BiLSTM to generate the representation on this granularity, which is viewed as a special channel of the sentence. The representations from different granularity are merged together to construct semantic feature map of the input sentence. Then, 2D CNN is employed to encode the feature map to capture the deeper semantic features contained in the sentence. Next, another 2D CNN is utilized to capture the interactive matching features between sentences, followed by 2D max-pooling and attention mechanism to generate the final matching representation. Finally, the matching degree of sentences are judged with a sigmoid function according to the matching representation. Extensive experiments are conducted on two real-world data sets. In comparison with benchmarks, the proposed model achieved remarkable results, and performed better or comparably with BERT-based models. Our work is beneficial to building a more powerful humanized interaction system with IoT devices.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. https://github.com/yurui12138/SFM.

  2. https://www.apple.com/siri/.

  3. https://www.microsoft.com/en-us/cortana.

  4. https://embedding.github.io/vectors/.

  5. Ei in the top row refers to the dimension of embeddings. In our experiments, the actual number of dimensions is 400. However, for the reason of simplification, we only keep the first 10 dimensions to show.

References

  1. Leem S-G, Yoo I-C, Yook D (2019) Multitask learning of deep neural network-based keyword spotting for IoT devices. IEEE Trans Consum Electron 65(2):188–194

    Article  Google Scholar 

  2. Ni P, Li Y, Li G, Chang V (2020) Natural language understanding approaches based on joint task of intent detection and slot filling for iot voice interaction. Neural Comput Appl 32(20):16149–16166

    Article  Google Scholar 

  3. de Barcelos SA, Gomes MM, da Costa CA, da Rosa RR, Barbosa JLV, Pessin G, Doncker GD, Federizzi G (2020) Intelligent personal assistants: a systematic literature review. Expert Syst Appl 147:113193

    Article  Google Scholar 

  4. Poria S, Cambria E, Winterstein G, Huang G-B (2014) Sentic patterns: dependency-based rules for concept-level sentiment analysis. Knowl Based Syst 69:45–63

    Article  Google Scholar 

  5. Xing X, Huimin L, Song J, Yang Y, Shen H, Li X (2019) Ternary adversarial networks with self-supervision for zero-shot cross-modal retrieval. IEEE Trans Cybern 77(17):21847–21860

    Google Scholar 

  6. Wang S, Hu L, Wang Y, Sheng QZ, Orgun M, Cao L (2019) Modeling multi-purpose sessions for next-item recommendations via mixture-channel purpose routing networks. In: Proceedings of the 28th international joint conference on artificial intelligence, pp 3771–3777

  7. Lan R, Sun L, Liu Z, Huimin L, Pang C, Luo X (2021) Madnet: a fast and lightweight network for single-image super resolution. IEEE Trans Cybern 51(3):1443–1453

    Article  Google Scholar 

  8. Jiang J-Y, Zhang M, Li C, Bendersky M, Golbandi N, Najork M (2019) Semantic text matching for long-form documents. In: Proceedings of the 2019 word wide web conference, pp 795–806

  9. Yang Z, Wang K, Li J, Huang Y, Zhang Y-J (2019) TS-RNN: text steganalysis based on recurrent neural networks. IEEE Signal Process Lett 26(12):1743–1747

    Article  Google Scholar 

  10. Tong Y, Liu YL, Wang J, Xin G (2019) Text steganography on RNN-generated lyrics. Math Biosci Eng MBE 16(5):5451–5463

    Article  MathSciNet  Google Scholar 

  11. Zhang X, Lu W, Zhang G, Li F, Wang S (2020) Chinese sentence semantic matching based on multi-granularity fusion model. In: Proceedings of the Pacific Asia knowledge discovery and data mining, pp 246–257

  12. Huimin L, Li Y, Chen M, Kim H, Serikawa S (2018) Brain intelligence: go beyond artificial intelligence. Mob Netw Appl 23(2):368–375

    Article  Google Scholar 

  13. Zhang Y, Wenpeng L, Weihua O, Guoqiang Zhang X, Zhang JC, Zhang W (2020) Chinese medical question answer selection via hybrid models based on CNN and GRU. Multimed Tools Appl 79(21–22):14751–14776

    Article  Google Scholar 

  14. Huimin L, Li Y, Shenglin M, Wang D, Kim H, Serikawa S (2017) Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Internet Things J 5(4):2315–2322

    Google Scholar 

  15. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  16. Mike S, Paliwal Kuldip K (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673–2681

    Article  Google Scholar 

  17. Cho K, van Merrienboer B, Gülçehre Ç, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder–decoder for statistical machine translation. In: Proceedings of the 2014 conference on empirical methods in natural language processing, pp 1724–1734

  18. Weizhi Liao Yu, Wang YY, Zhang X, Ma P (2020) Improved sequence generation model for multi-label classification via CNN and initialized fully connection. Neurocomputing 382:188–195

    Article  Google Scholar 

  19. Lao Y, Gao S (2019) A topic matching based CNN for sentence classification. In: Proceedings of the 3rd international conference on innovation in artificial intelligence, pp 45–49

  20. Zhang C, Zhang W, Zha D, Ren P, Mu N (2019) A multi-granularity neural network for answer sentence selection. In: Proceedings of the international joint conference on neural networks, pp 1–7

  21. Zhang R, Lu W, Wang S, Peng X, Yu R, Gao Y (2020) Chinese clinical named entity recognition based on stacked neural network. Concurr Comput Pract Exp e5775

  22. Lingyun X, Guo GY, Jingming SVS, Yang P (2020) A convolutional neural network-based linguistic steganalysis for synonym substitution steganography. Math Biosci Eng 17(2):1041–1058

    Article  MathSciNet  Google Scholar 

  23. Liu F, Zheng J, Zheng L, Chen C (2020) Combining attention-based bidirectional gated recurrent neural network and two-dimensional convolutional neural network for document-level sentiment classification. Neurocomputing 371:39–50

    Article  Google Scholar 

  24. Zhou P, Qi Z, Zheng S, Xu J, Bao H, Xu B (2016) Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling. In: Proceedings of the 26th international conference on computational linguistics, pp 3485–3495

  25. Wang D, Chen D, Song B, Guizani N, Xiaoyan Yu, Xiaojiang D (2018) From IoT to 5G I-IoT: the next generation IoT-based intelligent algorithms and 5G technologies. IEEE Commun Mag 56(10):114–120

    Article  Google Scholar 

  26. Wenpeng L, Zhang X, Huimin L, Li F (2020) Deep hierarchical encoding model for sentence semantic matching. J Vis Commun Image Represent 71:102794

    Article  Google Scholar 

  27. Nie Y, Chen H, Bansal M (2019) Combining fact extraction and verification with neural semantic matching networks. In: Proceedings of the thirty-third AAAI conference on artificial intelligence, pp 6859–6866

  28. Yu W, Wei W, Xing C, Can X, Li Z, Zhou M (2019) A sequential matching framework for multi-turn response selection in retrieval-based chatbots. Comput Linguist 45(1):163–197

    Article  MathSciNet  Google Scholar 

  29. Niu G, Xu H, He B, Xiao X, Wu H, Sheng G (2019) Enhancing local feature extraction with global representation for neural text classification. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, pp 496–506

  30. Yang R, Zhang J, Gao X, Ji F, Chen H (2019) Simple and effective text matching with richer alignment features. In: Proceedings of the 57th conference of the association for computational linguistics, pp 4699–4709

  31. Xiaomei Yu, Feng W, Wang H, Chu Q, Chen Q (2020) An attention mechanism and multi-granularity-based BI-LSTM model for Chinese q&a system. Soft Comput 24(8):5831–5845

    Article  Google Scholar 

  32. Hao J, Wang X, Shi S, Zhang J, Tu Z (2019) Multi-granularity self-attention for neural machine translation. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, pp 887–897

  33. Wenpeng L, Zhang Y, Wang S, Huang H, Liu Q, Luo S (2021) Concept representation by learning explicit and implicit concept couplings. IEEE Intell Syst 36(1):6–15

    Article  Google Scholar 

  34. Chen J, Chen Q, Liu X, Yang H, Lu D, Tang B (2018) The BQ corpus: a large-scale domain-specific Chinese corpus for sentence semantic equivalence identification. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 4946–4951

  35. Liu X, Chen Q, Deng C, Zeng H, Chen J, Li D, Tang B (2018) LCQMC: a large-scale Chinese question matching corpus. In: Proceedings of the 27th international conference on computational linguistics, pp 1952–1962

  36. Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  37. Nair V, Hinton Geoffrey E (2010) Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning, pp 807–814

  38. Wang Z, Hamza W, Florian R (2017) Bilateral multi-perspective matching for natural language sentences. In: Proceedings of the twenty-sixth international joint conference on artificial intelligence, pp 4144–4150

  39. Gong Y, Luo H, Zhang J (2018) Natural language inference over interaction space. In: Proceedings of the 6th international conference on learning representations, pp 1–15

  40. Zhang X, Wenpeng L, Li F, Peng X, Zhang R (2019) Deep feature fusion model for sentence semantic matching. Comput Mater Continua 61(2):601–616

    Article  Google Scholar 

  41. Huang Q, Bu J, Xie W, Yang S, Wu W, Liu L (2019) Multi-task sentence encoding model for semantic retrieval in question answering systems. In: Proceedings of the international joint conference on neural networks, pp 1–8

  42. Liu W, Zhou P, Wang Z, Zhao Z, Deng H, Ju Q (2020) Fastbert: a self-distilling BERT with adaptive inference time. In: Proceedings of the 58th annual meeting of the association for computational linguistics, pp 6035–6044

  43. Sun Y, Wang S, Li Y-K, Feng S, Tian H, Wu H, Wang H (2020) ERNIE 2.0: a continual pre-training framework for language understanding. In: Proceedings of the thirty-fourth AAAI conference on artificial intelligence, pp 8968–8975

Download references

Acknowledgements

The research work is partly supported by National Nature Science Foundation of China under Grant No.61502259, National Key R&D Program of China under Grant No.2018YFC0831700, and Key Program of Science and Technology of Shandong under Grant No.2020CXGC010901 and No.2019JZZY020124.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Qilu University of Technology (Shandong Academy of Sciences), Shandong, 250014, China

    Rui Yu, Wenpeng Lu, Xu Zhang & Jiguo Yu

  2. School of Engineering, Kyushu Institute of Technology, Kitakyushu, 8048550, Japan

    Huimin Lu

  3. Department of Computing, Macquarie University, Sydney, NSW, 2000, Australia

    Shoujin Wang

  4. oOh! Media, North Sydney, NSW, 2060, Australia

    Fangfang Li

Authors
  1. Rui Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenpeng Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huimin Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shoujin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fangfang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jiguo Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wenpeng Lu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, R., Lu, W., Lu, H. et al. Sentence pair modeling based on semantic feature map for human interaction with IoT devices. Int. J. Mach. Learn. & Cyber. 12, 3081–3099 (2021). https://doi.org/10.1007/s13042-021-01349-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-021-01349-x

Keywords

Search

Navigation