Skip to main content
SpringerLink
Log in

A hybrid of XGBoost and aspect-based review mining with attention neural network for user preference prediction

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

With the rapid development of the internet, users tend to refer to the rating scores or review opinions on social platforms. Most recommendation systems use collaborative filtering (CF) methods to recommend items based on users’ ratings. The rating-based CF methods do not consider users’ review opinions on different aspects of items. The accuracy of the rating predictions can be effectively improved by considering the latent semantics and various aspects of user reviews. In this paper, a novel rating prediction method is proposed according to an attention-based gated recurrent unit (GRU) deep learning model with semantic aspects. A two-phase method is proposed herein; it combines the word attention mechanism and review semantics to extract aspect features from user preferences. In the first phase, a bidirectional GRU neural network is adopted according to word attention in order to extract important words from users’ reviews. In the second phase, we split users’ reviews into words, and generate the aspect-based attention semantic vectors from these reviews based on Latent Dirichlet Allocation and the attention weights of the chosen words. The XGBoost method is then adopted to predict user preference ratings based on the aspect-based attention semantic vectors. The experimental results show that the proposed method outperforms traditional prediction methods and effectively improves the accuracy of predictions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://www.yelp.com/dataset

References

  1. Adomavicius G, Tuzhilin A (2005) Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 17:734–749

    Article  Google Scholar 

  2. Al-Smadi M, Talafha B, Al-Ayyoub M, Jararweh Y (2018) Using long short-term memory deep neural networks for aspect-based sentiment analysis of Arabic reviews. Int J Mach Learn Cybern. https://doi.org/10.1007/s13042-018-0799-4

    Article  Google Scholar 

  3. Bagheri A, Saraee M, De Jong F (2013) Care more about customers: unsupervised domain-independent aspect detection for sentiment analysis of customer reviews. Knowl Based Syst 52:201–213

    Article  Google Scholar 

  4. Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: International conference on learning representations

  5. Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet Allocation. J Mach Learn Res 3:993–1022

    MATH  Google Scholar 

  6. Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: Proceedings of the international conference on computational statistics (COMPSTAT). Physica-Verlag HD, pp 177–186

  7. Brody S, Elhadad NA (2010) Unsupervised aspect-sentiment model for online reviews. In: Human language technologies: the 2010 annual conference of the North American chapter of the association for computational linguistics, Los Angeles, California Association for Computational Linguistics, pp 804–812

  8. Chen T, Guestrin C (2016) Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, San Francisco, California, USA. ACM, pp 785–794

  9. Cho K, Van Merriënboer B, Gulcehre C et al (2014) Learning phrase representations using RNN encoder–decoder for statistical machine translation. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP) Doha, Qatar. pp 1724–1734

  10. Da’u A, Salim N, Rabiu I, Osman A (2020) Recommendation system exploiting aspect-based opinion mining with deep learning method. Inf Sci 512:1279–1292. doi:https://doi.org/10.1016/j.ins.2019.10.038

    Article  Google Scholar 

  11. Dong R, Schaal M, O’Mahony MP, Smyth B (2013) Topic extraction from online reviews for classification and recommendation. In: Proceedings of the twenty-third international joint conference on artificial intelligence, Beijing, China. pp 1310–1316

  12. Ganu G, Elhadad N, Marian A (2009) Beyond the stars: improving rating predictions using review text content. In: Twelfth international workshop on the web and databases Providence, Rhode Island, USA. pp 1–6

  13. Gong S (2010) A collaborative filtering recommendation algorithm based on user clustering and item clustering. J Softw 5:745–752

    Article  Google Scholar 

  14. Hinton G, Deng L, Yu D et al (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Proc Mag 29:82–97

    Article  Google Scholar 

  15. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9:1735–1780

    Article  Google Scholar 

  16. Hopfield JJ (1982) Neural networks and physical systems with emergent collective computational abilities. In: Proceedings of the national academy of sciences of the United States of America, vol 8. pp 2554–2558

  17. Hu M, Liu B (2004) Mining and summarizing customer reviews. In: Proceedings of the tenth ACM SIGKDD international conference on knowledge discovery and data mining, Seattle, WA, USA. ACM, pp 168–177

  18. Huang J, Rogers S, Joo E (2014) Improving restaurants by extracting subtopics from Yelp reviews. In: iConference 2014 (Social Media Expo)

  19. Jo Y, Oh AH (2011) Aspect and sentiment unification model for online review analysis. In: Proceedings of the fourth ACM international conference on web search and data mining, Hong Kong, China ACM, pp 815–824

  20. Kanungo T, Mount DM, Netanyahu NS et al (2002) An efficient k-means clustering algorithm: analysis and implementation. IEEE Trans Pattern Anal 24:881–892

    Article  Google Scholar 

  21. Kim Y (2014) Convolutional neural networks for sentence classification. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP) Doha, Qatar. pp 1746–1751

  22. Koren Y, Bell R, Volinsky C (2009) Matrix factorization techniques for recommender systems. Computer 42:30–37. https://doi.org/10.1109/MC.2009.263

    Article  Google Scholar 

  23. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: The 25th international conference on neural information processing systems, Lake Tahoe, Nevada, pp 1097–1105

  24. Ling G, Lyu MR, King I (2014) Ratings meet reviews, a combined approach to recommend. In: Proceedings of the 8th ACM conference on recommender systems, Foster City, Silicon Valley, California, USA. ACM, pp 105–112

  25. Lipton ZC, Berkowitz J, Elkan C (2015) A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv:150600019

  26. Liu B, Hu M, Cheng J (2005) Opinion observer: analyzing and comparing opinions on the web. In: Proceedings of the 14th international conference on world wide web, Chiba, Japan. ACM, pp 342–351

  27. Liu D-R, Chen K-Y, Chou Y-C, Lee J-H (2018) Online recommendations based on dynamic adjustment of recommendation lists. Knowl Based Syst 161:375–389

    Article  Google Scholar 

  28. Lops P, De Gemmis M, Semeraro G (2011) Content-based recommender systems: state of the art and trends. In: Ricci F, Rokach L, Shapira B, Kantor PB (eds) Recommender systems handbook. Springer, Boston, pp 73–105

    Chapter  Google Scholar 

  29. Lu B, Ott M, Cardie C, Tsou BK (2011) Multi-aspect sentiment analysis with topic models. In: The IEEE 11th international conference on data mining workshops (ICDMW), Vancouver, Canada. IEEE, pp 81–88

  30. McAuley J, Leskovec JH (2013) Factors and hidden topics: understanding rating dimensions with review text. In: Proceedings of the 7th ACM conference on recommender systems, Hong Kong, China. ACM, pp 165–172

  31. Pennington J, Socher R, Manning C (2014) Glove: Global vectors for word representation. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP), Doha, Qatar. Association for Computational Linguistics, pp 1532–1543

  32. Qiao X, Peng C, Liu Z, Hu Y (2019) Word-character attention model for Chinese text classification. Int J Mach Learn Cybern. https://doi.org/10.1007/s13042-019-00942-5

    Article  Google Scholar 

  33. Sarwar B, Karypis G, Konstan J, Riedl J (2001) Item-based collaborative filtering recommendation algorithms. In: Proceedings of the 10th international conference on world wide web, Hong Kong, China. ACM, pp 285–295

  34. Seo S, Huang J, Yang H, Liu Y (2017) Interpretable convolutional neural networks with dual local and global attention for review rating prediction. In: Proceedings of the eleventh ACM conference on recommender systems, Como, Italy. ACM, pp 297–305

  35. Seo S, Huang J, Yang H, Liu Y (2017) Representation learning of users and items for review rating prediction using attention-based convolutional neural network. In: 3rd International workshop on machine learning methods for recommender systems (MLRec)

  36. Serrano-Guerrero J, Chiclana F, Olivas JA et al (2020) A T1OWA fuzzy linguistic aggregation methodology for searching feature-based opinions. Knowl Based Syst 189:105131. https://doi.org/10.1016/j.knosys.2019.105131

    Article  Google Scholar 

  37. Serrano-Guerrero J, Olivas JA, Romero FP, Herrera-Viedma E (2015) Sentiment analysis: a review and comparative analysis of web services. Inf Sci 311:18–38. https://doi.org/10.1016/j.ins.2015.03.040

    Article  Google Scholar 

  38. Socher R, Perelygin A, Wu J et al (2013) Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the conference on empirical methods in natural language processing, Seattle, Washington, USA. pp 1631–1642

  39. Sun S, Luo C, Chen J (2017) A review of natural language processing techniques for opinion mining systems. Inform Fusion 36:10–25. doi:https://doi.org/10.1016/j.inffus.2016.10.004

    Article  Google Scholar 

  40. Tang D, Qin B, Liu T (2015) Document modeling with gated recurrent neural network for sentiment classification. In: Proceedings of the conference on empirical methods in natural language processing, Lisbon, Portugal. pp 1422–1432

  41. Tang D, Qin B, Liu T, Yang Y (2015) User modeling with neural network for review rating prediction. In: Proceedings of 24th international joint conference on artificial intelligence. pp 1340–1346

  42. Titov I, McDonald R (2008) Modeling online reviews with multi-grain topic models. In: Proceedings of the 17th international conference on world wide web, Beijing, China. ACM, pp 111–120

  43. Wang H, Lu Y, Zhai C (2010) Latent aspect rating analysis on review text data: a rating regression approach. In: Proceedings of the 16th ACM SIGKDD international conference on knowledge discovery and data mining, Washington, DC, USA. ACM, pp 783–792

  44. Wu Y, Ester M (2015) FLAME: a probabilistic model combining aspect based opinion mining and collaborative filtering. In: Proceedings of the eighth ACM international conference on web search and data mining, Shanghai, China. ACM, pp 199–208

  45. Yang Z, Yang D, Dyer C et al (2016) Hierarchical attention networks for document classification. In: Proceedings of the conference of the North American chapter of the association for computational linguistics: human language technologies, San Diego, California, USA. Association for Computational Linguistics, pp 1480–1489

  46. Young T, Hazarika D, Poria S, Cambria E (2018) Recent trends in deep learning based natural language processing. IEEE Comput Intell Mag 13:55–75. https://doi.org/10.1109/MCI.2018.2840738

    Article  Google Scholar 

  47. Zhang Y, Wallace B (2015) A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification. In: Proceedings of the 8th international joint conference on natural language processing, Taipei, Taiwan. pp 253–263

  48. Zheng L, Noroozi V, Yu PS (2017) Joint deep modeling of users and items using reviews for recommendation. In: Proceedings of the tenth ACM international conference on web search and data mining, Cambridge, United Kingdom. ACM, pp 425–434

  49. Zhuang L, Schouten K, Frasincar F (2020) SOBA: semi-automated ontology builder for aspect-based sentiment analysis. J Web Semant 60:100544. https://doi.org/10.1016/j.websem.2019.100544

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by the Ministry of Science and Technology of Taiwan under Grant number: MOST 105-2410-H-009-033-MY3 and MOST 106-2410-H-033-013.

Author information

Authors and Affiliations

  1. Department of Information Management, Chung Yuan Christian University, No. 200, Zhongbei Rd., Zhongli Dist., Taoyuan, 320, Taiwan

    Chin-Hui Lai

  2. Institute of Information Management, National Chiao Tung University, No. 1001, Daxue Rd., East Dist., Hsinchu, 300, Taiwan

    Duen-Ren Liu & Kun-Sin Lien

Authors
  1. Chin-Hui Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Duen-Ren Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kun-Sin Lien
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chin-Hui Lai.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lai, CH., Liu, DR. & Lien, KS. A hybrid of XGBoost and aspect-based review mining with attention neural network for user preference prediction. Int. J. Mach. Learn. & Cyber. 12, 1203–1217 (2021). https://doi.org/10.1007/s13042-020-01229-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-020-01229-w

Keywords

Search

Navigation