Skip to main content

Advertisement

SpringerLink
Log in

Distant Supervised Construction and Evaluation of a Novel Dataset of Emotion-Tagged Social Media Comments in Spanish

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Tagged language resources are an essential requirement for developing machine-learning text-based classifiers. However, manual tagging is extremely time consuming and the resulting datasets are rather small, containing only a few thousand samples. Basic emotion datasets are particularly difficult to classify manually because categorization is prone to subjectivity, and thus, redundant classification is required to validate the assigned tag. Even though, in recent years, the amount of emotion-tagged text datasets in Spanish has been growing, it cannot be compared with the number, size, and quality of the datasets in English. Quality is a particularly concerning issue, as not many datasets in Spanish included a validation step in the construction process. In this article, a dataset of social media comments in Spanish is compiled, selected, filtered, and presented. A sample of the dataset is reclassified by a group of psychologists and validated using the Fleiss Kappa interrater agreement measure. Error analysis is performed by using the Sentic Computing tool BabelSenticNet. Results indicate that the agreement between the human raters and the automatically acquired tag is moderate, similar to other manually tagged datasets, with the advantages that the presented dataset contains several hundreds of thousands of tagged comments and it does not require extensive manual tagging. The agreement measured between human raters is very similar to the one between human raters and the original tag. Every measure presented is in the moderate agreement zone and, as such, suitable for training classification algorithms in sentiment analysis field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Cambria E. Affective computing and sentiment analysis. IEEE Intell Syst. 2016;31(2):102–7.

    Google Scholar 

  2. Picard R. Affective Computing. MIT Press; 1997.

  3. Cambria E, Poria S, Gelbukh A, Thelwall M. Sentiment analysis is a big suitcase. IEEE Intell Syst. 2017;32(6):74–80.

    Article  Google Scholar 

  4. Chaturvedi I, Cambria E, Vilares D. Lyapunov filtering of objectivity for Spanish Sentiment Model. In: 2016 International Joint Conference on Neural Networks (IJCNN). Vancouver, British Columbia, Canada: IEEE; 2016. p. 4474–4481.

  5. Majumder N, Poria S, Peng H, Chhaya N, Cambria E, Gelbukh A, et al. Sentiment and Sarcasm Classification With Multitask Learning. IEEE Intell Syst. 2019 May-June 1;34(3):38–43.

  6. Majumder N, Poria S, Gelbukh A, Cambria E. Deep learning-based document modeling for personality detection from text. IEEE Intell Syst. 2017;32(2):74–9.

    Article  Google Scholar 

  7. Medhat W, Hassan A, Korashy H. Sentiment analysis algorithms and applications: a survey. Ain Shams Eng J. 2014;5(4):1093–113.

    Article  Google Scholar 

  8. Cambria E, Hussain A, Havasi C, Eckl C. Sentic Computing: Exploitation of Common Sense for the Development of Emotion-Sensitive Systems. In: Esposito A, Campbell N, Vogel C, Hussain A, Nijholt A, editors. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Berlin, Heidelberg: Springer Berlin Heidelberg; 2010. p. 148–156. (Lecture Notes in Computer Science; vol. 5967).

  9. Bi J-W, Liu Y, Fan Z-P, Cambria E. Modelling customer satisfaction from online reviews using ensemble neural network and effect-based Kano model. Int J Prod Res. 2019;57(22):7068–88.

    Article  Google Scholar 

  10. Chen L, Qi L. Social opinion mining for supporting buyers’ complex decision making: exploratory user study and algorithm comparison. Soc Netw Anal Min. 2011;1(4):301–20.

    Article  Google Scholar 

  11. Bae Y, Lee H. Sentiment analysis of twitter audiences: measuring the positive or negative influence of popular twitterers. J Am Soc Inf Sci Technol. 2012;63(12):2521–35.

    Article  Google Scholar 

  12. Mahata D, Friedrichs J, Hitkul, Shah RR. Phramacovigilance - exploring deep learning techniques for identifying mentions of medication intake from twitter. 2018. arXiv preprint arXiv 1805.06375

  13. Wang Z, Chong CS, Lan L, Yang Y, Beng S, Ho JC. Tong Fine-grained sentiment analysis of social media with emotion sensing. In, 2016 Future Technologies Conference (FTC) [Internet] San Francisco, California, USA: IEEE 2016;1361-1364

  14. Munezero M, Montero CS, Sutinen E, Pajunen J. Are they different? affect, feeling, emotion, sentiment, and opinion detection in text. IEEE Trans Affect Comput. 2014 Apr-June 1;5(2):101–111.

  15. Wang Z, Ho S-B, Cambria E. A review of emotion sensing: categorization models and algorithms. Multimed Tools Appl. 2020;3:1–30.

    Google Scholar 

  16. Ekman P, Friesen WV. Constants across cultures in the face and emotion. J Pers Soc Psychol. 1971;17(2):124–9.

    Article  Google Scholar 

  17. Susanto Y, Livingstone AG, Ng BC, Cambria E, Cambria E. The hourglass model revisited. IEEE Intell Syst. 2020 Sept-Oct 1;35(5):96–102.

  18. Mintz M, Bills S, Snow R, Jurafsky D. Distant supervision for relation extraction without labeled data. In: Su K-Y, Su J, Wiebe J, Haizhou L, editors. Proceedings of the 47th Annual Meeting ofthe ACL and the 4th IJCNLP of the AFNLP. Suntec, Singapore: Association for Computational Linguistics and Asian Federation of Natural Language Processing Associations; 2009. p. 1003–1011.

  19. Pool C, Nissim M. Distant supervision for emotion detection using Facebook reactions. 2016. arXiv preprint arXiv 1611.02988

  20. Kaur W, Balakrishnan V, Rana O, Sinniah A. Liking, sharing, commenting and reacting on Facebook: user behaviors’ impact on sentiment intensity. Telemat Informatics. 2019;39(June):25–36.

    Article  Google Scholar 

  21. Tian Y, Galery T, Dulcinati G, Molimpakis E, Sun C. Facebook sentiment: reactions and emojis. In: Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media. Stroudsburg, PA, USA: Association for Computational Linguistics; 2017. p. 11–16.

  22. Balakrishnan V, Govindan V, Arshad NI, Shuib L, Cachia E. Facebook user reactions and emotion: an analysis of their relationships among the online diabetes community. Malaysian J Comput Sci. 2019;Special Issue 3:87–97.

  23. Bilal M, Malik N, Bashir N, Marjani M, Hashem IAT, Gani A. Profiling social media campaigns and political influence: the case of pakistani politics. In: 2019 13th International Conference on Mathematics, Actuarial Science, Computer Science and Statistics (MACS). Karachi, Pakistan, Pakistan: IEEE; 2019. p. 1–7.

  24. Hoque MT, Islam A, Ahmed E, Mamun KA, Huda MN. Analyzing performance of different machine learning approaches with doc2vec for classifying sentiment of bengali natural language. In: 2019 International Conference on Electrical, Computer and Communication Engineering (ECCE). Cox’s Bazar, Bangladesh: IEEE; 2019. p. 1–5.

  25. Raad BT, Philipp B, Patrick H, Christoph M. ASEDS: Towards Automatic Social Emotion Detection System Using Facebook Reactions. In: 2018 IEEE 20th International Conference on High Performance Computing and Communications; IEEE 16th International Conference on Smart City; IEEE 4th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). Exeter, United Kingdom: IEEE; 2018. p. 860–866.

  26. Baj-Rogowska A. Sentiment analysis of Facebook posts: The Uber case. In: 2017 Eighth International Conference on Intelligent Computing and Information Systems (ICICIS). Cairo, Egypt: IEEE; 2017. p. 391–395.

  27. Sandoval-Almazan R, Valle-Cruz D. Facebook impact and sentiment analysis on political campaigns. In: Proceedings of the 19th Annual International Conference on Digital Government Research Governance in the Data Age - dgo ’18. New York, New York, USA: ACM Press; 2018. p. 1–7.

  28. Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971;76(5):378–82.

    Article  Google Scholar 

  29. Mercado V, Villagra A, Errecalde M. Political alignment identification: a study with documents of Argentinian journalists. J Comput Sci Technol. 2020;20(1):43–52.

    Article  Google Scholar 

  30. Lo SL, Cambria E, Chiong R, Cornforth D. Multilingual sentiment analysis: from formal to informal and scarce resource languages. Artif Intell Rev. 2017;48(4):499–527.

    Article  Google Scholar 

  31. Cambria E, Li Y, Xing FZ, Poria S, Kwok K. SenticNet 6: Ensemble application of symbolic and subsymbolic AI for sentiment analysis. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management. New York, NY, USA: ACM; 2020. p. 105–114.

  32. Vilares D, Peng H, Satapathy R, CambriaE. BabelSenticNet: A commonsense reasoning framework for multilingual sentiment analysis. In, 2018 IEEE Symposium Series on Computational Intelligence (SSCI) Bangalore, India: IEEE 2018 1292 1298

  33. Justo R, Alcaide JM, Torres MI, Walker M. Detection of sarcasm and nastiness: new resources for Spanish language. Cognit Comput. 2018;10(6):1135–51.

    Article  Google Scholar 

  34. Dashtipour K, Poria S, Hussain A, Cambria E, Hawalah AYA, Gelbukh A, et al. Multilingual sentiment analysis: state of the art and independent comparison of techniques. Cognit Comput. 2016;8(4):757–71.

    Article  Google Scholar 

  35. Roth B, Barth T, Wiegand M, Klakow D. A survey of noise reduction methods for distant supervision. In: AKBC 2013 - Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, Co-located with CIKM 2013. San Francisco, California: Association for Computing Machinery; 2013. p. 73–77.

  36. Go A, Bhayani R, Huang L. Twitter sentiment classification using distant supervision. Technical Report Stanford University, 2010. Available from: https://www-cs.stanford.edu/people/alecmgo/papers/TwitterDistantSupervision09.pdf. Accessed 15 May 2020.

  37. Bandhakavi A, Wiratunga N, Massie S, Padmanabhan D. Lexicon generation for emotion detection from text. IEEE Intell Syst. 2017;32(1):102–8.

    Article  Google Scholar 

  38. Ahmad Refaee EA. Sentiment analysis for micro-blogging platforms in arabic [dissertation on the Internet]. Edinburgh, United Kingdom: Heriot-Watt University; 2016. [cited 2020 May 15]. Available from: https://www.ros-test.hw.ac.uk/bitstream/handle/10399/3166/RefaeeE_0816_macs.pdf?sequence=1&isAllowed=y

  39. Suttles J, Ide N. Distant supervision for emotion classification with discrete binary values. In: International Conference on Intelligent Text Processing and Computational Linguistics. Berlin, Heidelberg: Springer; 2013. p. 121–136.

  40. Felbo B, Mislove A, Søgaard A, Rahwan I, Lehmann S. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In: Palmer M, Hwa R, Riedel S, editors. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA, USA: Association for Computational Linguistics; 2017. p. 1615–1625.

  41. Moctezuma D, Graff M, Miranda-Jiménez S, Tellez ES, Coronado A, Sánchez CN, et al. A Genetic programming approach to sentiment analysis for twitter. In: Villena Román J, García Cumbreras MA, Martínez Cámara E, Díaz Galiano MC, García Vega M, editors. Proceedings of TASS 2017: Workshop on Sentiment Analysis at SEPLN co-located with 33nd SEPLN Conference [Internet]; 2017 Sept 19; CEUR Workshop Proc. Volume 1896, 2017 [cited 2020 May 15]. p. 23–28. Available from: http://ceur-ws.org/Vol-1896/p1_ingeotec_tass2017.pdf

  42. Martín C, Aguilar RM, Torres JM, Díaz S. Supervisión remota en el entrenamiento de un clasificador de sentimientos en comentarios turísticos. In: XXXIX Jornadas de Automática [Internet]; 2018 Sept 7–9; Badajoz, Spain. Comité Español de Automática (CEA); 2018 [cited 2020 May 15]. p. 644–650. Available from: http://dehesa.unex.es/bitstream/handle/10662/8530/978-84-09-04460-3_644.pdf?sequence=1&isAllowed=y

  43. Sociedad Española del Procesamiento del Lenguaje Natural (SEPLN). Taller de Análisis de sentimientos en Español (TASS) [Internet]. 2020 [cited 15 May 2020] Available from: http://tass.sepln.org

  44. Cumbreras MÁG, Gonzalo J, Cámara EM, Unanue RM, Rosso P, Carrillo-de-Albornoz J, et al., editors. Proc Iber Lang Eval Forum (IberLEF 2019) co-located with 35th Conf Spanish Soc Nat Lang Process (SEPLN 2019) [Internet]. CEUR Workshop Proc. Volume 2421, 2019 [cited 2020 May 15]. Available from: http://ceur-ws.org/Vol-2421/

  45. Broß J. Aspect-oriented sentiment analysis of customer reviews using distant supervision techniques [dissertation on the Internet]. Berlin, Germany: Free Universitat Berlin; 2013. [cited 2020 May 15] Available from: https://refubium.fu-berlin.de/bitstream/handle/fub188/6693/Dissertation_Juergen_Bross.pdf;jsessionid=C2E12B8B1868AA5AC7167DAB14296BAE?sequence=1

  46. Sahni T, Chandak C, Reddy N, Singh M. Efficient twitter sentiment classification using subjective distant supervision. In: 2017 9th International Conference on Communication Systems and Networks (COMSNETS). Bangalore, India: IEEE; 2017. p. 548–553.

  47. Refaee E, Rieser V. Evaluating distant supervision for subjectivity and sentiment analysis on arabic twitter feeds. In: Proceedings of the EMNLP 2014 Workshop on Arabic Natural Langauge Processing (ANLP). Stroudsburg, PA, USA: Association for Computational Linguistics; 2014. p. 174–179.

  48. Deriu J, Lucchi A, De Luca V, Severyn A, Müller S, Cieliebak M, et al. Leveraging large amounts of weakly supervised data for multi-language sentiment classification. In: WWW '17 Companion: Proceedings of the 26th International Conference on World Wide Web Companion. Republic and Canton of Geneva, Switzerland: International World Wide Web Conferences Steering Committee; 2017. p. 1045–1052.

  49. Marchetti-Bowick M, Chambers N. Learning for microblogs with distant supervision: political forecasting with twitter. In: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. Avignon, France: Association for Computational Linguistics; 2012. p. 603–612.

  50. Carletta J. Squibs and discussions: assessing agreement on classification tasks: the kappa statistic. Comput Linguist. 1996;22(2):248–54.

    Google Scholar 

  51. Hearst MA. TextTiling: segmenting text into multi-paragraph subtopic passages. Comput Linguist. 1997;23(1):33–64.

    Google Scholar 

  52. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37–46.

    Article  Google Scholar 

  53. Gambino OJ, Calvo H. Predicting emotional reactions to news articles in social networks. Comput Speech Lang. 2019;58:280–303.

    Article  Google Scholar 

  54. Chatterjee A, Narahari KN, Joshi M, Agrawal P. SemEval-2019 Task 3: EmoContext Contextual Emotion Detection in Text. In: May J, Shutova E, Herbelot A, Zhu X, Apidianaki M, Mohammad SM, editors. Proceedings of the 13th International Workshop on Semantic Evaluation. Stroudsburg, PA, USA: Association for Computational Linguistics; 2019. p. 39–48.

  55. Kemp S. Digital 2020: 3.8 billion people use social media [Internet]. We Are Social Ltd; 2020 [updated 2020 Jan 30; cited 2020 May 15]. Available from: https://wearesocial.com/blog/2020/01/digital-2020-3-8-billion-people-use-social-media

  56. Becerra M. Medios digitales en Argentina: la película y la foto [Internet]. Letra P; 2018 [updated 2018 Sept 20; cited 2020 May 15]. Available from: https://www.letrap.com.ar/nota/2018-9-20-16-3-0-medios-digitales-en-argentina-la-pelicula-y-la-foto

  57. Facebook. Facebook API Graph [Internet]. 2020 [cited 15 May 2020] Available from: http://developers.facebook.com

  58. Bird S, Klein E, Loper E. Natural language processing with python. O'Reilly Media Inc.; 2009.

  59. Al-Rfou R. PYCLD2 - Python bindings to CLD2 [Internet]. 2020 [cited 15 May 2020]. Available from: https://pypi.org/project/pycld2/

  60. Han S. googletrans [Internet]. 2015 [cited 15 May 2020]. Available from: https://pypi.org/project/googletrans/

  61. Craker N, March E. The dark side of Facebook®: The Dark Tetrad, negative social potency, and trolling behaviours. Pers Individ Dif. 2016;102:79–84.

    Article  Google Scholar 

  62. Google. Google Forms [Internet]. 2020 [cited 15 May 2020] Available from: https://www.google.com/intl/es-419_ar/forms/about/

  63. Hsueh P, Melville P, Sindhwani V. Data quality from crowdsourcing: A Study of Annotation Selection Criteria. In: Ringger E, Haertel R, Tomanek K, editors. Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing - HLT '09. Morristown, NJ, USA: Association for Computational Linguistics; 2009. p. 27–35. Available from: https://www.aclweb.org/anthology/W09-1904.pdf

  64. Poria S, Cambria E, Gelbukh A, Bisio F, Hussain A. Sentiment data flow analysis by means of dynamic linguistic patterns. IEEE Comput Intell Mag. 2015;10(4):26–36.

    Article  Google Scholar 

  65. Burdisso SG, Errecalde M, Montes-y-Gómez M. PySS3: A Python package implementing a novel text classifier with visualization tools for Explainable AI. 2019. arXiv preprint arXiv 1912.09322

  66. Ferretti E, Errecalde M, Rosso P. Does semantic information help in the text categorization task? J Intell Syst. 2008;17(1–3):91–106.

    Google Scholar 

Download references

Acknowledgments

This publication was supported by public funds, and the authors would like to thank the following institutions and/or agencies:

• Instituto de Investigación y Transferencia en Tecnología (Centro CICPBA), Universidad Nacional del Noroeste de Buenos Aires, Argentina.

• Secretaría de Investigación, Desarrollo y Transferencia de la Universidad Nacional del Noroeste de Buenos Aires, Argentina.

• Gobierno de Aragón, a través del proyecto AffectiveLab-T60-20R, España. Instituto de Investigación en Informática LIDI (Centro CICPBA), Facultad de Informática, Universidad Nacional de La Plata, Argentina.

• Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), Argentina.

• Comisión de Investigaciones Científicas de la Provincia de Buenos Aires (CIC-PBA), Argentina.

• Ministerio de Ciencia, Innovación y Universidades (MCIU), España (contrato RTI2018-096986-B-C31).

The authors would also like to especially thank the psychologists who collaborated with this research by participating in the dataset validation process and the Colegio de Psicólogos de la Provincia de Buenos Aires-Distrito III, for liaising with the professionals.

Funding

This study was funded by:

• Consejo Nacional de Investigaciones Científicas y Técnicas, Argentina, (RESOL-2020-131-APN-DIR#CONICET)

• Comisión de Investigaciones Científicas de la Provincia de Buenos Aires, Argentina (Acta No 1491).

• Convocatoria para la Acreditación de Proyectos y Solicitud de Subsidios de Investigación Bianuales 2019, Secretaría de Investigación Desarrollo y Transferencia, Universidad Nacional del Noroeste de la Provincia de Buenos Aires, Argentina, (EXP 536-2019).

• Gobierno de Aragón, España (AffectiveLab-T60-20R).

• Ministerio de Ciencia, Innovación y Universidades, España, (contrato RTI2018-096986-B-C31).

Author information

Authors and Affiliations

  1. Instituto de Investigación y Transferencia en Tecnología (ITT), (Centro CICPBA), Universidad Nacional del Noroeste de Buenos Aires, Junín, Buenos Aires, Argentina

    Juan Pablo Tessore & Leonardo Martín Esnaola

  2. Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), Ciudad Autónoma de Buenos Aires, Argentina

    Juan Pablo Tessore

  3. Facultad de Informática, Instituto de Investigación en Informática LIDI (Centro CICPBA), Universidad Nacional de La Plata, La Plata, Buenos Aires, Argentina

    Laura Lanzarini

  4. Departamento de Informática e Ingeniería de Sistemas, Universidad de Zaragoza, Aragon, Zaragoza, España

    Sandra Baldassarri

  5. Instituto de Investigación en Ingeniería (I3A), Universidad de Zaragoza, Zaragoza, Aragon, España

    Sandra Baldassarri

Authors
  1. Juan Pablo Tessore
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leonardo Martín Esnaola
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Laura Lanzarini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sandra Baldassarri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan Pablo Tessore.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Dataset Disclaimer Given that the dataset consists of a compilation of public user comments on Facebook, the authors of this article expressly declare that the users are solely responsible for their posts and that, under no circumstances, the content reflects the opinions of the authors or express verified information.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (CSV 357 MB)

Supplementary file2 (CSV 7.57 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tessore, J.P., Esnaola, L.M., Lanzarini, L. et al. Distant Supervised Construction and Evaluation of a Novel Dataset of Emotion-Tagged Social Media Comments in Spanish. Cogn Comput 14, 407–424 (2022). https://doi.org/10.1007/s12559-020-09800-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-020-09800-x

Keywords

Search

Navigation