Hostname: page-component-8448b6f56d-xtgtn Total loading time: 0 Render date: 2024-04-24T16:08:32.345Z Has data issue: false hasContentIssue false
  • English
  • Français

Is your document novel? Let attention guide you. An attention-based model for document-level novelty detection

Published online by Cambridge University Press:  24 April 2020

Tirthankar Ghosal ,
Vignesh Edithal ,
Asif Ekbal ,
Pushpak Bhattacharyya ,
Srinivasa Satya Sameer Kumar Chivukula  and
George Tsatsaronis
Tirthankar Ghosal*
Affiliation:
Indian Institute of Technology Patna, Bihta, Bihar, Patna801103, India
Vignesh Edithal
Affiliation:
Indian Institute of Technology Patna, Bihta, Bihar, Patna801103, India
Asif Ekbal
Affiliation:
Indian Institute of Technology Patna, Bihta, Bihar, Patna801103, India
Pushpak Bhattacharyya
Affiliation:
Indian Institute of Technology Patna, Bihta, Bihar, Patna801103, India
Srinivasa Satya Sameer Kumar Chivukula
Affiliation:
Elsevier, Amsterdam, Netherlands
George Tsatsaronis
Affiliation:
Elsevier, Amsterdam, Netherlands
*
*Corresponding author. E-mail: tirthankar.slg@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

Detecting, whether a document contains sufficient new information to be deemed as novel, is of immense significance in this age of data duplication. Existing techniques for document-level novelty detection mostly perform at the lexical level and are unable to address the semantic-level redundancy. These techniques usually rely on handcrafted features extracted from the documents in a rule-based or traditional feature-based machine learning setup. Here, we present an effective approach based on neural attention mechanism to detect document-level novelty without any manual feature engineering. We contend that the simple alignment of texts between the source and target document(s) could identify the state of novelty of a target document. Our deep neural architecture elicits inference knowledge from a large-scale natural language inference dataset, which proves crucial to the novelty detection task. Our approach is effective and outperforms the standard baselines and recent work on document-level novelty detection by a margin of $\sim$ 3% in terms of accuracy.

Keywords

Document-Level Novelty DetectionDecomposable AttentionNatural Language InferenceDocument Classification
Type
Article
Information
Natural Language Engineering , Volume 27 , Issue 4 , July 2021 , pp. 427 - 454
Check for updates
Copyright
© Cambridge University Press 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Allan, J., Gupta, R. and Khandelwal, V. (2001). Topic models for summarizing novelty. In ARDA Workshop on Language Modeling and Information Retrieval. Pittsburgh, Pennsylvania.Google Scholar
Allan, J., Lavrenko, V. and Jin, H. (2000). First story detection in TDT is hard. In Proceedings of the Ninth International Conference on Information and Knowledge Management. ACM, pp. 374–381.CrossRefGoogle Scholar
Allan, J., Papka, R. and Lavrenko, V. (1998). On-line new event detection and tracking. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, pp. 37–45.CrossRefGoogle Scholar
Allan, J., Wade, C. and Bolivar, A. (2003). Retrieval and novelty detection at the sentence level. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval. ACM, pp. 314–321.CrossRefGoogle Scholar
Arora, S., Liang, Y. and Ma, T. (2016). A simple but tough-to-beat baseline for sentence embeddings. In 5th International Conference on Learning Representations, 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings.Google Scholar
Bagga, A. and Baldwin, B. (1999). Cross-document event coreference: Annotations, experiments, and observations. In Proceedings of the Workshop on Coreference and its Applications. Association for Computational Linguistics, pp. 1–8.CrossRefGoogle Scholar
Bahdanau, D., Cho, K. and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473 .Google Scholar
Bentivogli, L., Clark, P., Dagan, I. and Giampiccolo, D. (2011). The seventh pascal recognizing textual entailment challenge. In TAC.Google Scholar
Bowman, S.R., Angeli, G., Potts, C. and Manning, C.D. (2015). A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17–21, 2015, pp. 632–642.CrossRefGoogle Scholar
Brants, T., Chen, F. and Farahat, A. (2003). A system for new event detection. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval. ACM, pp. 330–337.CrossRefGoogle Scholar
Bysani, P. (2010). Detecting novelty in the context of progressive summarization. In Proceedings of the NAACL HLT 2010 Student Research Workshop. Association for Computational Linguistics, pp. 13–18.Google Scholar
Carbonell, J. and Goldstein, J. (1998). The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, pp. 335–336.CrossRefGoogle Scholar
Carpenter, G.A., Rubin, M.A. and Streilein, W.W. (1997). Artmap-fd: Familiarity discrimination applied to radar target recognition. In Proceedings of International Conference on Neural Networks (ICNN’97), vol. 3. IEEE, pp. 1459–1464.CrossRefGoogle Scholar
Cer, D., Yang, Y., Kong, S., Hua, N., Limtiaco, N., John, R.S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., Strope, B. and Kurzweil, R. (2018). Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018: System Demonstrations, Brussels, Belgium, October 31-November 4, 2018, pp. 169–174.CrossRefGoogle Scholar
Chandar, P. and Carterette, B. (2013). Preference based evaluation measures for novelty and diversity. In SIGIR.CrossRefGoogle Scholar
Clarke, C.L.A., Craswell, N., Soboroff, I. and Ashkan, A. (2011). A comparative analysis of cascade measures for novelty and diversity. In WSDM.CrossRefGoogle Scholar
Clarke, C.L.A., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkan, A., Büttcher, S. and MacKinnon, I. (2008). Novelty and diversity in information retrieval evaluation. In SIGIR.CrossRefGoogle Scholar
Collins-Thompson, K., Ogilvie, P., Zhang, Y. and Callan, J. (2002). Information filtering, novelty detection, and named-page finding. In TREC.Google Scholar
Conneau, A., Kiela, D., Schwenk, H., Barrault, L. and Bordes, A. (2017). Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9–11, 2017, pp. 670–680.CrossRefGoogle Scholar
Dagan, I., Roth, D., Sammons, M. and Zanzotto, F.M. (2013). Recognizing Textual Entailment: Models and Applications. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers, pp. 1–220.CrossRefGoogle Scholar
Dasgupta, D. and Forrest, S. (1996). Novelty detection in time series data using ideas from immunology. In Proceedings of the International Conference on Intelligent Systems, pp. 82–87.Google Scholar
Dasgupta, T. and Dey, L. (2016). Automatic scoring for innovativeness of textual ideas. In Workshops at the Thirtieth AAAI Conference on Artificial Intelligence.Google Scholar
Fleiss, J.L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin 76(5), 378.CrossRefGoogle Scholar
Franz, M., Ittycheriah, A., McCarley, J.S. and Ward, T. (2001). First story detection: Combining similarity and novelty based approaches. In Topic Detection and Tracking Workshop Report, pp. 193–206.Google Scholar
Gabrilovich, E., Dumais, S. and Horvitz, E. (2004). Newsjunkie: Providing personalized newsfeeds via analysis of information novelty. In Proceedings of the 13th International Conference on World Wide Web. ACM, pp. 482–490.CrossRefGoogle Scholar
Gamon, M. (2006). Graph-based text representation for novelty detection. In Proceedings of the First Workshop on Graph Based Methods for Natural Language Processing. Association for Computational Linguistics, pp. 17–24.CrossRefGoogle Scholar
Ghosal, T., Edithal, V., Ekbal, A., Bhattacharyya, P., Tsatsaronis, G. and Chivukula, S.S.S.K. (2018a). Novelty goes deep. A deep neural solution to document level novelty detection. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20–26, 2018, pp. 2802–2813.Google Scholar
Ghosal, T., Salam, A., Tiwary, S., Ekbal, A. and Bhattacharyya, P. (2018b). TAP-DLND 1.0 : A corpus for document level novelty detection. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7–12, 2018.Google Scholar
Guh, R.-S., Zorriassatine, F., Tannock, J. and O’Brien, C. (1999). On-line control chart pattern detection and discriminationa neural network approach. Artificial Intelligence in Engineering 13(4), 413–425.CrossRefGoogle Scholar
Harman, D. (2002). Overview of the TREC 2002 novelty track. In Proceedings of The Eleventh Text REtrieval Conference, TREC 2002, Gaithersburg, Maryland, USA, November 19–22, 2002. Google Scholar
Karkali, M., Rousseau, F., Ntoulas, A. and Vazirgiannis, M. (2013). Efficient online novelty detection in news streams. In WISE (1), pp. 57–71.CrossRefGoogle Scholar
King, S., King, D., Astley, K., Tarassenko, L., Hayton, P. and Utete, S. (2002). The use of novelty detection techniques for monitoring high-integrity plant. In Proceedings of the International Conference on Control Applications, vol. 1. IEEE, pp. 221–226.CrossRefGoogle Scholar
Kwee, A.T., Tsai, F.S. and Tang, W. (2009). Sentence-level novelty detection in english and malay. In Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, pp. 40–51.CrossRefGoogle Scholar
Lai, A., Bisk, Y. and Hockenmaier, J. (2017). Natural language inference from multiple premises. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27–December 1, 2017 - Volume 1: Long Papers, pp. 100–109.Google Scholar
Le, Q. and Mikolov, T. (2014). Distributed representations of sentences and documents. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 1188–1196.Google Scholar
Lee, S. (2015). Online sentence novelty scoring for topical document streams. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 567–572.CrossRefGoogle Scholar
Li, X. and Croft, W.B. (2005). Novelty detection based on sentence level patterns. In Proceedings of the 14th ACM International Conference on Information and Knowledge Management. ACM, pp. 744–751.CrossRefGoogle Scholar
Lin, Z., Feng, M., Santos, C.N.d., Yu, M., Xiang, B., Zhou, B. and Bengio, Y. (2017). A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130 .Google Scholar
Liu, Y., Sun, C., Lin, L. and Wang, X. (2016). Learning natural language inference using bidirectional LSTM model and inner-attention. CoRR abs/1605.09090 .Google Scholar
Manikopoulos, C. and Papavassiliou, S. (2002). Network intrusion and fault detection: A statistical anomaly approach. IEEE Communications Magazine 40(10), 76–82.CrossRefGoogle Scholar
Mishra, S. and Torvik, V.I. (2016). Quantifying conceptual novelty in the biomedical literature. D-Lib Magazine 22(9/10).CrossRefGoogle ScholarPubMed
Mou, L., Men, R., Li, G., Xu, Y., Zhang, L., Yan, R. and Jin, Z. (2015). Recognizing entailment and contradiction by tree-based convolution. CoRR abs/1512.08422 .Google Scholar
Parikh, A.P., Täckström, O., Das, D. and Uszkoreit, J. (2016). A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1–4, 2016, pp. 2249–2255.CrossRefGoogle Scholar
Ru, L., Zhao, L., Zhang, M. and Ma, S. (2004). Improved feature selection and redundance computing-thuir at trec 2004 novelty track. In TREC.Google Scholar
Schiffman, B. and McKeown, K.R. (2005). Context and learning in novelty detection. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 716–723.CrossRefGoogle Scholar
Soboroff, I. (2004). Overview of the TREC 2004 novelty track. In Proceedings of the Thirteenth Text REtrieval Conference, TREC 2004, Gaithersburg, Maryland, USA, November 16–19, 2004. Google Scholar
Soboroff, I. and Harman, D. (2003). Overview of the TREC 2003 novelty track. In Proceedings of The Twelfth Text REtrieval Conference, TREC 2003, Gaithersburg, Maryland, USA, November 18–21, 2003, pp. 38–53.Google Scholar
Soboroff, I. and Harman, D. (2005). Novelty detection: The trec experience. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 105–112.CrossRefGoogle Scholar
Stokes, N. and Carthy, J. (2001). First story detection using a composite document representation. In Proceedings of the First International Conference on Human Language Technology Research. Association for Computational Linguistics, pp. 1–8.CrossRefGoogle Scholar
Tang, W., Tsai, F.S. and Chen, L. (2010). Blended metrics for novel sentence mining. Expert Systems with Applications 37(7), 5172–5177.CrossRefGoogle Scholar
Tarassenko, L., Hayton, P., Cerneaz, N. and Brady, M. (1995). Novelty detection for the identification of masses in mammograms. In 1995 Fourth International Conference on Artificial Neural Networks, Cambridge, UK, pp. 442–447.CrossRefGoogle Scholar
Tax, D.M. and Duin, R.P. (1998). Outlier detection using classifier instability. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR). Springer, pp. 593–601.CrossRefGoogle Scholar
Tsai, F.S., Tang, W. and Chan, K.L. (2010). Evaluation of novelty metrics for sentence-level novelty mining. Information Sciences 180(12), 2359–2374.CrossRefGoogle Scholar
Tsai, F.S. and Zhang, Y. (2011). D2s: Document-to-sentence framework for novelty detection. Knowledge and Information Systems 29(2), 419–433.CrossRefGoogle Scholar
Verheij, A., Kleijn, A., Frasincar, F. and Hogenboom, F. (2012). A comparison study for novelty control mechanisms applied to web news stories. In 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology (WI-IAT), vol. 1. IEEE, pp. 431–436.CrossRefGoogle Scholar
Wayne, C.L. (1997). Topic detection and tracking (tdt). In Workshop Held at the University of Maryland on, vol. 27. Citeseer, pp. 28.Google Scholar
Yang, Y., Pierce, T. and Carbonell, J. (1998). A study of retrospective and on-line event detection. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, pp. 28–36.CrossRefGoogle Scholar
Yang, Y., Zhang, J., Carbonell, J. and Jin, C. (2002). Topic-conditioned novelty detection. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 688–693. ACM.CrossRefGoogle Scholar
Zhang, M., Song, R., Lin, C., Ma, S., Jiang, Z., Jin, Y., Liu, Y., Zhao, L. and Ma, S. (2003). Expansion-based technologies in finding relevant and new information: Thu TREC 2002: Novelty track experiments. In NIST SPECIAL PUBLICATION SP (251), 586–590.Google Scholar
Zhang, Y., Callan, J. and Minka, T. (2002). Novelty and redundancy detection in adaptive filtering. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, pp. 81–88.CrossRefGoogle Scholar
Zhang, Y. and Tsai, F.S. (2009). Combining named entities and tags for novel sentence detection. In Proceedings of the WSDM’09 Workshop on Exploiting Semantic Annotations in Information Retrieval. ACM, pp. 30–34.CrossRefGoogle Scholar
Zhao, P. and Lee, D.L. (2016). How much novelty is relevant? it depends on your curiosity. In 39th International ACM SIGIR Conference on Research and Development, Pisa, Italy, pp. 100.Google Scholar