Skip to main content
SpringerLink
Log in

Towards bridging the neuro-symbolic gap: deep deductive reasoners

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Symbolic knowledge representation and reasoning and deep learning are fundamentally different approaches to artificial intelligence with complementary capabilities. The former are transparent and data-efficient, but they are sensitive to noise and cannot be applied to non-symbolic domains where the data is ambiguous. The latter can learn complex tasks from examples, are robust to noise, but are black boxes; require large amounts of –not necessarily easily obtained– data, and are slow to learn and prone to adversarial examples. Either paradigm excels at certain types of problems where the other paradigm performs poorly. In order to develop stronger AI systems, integrated neuro-symbolic systems that combine artificial neural networks and symbolic reasoning are being sought. In this context, one of the fundamental open problems is how to perform logic-based deductive reasoning over knowledge bases by means of trainable artificial neural networks. This paper provides a brief summary of the authors’ recent efforts to bridge the neural and symbolic divide in the context of deep deductive reasoners. Throughout the paper we will discuss strengths and limitations of models in term of accuracy, scalability, transferability, generalizabiliy, speed, and interpretability, and finally, will talk about possible modifications to enhance desirable capabilities. More specifically, in terms of architectures, we are looking at Memory-augmented networks, Logic Tensor Networks, and compositions of LSTM models to explore their capabilities and limitations in conducting deductive reasoning. We are applying these models on Resource Description Framework (RDF), first-order logic, and the description logic \(\mathcal {E}{\mathscr{L}}^{+}\) respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. The topic is being investigated, of course, with some recent progress being made. For example, [1] report on an application of deep learning to planning, and explicitly frame it as work towards bridging the “subsymbolic-symbolic boundary.”

  2. Evrahimi et al [22] is under review at AAAI-MAKE 2021 symposium at the time of submitting this journal paper.

  3. Induction like in Inductive Logic Programming or Relational Learning has statistical aspects and is much closer in nature to a machine learning task, and thus arguably easier to tackle using machine learning approaches.

  4. Any may be too grandiose a statement, but these are the ones we are looking at.

  5. Attempting to find finite representations for infinite sets – in the cases where this would even be reasonably possible – would add another layer of complication which we are currently not considering.

  6. In fact, it comes with three different ones, but we have only considered the most comprehensive one, the RDFS Semantics.

  7. Datalog is equivalent to function-free definite logic programming [34].

  8. https://github.com/thunlp/KRLPapers has an extensive listing of existing work on knowledge embeddings.

  9. See [10, 70] for a recent survey.

  10. Some deductive entailment algorithms can even be understood as simply a type of syntax rewriting systems.

  11. https://lod-cloud.net/

  12. http://lodlaundromat.org/

  13. https://jena.apache.org

  14. https://github.com/vinid/ltns-experiments

  15. We will use the prefix dbr: to refer to DBpedia entities.

  16. https://github.com/vinid/logical_commonsense

  17. https://psl.linqs.org/

  18. DNN cannot be used because the training consists of just positive instantiated atoms, the network would eventually just learn to output 1 for every input.

  19. https://github.com/aaronEberhart/ERCompletionReasoningLSTM

References

  1. Asai M, Fukunaga A (2018) Classical planning in deep latent space: Bridging the subsymbolic-symbolic boundary. In: McIlraith SA, Weinberger KQ (eds) Proceedings of the Thirty-Second AAAI Conference on artificial intelligence, New Orleans, Louisiana, USA, February 2-7, 2018, AAAI Press

  2. Bach SH, Broecheler M, Huang B, Getoor L (2017) Hinge-loss Markov random fields and probabilistic soft logic. J Mach Learn Res 18:1–67

    MathSciNet  MATH  Google Scholar 

  3. Bader S, Hitzler P, Hölldobler S (2008) Connectionist model generation: a first-order approach. Neurocomputing 71(13-15):2420–2432

    Article  Google Scholar 

  4. Bader S, Hitzler P, Hölldobler S, Witzel A (2007) A fully connectionist model generator for covered first-order logic programs. In: Veloso MM (ed) IJCAI 2007, Proceedings of the 20th international joint conference on artificial intelligence, Hyderabad, India, January 6-12, 2007, pp 666–671

  5. Bahdanau D, Bosc T, Jastrzebski S, Grefenstette E, Vincent P, Bengio Y (2017) Learning to compute word embeddings on the fly. arXiv:1706.00286

  6. Besold TR, d’Avila Garcez A, Bader S, Bowman H, Domingos P, Hitzler P, Kühnberger K, Lamb L, Lowd D, Lima PMV, de Penning L, Pinkas G, Poon H, Zaverucha G (2017) Neural-symbolic learning and reasoning: A survey and interpretation. arXiv:1711.03902

  7. Bianchi F, Hitzler P (2019) On the capabilities of logic tensor networks for deductive reasoning. In: AAAI Spring symposium: combining machine learning with knowledge engineering

  8. Bianchi F, Palmonari M, Hitzler P, Serafini L (2019) Complementing logical reasoning with sub-symbolic commonsense. In: International joint conference on rules and reasoning, Springer, pp 161–170

  9. Bianchi F, Palmonari M, Nozza D (2018) Towards encoding time in text-based entity embeddings. In: International semantic web conference, Springer, pp 56–71

  10. Bianchi F, Rossiello G, Costabello L, Palmonari M, Minervini P (2020) Knowledge graph embeddings and explainable AI arXiv:2004.14843

  11. Bojanowski P, Grave E, Joulin A, Mikolov T (2017) Enriching word vectors with subword information. Trans Assoc Comput Linguist 5:135–146

    Article  Google Scholar 

  12. Bordes A, Usunier N, García-Durán A, Weston J, Yakhnenko O (2013) Translating embeddings for modeling multi-relational data. In: Burges CJC, Bottou L, Ghahramani Z, Weinberger KQ (eds) Advances in neural information processing systems 26: 27th Annual conference on neural information processing systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp 2787–2795

  13. Cai H, Zheng VW, Chang K (2018) A comprehensive survey of graph embedding: problems, techniques and applications. IEEE Trans Knowl Data Eng

  14. Confalonieri R, Besold TR, Weyde T, Creel K, Lombrozo T, Mueller ST, Shafto P (2019) What makes a good explanation? cognitive dimensions of explaining intelligent machines. In: Goel AK, Seifert CM, Freksa C. (eds) Proceedings of the 41th Annual meeting of the cognitive science society, CogSci 2019: Creativity + Cognition + Computation, Montreal, Canada, July 24-27, 2019, pp 25–26. cognitivesciencesociety.org. https://mindmodeling.org/cogsci2019/papers/0013/index.html

  15. Cyganiak R., Wood D, Lanthaler M (eds.) RDF 1.1 Concepts and Abstract Syntax. W3C Recommendation 25 February 2014 (2014). Available from http://www.w3.org/TR/rdf11-concepts/

  16. Das R, Dhuliawala S, Zaheer M, Vilnis L, Durugkar I, Krishnamurthy A, Smola A, McCallum A (2018) Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. In: International conference on learning representations

  17. De Raedt L, Kimmig A (2015) Probabilistic (logic) programming concepts. Mach Learn 100 (1):5–47

    Article  MathSciNet  Google Scholar 

  18. De Silva TS, MacDonald D, Paterson G, Sikdar KC, Cochrane B (2011) Systematized nomenclature of medicine clinical terms (SNOMED CT) to represent computed tomography procedures. Comput Methods Prog Biomed 101(3):324–329. https://doi.org/10.1016/j.cmpb.2011.01.002

    Article  Google Scholar 

  19. Donadello I, Serafini L, d’Avila Garcez A (2017) Logic tensor networks for semantic image interpretation. In: Sierra C (ed) Proceedings of the twenty-sixth international joint conference on artificial intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pp 1596–1602. ijcai.org

  20. Dong H, Mao J, Lin T, Wang C, Li L, Zhou D (2018) Neural logic machines. In: International conference on learning representations

  21. Eberhart A, Ebrahimi M, Zhou L, Shimizu C, Hitzler P (2020) Completion reasoning emulation for the description logic EL+. In: Martin A, Hinkelmann K, Fill H, Gerber A, Lenat D, Stolle R, van Harmelen F (eds) Proceedings of the AAAI 2020 Spring symposium on combining machine learning and knowledge engineering in practice, AAAI-MAKE 2020, Palo Alto, CA, USA, March 23-25, 2020, Volume I, CEUR Workshop Proceedings. CEUR-WS.org. http://ceur-ws.org/Vol-2600/paper5.pdf, vol 2600

  22. Ebrahimi M, Sarker MK, Bianchi F, Xie N, Doran D, Hitzler P (2018) Reasoning over RDF knowledge bases using deep learning arXiv:1811.04132

  23. Fung P, Wu C, Madotto A (2018) Mem2seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems. In: Gurevych I, Miyao Y (eds) Proceedings of the 56th annual meeting of the association for computational linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp 1468–1478. Association for Computational Linguistics

  24. d’Avila Garcez A, Lamb L, Gabbay DM (2009) Neural-symbolic cognitive reasoning. Springer, Heidelberg

    MATH  Google Scholar 

  25. d’Avila Garcez A, Besold TR, Raedt LD, Földiák P, Hitzler P, Icard T, Kühnberger K, Lamb L, Miikkulainen R, Silver DL (2015) Neural-symbolic learning and reasoning: Contributions and challenges. In: 2015 AAAI spring symposia, Stanford University, Palo Alto, California, USA, March 22-25, 2015. http://www.aaai.org/ocs/index.php/SSS/SSS15/paper/view/10281. AAAI Press

  26. Grefenstette E (2013) Towards a formal distributional semantics: Simulating logical calculi with tensors. In: Second joint conference on lexical and computational semantics (* SEM), Volume 1: Proceedings of the main conference and the shared task: semantic textual similarity, pp 1–10

  27. Grefenstette E, Hermann KM, Suleyman M, Blunsom P (2015) Learning to transduce with unbounded memory. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in neural information processing systems 28: Annual conference on neural information processing systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp 1828–1836

  28. Gust H, Kühnberger K, Geibel P (2007) Learning models of predicate logical theories with neural networks based on topos theory. In: Hammer B, Hitzler P (eds) Perspectives of neural-symbolic integration, studies in computational intelligence, vol 77. Springer, pp 233–264

  29. Hammer B, Hitzler P (eds) (2007) Perspectives of neural-symbolic integration, studies in computational intelligence, vol 77. Springer, Berlin

    Google Scholar 

  30. Hitzler P (2021) Semantic Web: A review of the field. Communications of the ACM. To appear

  31. Hitzler P, Hölldobler S, Seda AK (2004) Logic programs and connectionist networks. J Appl Logic 2(3):245–272

    Article  MathSciNet  Google Scholar 

  32. Hitzler P, Krötzsch M, Parsia B, Patel-Schneider PF, Rudolph S OWL 2 Web Ontology Language: primer (Second Edition). W3C Recommendation 11 December 2012 (2012). Available from http://www.w3.org/TR/owl2-primer/

  33. Hitzler P, Krötzsch M, Rudolph S (2010) Foundations of semantic web technologies. Chapman & Hall/CRC, Boca Raton

    Google Scholar 

  34. Hitzler P, Seda AK (2011) Mathematical aspects of logic programming semantics. Chapman and Hall / CRC studies in informatics series. CRC Press, Boca Raton

    MATH  Google Scholar 

  35. Hohenecker P, Lukasiewicz T (2020) Ontology reasoning with deep neural networks. J Artif Intell Res 68:503–540

    MathSciNet  MATH  Google Scholar 

  36. Hölldobler S, Kalinke Y (1994) Ein massiv paralleles Modell für die Logikprogrammierung. In: WLP, pp 89–92

  37. Kazakov Y, Krötzsch M, Simančík F (2012) Elk: a reasoner for OWL EL ontologies. System Description

  38. Kiros R, Zhu Y, Salakhutdinov RR, Zemel R, Urtasun R, Torralba A, Fidler S (2015) Skip-thought vectors. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in neural information processing systems 28. Curran Associates, Inc., pp 3294–3302

  39. Klir G, Yuan B (1995) Fuzzy sets and fuzzy logic, vol 4, Prentice hall, New Jersey

  40. Kulmanov M, Liu-Wei W, Yan Y, Hoehndorf R (2019) EL embeddings: geometric construction of models for the description logic EL++. In: Proceedings of the 28th International joint conference on artificial intelligence, AAAI Press, pp 6103–6109

  41. Lao N, Mitchell T, Cohen WW (2011) Random walk inference and learning in a large scale knowledge base. In: Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, pp 529–539

  42. Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: International conference on machine learning, pp 1188–1196

  43. Lin Y, Liu Z, Sun M, Liu Y, Zhu X (2015) Learning entity and relation embeddings for knowledge graph completion. In: Bonet B, Koenig S (eds) Proceedings of the Twenty-Ninth AAAI conference on artificial intelligence, January 25-30, 2015, Austin, Texas, USA, AAAI Press, pp 2181–2187

  44. Ling W, Dyer C, Black AW, Trancoso I, Fermandez R, Amir S, Marujo L, Luís T (2015) Finding function in form: Compositional character models for open vocabulary word representation. In: Proceedings of the 2015 conference on empirical methods in natural language processing, pp 1520–1530

  45. Makni B, Hendler J (2019) Deep learning for noise-tolerant RDFS reasoning. Semantic Web 10(5):823–862

    Article  Google Scholar 

  46. McCarthy J (1988) Epistemological challenges for connectionism. Behav Brain Sci 44

  47. McCulloch WS, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys 5(4):115–133

    Article  MathSciNet  Google Scholar 

  48. Meza-Ruiz I, Riedel S (2009) Jointly identifying predicates, arguments and senses using Markov logic. In: Proceedings of human language technologies: The 2009 annual conference of the north american chapter of the association for computational linguistics, pp 155–163

  49. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp 3111–3119

  50. Minervini P, Bošnjak M, Rocktäschel T, Riedel S, Grefenstette E (2020) Differentiable reasoning on large knowledge bases and natural language. In: Proceedings of the AAAI conference on artificial intelligence. https://doi.org/10.1609/aaai.v34i04.5962, vol 34, pp 5182–5190

  51. Minervini P, Riedel S, Stenetorp P, Grefenstette E, Rocktäschel T (2020) Learning reasoning strategies in end-to-end differentiable proving. In: ICML

  52. Neelakantan A, Roth B, McCallum A (2015) Compositional vector space models for knowledge base completion. In: Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (Volume 1: Long Papers), pp 156–166

  53. Nguyen DQ, Nguyen DQ, Nguyen TD, Phung D (2019) A convolutional neural network-based model for knowledge base completion and its application to search personalization. Semantic Web 10(5):947–960

    Article  Google Scholar 

  54. Pennington J, Socher R, Manning C (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543

  55. Raghu D, Gupta N (2018) Mausam: Hierarchical pointer memory network for task oriented dialogue. arXiv:1805.01216

  56. Richardson M, Domingos P (2006) Markov logic networks. Mach Learn 62(1-2):107–136

    Article  Google Scholar 

  57. Ristoski P, Paulheim H (2016) RDF2Vec: RDF graph embeddings for data mining. In: International semantic web conference. Springer, pp 498–514

  58. Rocktȧschel T, Riedel S (2017) End-to-end differentiable proving. In: Guyon I, von Luxburg U, Bengio S, Wallach HM, Fergus R, Vishwanathan SVN, Garnett R (eds) Advances in neural information processing systems 30: Annual conference on neural information processing systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp 3791–3803

  59. Rocktäschel T, Riedel S (2017) End-to-end differentiable proving. In: Advances in neural information processing systems, pp 3788–3800

  60. Rocktäschel T, Singh S, Riedel S (2015) Injecting logical background knowledge into embeddings for relation extraction. In: Proceedings of the 2015 conference of the north american chapter of the association for computational linguistics: human language technologies, pp 1119–1129

  61. Serafini L, d’Avila Garcez A (2016) Logic tensor networks: Deep learning and logical reasoning from data and knowledge. arXiv:1606

  62. Serafini L, d’Avila Garcez A (2016) Learning and reasoning with logic tensor networks. In: Adorni G, Cagnoni S, Gori M, Maratea M (eds) AI⋆A 2016: Advances in artificial intelligence – XVth international conference of the italian association for artificial intelligence, Genova, Italy, November 29 – December 1, 2016, Proceedings, Lecture Notes in Computer Science, vol 10037. Springer, pp 334–348

  63. Serafini L, d’Avila Garcez A (2016) Logic tensor networks: Deep learning and logical reasoning from data and knowledge. In: Besold TR, Lamb LC, Serafini L, Tabor W (eds) Proceedings of the 11th international workshop on neural-symbolic learning and reasoning (NeSy’16) co-located with the Joint Multi-Conference on Human-Level Artificial Intelligence (HLAI 2016), New York City, NY, USA, July 16-17, 2016., CEUR Workshop Proceedings. CEUR-WS.org, vol 1768

  64. Shastri L (1999) Advances in SHRUTI-A neurally motivated model of relational knowledge representation and rapid inference using temporal synchrony. Appl Intell 11(1):79–108

    Article  Google Scholar 

  65. Shastri L (2007) SHRUTI: A neurally motivated architecture for rapid, scalable inference. In: Hammer B, Hitzler P (eds) Perspectives of neural-symbolic integration, studies in computational intelligence, vol 77. Springer, pp 183–203

  66. Socher R, Chen D, Manning CD, Ng AY (2013) Reasoning with neural tensor networks for knowledge base completion. In: Burges CJC, Bottou L, Ghahramani Z, Weinberger KQ (eds) Advances in neural information processing systems 26: 27th annual conference on neural information processing systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., pp 926–934

  67. Sukhbaatar S, Szlam A, Weston J, Fergus R (2015) End-to-end memory networks. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in neural information processing systems 28: annual conference on neural information processing systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp 2440–2448

  68. Towell GG, Shavlik JW (1994) Knowledge-based artificial neural networks. Artif Intell 70 (1-2):119–165

    Article  Google Scholar 

  69. Trouillon T, Welbl J, Riedel S, Gaussier E, Bouchard G (2016) Complex embeddings for simple link prediction. In: Balcan M, Weinberger KQ (eds) Proceedings of the 33nd International conference on machine learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, JMLR Workshop and Conference Proceedings. JMLR.org, vol 48, pp 2071–2080

  70. Wang Q, Mao Z, Wang B, Guo L (2017) Knowledge graph embedding: A survey of approaches and applications. IEEE Trans Knowl Data Eng 29(12):2724–2743

    Article  Google Scholar 

  71. Wang WY, Cohen WW (2016) Learning first-order logic embeddings via matrix factorization. In: IJCAI, pp 2132–2138

  72. Wang Z, Zhang J, Feng J, Chen Z (2014) Knowledge graph embedding by translating on hyperplanes. In: Brodley CE, Stone P (eds) Proceedings of the Twenty-Eighth AAAI Conference on artificial intelligence, July 27 -31, 2014, Quėbec City, Quėbec, Canada., AAAI Press, pp 1112–1119

  73. Weston J, Chopra S, Bordes A (2014) Memory networks. arXiv:1410.3916. Published as ICLR 2015 conference paper

  74. Wikibooks contributors: Algorithm implementation/strings/levensh- tein distance (2019 (accessed November 19, 2019)). https://en.wikibooks.org/wiki/Algorithm_Implementation/Strings/Levenshtein_distance#Python

  75. Yang B, Yih W, He X, Gao J, Deng L (2014) Embedding entities and relations for learning and inference in knowledge bases. arXiv:1412.6575

  76. Yang F, Yang Z, Cohen WW (2017) Differentiable learning of logical rules for knowledge base reasoning. In: Advances in neural information processing systems, pp 2319–2328

Download references

Acknowledgements

This work was supported by the Air Force Office of Scientific Research under award number FA9550-18-1-0386 and by the National Science Foundation (NSF) under award OIA-2033521 “KnowWhereGraph: Enriching and Linking Cross-Domain Knowledge Graphs using Spatially-Explicit AI Technologies.”

Author information

Authors and Affiliations

  1. Data Semantics Laboratory, Kansas State University, Manhattan, KS, USA

    Monireh Ebrahimi, Aaron Eberhart & Pascal Hitzler

  2. Bocconi University, Milan, Italy

    Federico Bianchi

Authors
  1. Monireh Ebrahimi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aaron Eberhart
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Federico Bianchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pascal Hitzler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Monireh Ebrahimi.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: 30th Anniversary Special Issue

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ebrahimi, M., Eberhart, A., Bianchi, F. et al. Towards bridging the neuro-symbolic gap: deep deductive reasoners. Appl Intell 51, 6326–6348 (2021). https://doi.org/10.1007/s10489-020-02165-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-02165-6

Keywords

Search

Navigation