Universal Natural Language Understanding

Even if today's large language models, like ChatGPT, have become extremely good at understanding linguistic meaning, it is still possible to trick them, for example with clever use of negation ('not', 'without', etc.) and other small words that change the meaning of otherwise frequently occurring patterns in their training material. One way to make the models more robust is to create meaning representations based on logic. In this project, we have tried to create such representations starting from syntactic representations that show the structure of sentences, and then translate them into semantic representations, which show the meaning of sentences. The syntactic representations we have used are in a formalism called Universal Dependencies (UD), which is a standard format that has been used for more than 100 languages. This makes it possible to create meaning representations for a wide array of languages. The project has created a rule-based system to translate UD to logic, but we have also trained language models to create logical representations directly. It turns out that language models are very good at creating logical representations for short sentences (up to 10 words), but make many errors when the sentences get longer. The rule-based system performs less well than the language models, but is more robust towards longer sentences. There is reason to believe that one could get a better result by combining rules and neural nets, but this has so far proved difficult. So-called 'neurosymbolic AI' is however an active research area and things may change in the future. The project has also studied so-calleed presuppositions, i.e. meaning that the language users take for granted when using particular expressions. For example, the sentence "I realized that Trump was American" presupposes that Trump is American. Presuppositions are special in that they "survive" under negation. "I did not realize that Trump was American" also presupposes that Trump is American. The project has constructed a large dataset with such sentences from the English Wikipedia. We then tried to train large language models to predict presuppositional meaning. Neither BERT nor ChatGPT was able to do that, showing that there are still big challenges in making language models understand linguistic meaning.

Project pipeline software and new datasets are available at https://github.com/Universal-NLU and will later be disposed at an institutional archive (likely Trolling - The repository of language and linguistics in Tromsø).

This project will use techniques from Glue semantics to derive semantic representations from UD syntax trees. It will build a software pipeline that can map text to meaning by combining a machine-learning approach to syntactic parsing with a largely rule-driven interface to deep, logic-founded semantic representations that improve considerably on the current state of the art. Moreover, the central part of the system will be based exclusively on information from the UD tree, which means that it can be used for any language that has a UD treebank (currently more than 70 languages). In addition, the project will develop tools for post-compositional enrichment of English and Norwegian meaning representations based on lexical knowledge encoded in resources available for those languages. Improved natural language understanding has the potential to help numerous computational tasks from web search to human-robot interaction and so the potential impact of the project is very large.