Next generation Earth observation data analysis by integrating human knowledge and AI

Today's AI models primarily learn by analyzing vast amounts of data, often thousands of examples, where the content of each example is pre-categorized or "labeled." Although this method is effective, it has a limitation: AI does not develop the same "common sense" that humans have. It doesn't understand basic principles about the world, like the laws of physics or general truths. As a result, AI sometimes makes predictions or answers that seem completely unreasonable or contradict what we know to be true. The goal of this project is to improve deep learning models by incorporating human knowledge, guidelines, and understanding of physics. We are exploring the use of knowledge graphs and hierarchical class structures to represent and integrate this knowledge. These improved models are then used to further develop automatic analysis of earth observation data. One of the concrete applications we are looking at is "zero-shot learning," where models predict classes that are not represented in the training data. By utilizing such knowledge, the training of deep learning models becomes more efficient and accurate, even with fewer data points. We have also started training a foundation model that is trained with non-annotated data using so-called self-supervised learning, and which will form the basis for further analysis. The project focuses on two central challenges within earth observation: mapping peatlands and mapping snow cover. We have already established benchmark results on peatland mapping using classic U-Net models. Expected results include improved methods for land cover mapping, which will contribute to better management of ecosystems in peatlands. In addition, improved snow cover maps will provide more accurate estimates of water resources available for hydropower production and better monitoring of climate change in the cryosphere. The project is a collaboration between the NR, UiT, NIBIO, and EdInsights.

Earth observation (EO) systems offer unprecedented volumes of data. To utilize this information, automated analysis of EO data is an ongoing research activity. Deep learning (DL) has revolutionized the analysis of image data and is currently state-of-the art for solving a wide range of remote sensing tasks. However, although DL can fit a relatively accurate mapping of the input data, it can easily fail to capture underlying rules and constraints. The project goals are to integrate human knowledge, constraints and physics into DL models, to develop semi-supervised learning (SSL) methods that utilizes human knowledge, and to develop methodology to estimate the corresponding pixel-wise uncertainties. Critical R&D challenges include design of knowledge graphs and how such graphs can be efficiently integrated into DL models, how to perform SSL for multi-sensor and multi-temporal data where modalities may be missing, how to utilize integrated knowledge for improved learning capabilities, and how to efficiently model the pixel-wise uncertainties. Methodology that enables the inclusion of human knowledge are of great interest within the research community and will have a big impact on next generation DL methods. This includes improved test accuracy, learning capabilities, trust, reliability, and understanding behind the reasoning. The methodology developed is generic and will therefore be applicable to other scientific challenges, outside of the stakeholders’ applications. The project focuses on two mapping challenges: wetland and snow cover. The outcome of this project will directly enhance our understanding of wetlands in Norway, and will contribute to reduced loss of wetland areas and maintenance of biodiversity and ecosystem services. Improved maps of snow cover will provide better estimates of water resources available for hydropower production. Moreover, improved snow cover estimates will enable better monitoring of climate changes in the cryosphere.