Multi-Modal Structure-Informed Machine Learning

This project supports a PhD project connected to the project PhysML: Structure-based machine learning for physical systems. One of the main barriers for applying machine learning on physical systems is the lack of robustness and trustworthiness of existing methods. PhysML will contribute to solving this challenge by combining machine learning models with the geometric properties of mathematical models in a hybrid analytics framework that alleviates the weaknesses of both individual approaches by leveraging their complementary strengths. The goal of this extension is to connect the research done and methods developed in PhysML to applications that are of interest to the Norwegian defence sector. This will be done by applying methods developed in PhysML to relevant use cases, but also to develop new methods that meet the needs of those use cases. We aim to extend ongoing research in hybrid machine learning by incorporating the measurement process via different sensors that each only provide partial information about the system and are typically considered as an entity. We wish to study the interplay between sensors and system modelling in the learning process, building modular models for sensor and system modelling together, allowing to use the same model under a change of available sensors, thus achieving Multi-Modal Structure-Informed Machine Learning (MMSIML). This would facilitate transfer learning between different sensors. E.g., a model can be trained with high-quality sensors and then applied with a different set of sensors. MMSIML would also be stable against missing sensor modalities (in contrast to classical missing data problems), which could occur if sensors fail during production, as long as the total information is still sufficient.

The PhysML project will contribute to solving this challenge by combining machine learning models with structure-based mathematical models. This leads to hybrid models that have the flexibility and ability to learn from data that comes with doing machine learning, while also having guarantees of stability and a behavior of the solution that is consistent with the underlying physics. PhysML will work towards two goals: i) Use machine learning to gain physical knowledge about the systems, and ii) use physical knowledge to obtain machine learning models that are open, trustable, robust, and flexible. A fundamental innovation in our approach is to utilize the assumed underlying structures of the system, such as symmetry or preservation of energy, and thus build on the established field called geometric numerical integration, which is the study of how to incorporate such structures in mathematical models. We are seeking funding for a PhD project within physics-informed ML, with applications to sensor data. We aim to extend ongoing research in hybrid machine learning by incorporating the measurement process via different sensors that each only provide partial information about the system and are typically considered as an entity. We wish to study the interplay between sensors and system modelling in the learning process, building modular models for sensor and system modelling together, allowing to use the same model under a change of available sensors, thus achieving Multi-Modal Structure-Informed Machine Learning (MMSIML). This would facilitate transfer learning between different sensors. E.g., a model can be trained with high-quality sensors and then applied with a different set of sensors. MMSIML would also be stable against missing sensor modalities (in contrast to classical missing data problems), which could occur if sensors fail during production, as long as the total information is still sufficient.