KnowMe AI - Sensor and machine learning based interpretation of non-verbal communication.

There are people who cannot communicate verbally due to varying degrees of cognitive challenges. Even though verbal language is absent, they communicate to different extents through sounds, facial expressions, and gestures. The goal of this project is to use passive sensors to observe the person and translate the composite expressions into meaningful information using advanced machine learning models. Technology that became available during the project period enables the generation of 3D skeleton models and 3D facial landmarks based on 2D imaging. The system will be able to be trained to translate a person's emotional expressions in real-time based on gestures, facial expressions, and sounds. This will provide a unique tool for communication between non-verbal clients and caregivers. Additionally, the features used to interpret expressions will be applied to search in the clients' video database. This is a challenging project that potentially could have a significant impact on many people's quality of life.

Prosjektet har kommet frem til metoder og algoritmer som kan benyttes for å tolke ikke-verbale uttrykk basert på video og lyd. Tolkningen baserer seg på gester (endring av kroppspositurer over en tidsperiode), ansiktsuttrykk og lydsekvenser. I tillegg er det utviklet metode for å slå opp i videodatabase basert på egenskaper utviklet i forbindelse med tolkning av uttrykk. Når alle resultatene fra prosjektet er integrert i KnowMe forventes det at nytteverdien til produktet øker betraktelig fordi man får en kontinuerlig tilbakemelding på klientens sinnstilstand og uttrykk. Ved å få tilbakemelding på klientens uttrykk er det lettere å møte vedkommende på hans eller hennes behov. Dette bidrar til økt livskvalitet for klienten. Økt nytteverdi vil gi økt utbredelse og dermed mer salg og bedre økonomi for selskapet.

Peoples who are not able to communicate verbally needs augmentative and alternative communication (AAC) to be understood and to interact with the surroundings. In order to improve the quality of life of these peoples, and to facilitate the work of guardians and other caregivers, there is a strong need for a solution that can translate non-verbal communication consisting of sound, facial expressions and body gestures, to something understandable. Today's methodology for translating expressions of persons in need of AAC is based on written notes describing expressions and signs (indexical signs), and the interpretation and response hypothesis. Caregivers often provide assistance to many different people, making it extremely difficult to learn the repertoire of expressions their individual clients have. This results in misinterpretation that leads to frustration, often violent behavior and resignation. To look up written notes take time, and the caregiver’s response time is relevant to how communication is perceived. This innovation aims to develop a system that records sound, facial expressions and body gestures and uses machine learning to interpret the compound expression in real-time. The goal is to facilitate quick and adequate response from the caregiver. The innovation shall be used in everyday situations, both indoors and outdoors. This will ease the everyday life for caregivers, improve quality of life for the person in question and most likely contribute to a positive development of cognitive skills and an extended expression repertoire. The solution will thus support the UN Convention on the Rights of Persons with Disabilities. The innovation is unique to the market and challenges the state of art in image sensor usage and machine learning, and hence is in need of extensive research. The final product will have great impact on the company growth and financial development. The project will be run in close cooperation with Sintef and Norsk Regnesentral.