Cooperative Human Activity Recognition and Localization for Healthcare and Wellbeing

In the 2030 Agenda for Sustainable Development, the United Nations (UN) has established a plan of action for people, planet, and prosperity. One of the goals in this agenda is to "ensure healthy lives and promote well-being for all at all ages." The achievement of this objective is jeopardized by several factors. We have an ageing population, and the means to support people's desire for an independent life are becoming increasingly scarce. Enabling people to remain independent for longer with a minimum cost to society is an increasing challenge that calls for technical solutions in form of human activity recognition (HAR) systems. In this project, we propose a radical new methodology for the design of HAR systems. Our vision is to recognize and to localize a large set of diverse user activities with a reliability of nearly 100%. Our vision will be enabled through the upcoming 5th generation (5G) of mobile communication systems combined with massive multiple-input multiple-output (MIMO) techniques and cooperative processing of signals from radio frequency sensors and audio sensors. We create new 5G services that enable the recognition of user activity, which is beneficial not only to all involved stakeholders, but to the entire society. The project consortium comprises 8 world-class academic and industry partners from 4 countries. The project is of interdisciplinary nature, as the project group consists of experts covering different disciplines, such as Information and Communication Technology (ICT), Mathematics, Computer Science, and eHealth. The project launch in January 2020 was severely affected by the Corona pandemic. Owing to global travel restrictions, the hiring of all PhD students and post-doctoral research fellows could not be completed before November 2020. In Q1 2020, the planned kick-off meeting has been replaced by an online meeting via Zoom. All hard- and software components required for collecting MIMO radio frequency sensor data and audio sensor data have been purchased. With the new equipment, the first measurement campaign has been completed in Q4 2020. A PhD course has been developed, which is currently offered to the PhD students working within CareWell. In the second year of the project, 2021, great progress was made in both the experimental and theoretical areas of research. Several measurement campaigns have collected combined multimode sensor data with radio frequency sensors, audio sensors, optical sensors, and wearable sensors. Multimode sensing systems have been shown to provide human activity detection with higher accuracy than single-mode sensing systems. A highlight was the development of a radio frequency-based pedometer that can also analyze gait symmetry, which is important for e-health applications in hospitals and at home. The 2nd online project meeting was organised in Q3 2021. In addition, a new lecture on "Intelligent High Frequency Sensing" was developed. In the third year of the project, 2022, we completed two paradigm shifts in the development of HAR systems. The first paradigm shift led to a new class of channel models for the design of HAR systems using radar sensors. This new class of channel models enabled the second paradigm shift from traditional experimental-based design approaches of HAR systems to a fully software-based design methodology. By using our software-based design approach, HAR systems can now be trained without involving humans. The basic idea is that humans in the design phase are replaced by avatars, which can be grouped together to represent a family of active users of different size, weight, and gender. By replacing humans with avatars, all physical experiments can be replaced by simulations, which drastically reduces costs and development time in the design phase of HAR systems. The 3rd project meeting was physically held at UiA in Q3 2022. In the final year of CareWell, we focused on the development of a hardware prototype of a human activity recognition (HAR) system using multiple radar sensors and machine learning techniques. The HAR system has been trained to recognize seven activities of daily living, including falls, walking, sitting down, standing up, picking up an object. In addition, the HAR system has been trained to recognize 17 sport activities in indoor environments, such as jumping, squats, putting one foot to the side (forwards), lifting legs, up and down movements of the arms, to name just a few examples. The developed prototype operates in real-time and has successfully demonstrated in a live demonstration to healthcare workers of Lillesand Municipality. The final physical project meeting was organised in Q3 2023. The research results of the CareWell project have been published in 18 journal papers, 11 conference papers, 3 book chapters, and 2 media/press articles. Furthermore, the project results have been disseminated in 3 keynote presentations and numerous invited talks at international conferences and universities.

Outcomes: The main project result is a prototype of a human activity recognition (HAR) system using multiple radar sensors and machine learning techniques. The HAR system has been trained to recognize seven activities of daily living, including falls, walking, sitting down, standing up, picking up an object. In addition, the HAR system has been trained to recognize 17 sport activities in indoor environments, such as jumping, squats, putting one foot to the side (forwards), lifting legs, up and down movements of the arms, to name just a few examples. The developed prototype operates in real-time and has successfully demonstrated in a live-demo to healthcare workers of Lillesand Municipality. One PhD student, Mr. Sahil Waqar (UiA), has submitted his PhD thesis in January 2024. During his 3-year contract at UiA, he published 5 journal papers as first author and co-authored 2 conference papers as second author. The other two PhD students, Mr. Masoud Mohtadifar (NTNU) and Mr. Hasan Najjar (UiA), are very late with their doctorates and were unable to complete their work in the expected time. The research results of the CareWell project have been published in 18 journal papers, 11 conference papers, 3 book chapters, and 2 media/press articles. Furthermore, the project results have been disseminated in 3 keynote presentations (ICTCT’20, MMS’23, and ECCE’23) and numerous invited talks at international conferences and universities. Impacts: CareWell offers a unique solution for monitoring systems for commercial companies providing services that are better to identify situations requiring attention. The proposed system collects a large amount of data that can be made available to various stakeholders, e.g., healthcare providers and policy makers, to further explore the short- and long-term trends in user activity. CareWell will bring substantial societal benefits and creates entirely new use cases in market segments beyond the healthcare and wellbeing sector, such as co-operative sports, interactive worker-robot collaboration, pedestrian safety systems, ambient assisted living, and Internet of Things. CareWell will create new market opportunities and services for 5G, which will strengthen the EU technology offer of innovative 5G solutions for industry.

In the 2030 Agenda for Sustainable Development, the United Nations (UN) has established a plan of action for people, planet, and prosperity. One of the goals in this agenda is to ''ensure healthy lives and promote well-being for all at all ages.'' The achievement of this objective is jeopardized by a number of factors. We have an ageing population, and the means to support people's desire for an independent life are becoming increasingly scarce. Enabling people to remain independent for longer with a minimum cost to society is an increasing challenge that calls for technical solutions in form of human activity recognition (HAR) systems. Current HAR systems include wearable sensors, which users can easily forget or ignore for comfort reasons. On the other hand, environmental sensors, such as video sensors, come at the cost of high privacy risks. Emergency buttons in current alternatives are ineffective if the patient cannot reach them or is incapacitated. These shortcomings prevent a wide adoption of current HAR techniques in the healthcare and wellbeing sector. In this project, we propose a radical new methodology for the design of HAR systems. Our vision is to recognize and to localize a large set of diverse user activities with a reliability of nearly 100%. Our vision will be enabled through the upcoming 5th generation (5G) of mobile communication systems combined with cooperative processing of signals from radio frequency sensors and audio sensors. We offer new services for 5G systems by using massive MIMO (multiple-input multiple-output) and cooperative techniques to recognize user activities and to provide localization benefits for the entire community. The project consortium comprises 8 world-class academic and industry partners from 4 countries. The project is of interdisciplinary nature, as the project group consists of experts covering different disciplines, such as Information and Communication Technology (ICT), Mathematics, Computer Science, and eHealth.