Tools and Methods for Autonomous Analysis of Human Activities from Wearable Device Sensor Data

The aim of AutoActive was to develop tools, methods and algorithms for research on human activities based on data from multiple sensors. The project was initiated in 2017 and finalized by the end of 2022. The results of the project will be valuable in a wide range of applications and have been explored in two specific cases: analysis of performance and technique within sports, with a focus on cross-country skiing, and chronic disease management, with a focus on patients with multiple sclerosis (MS). The project has been led by SINTEF Digital, Department of Smart Sensors and Microsystems, and involved research partners from five other environments: 1) NTNU (Center for Elite Sports Research; SenTIF), 2) Olympiatoppen, 3) Oslo University Hospital (Department of Neurology), 4) MS-Senteret Hakadal, and 5) The University of Oslo (Digital signal processing and image analysis). The project has had challenges with personnel and shutdown related to the Covid-19 pandemic, but despite this managed to complete all the planned data collections. The collaboration between the partners has been excellent. The project has published 21 scientific articles in peer-reviewed journals, 16 popular science publications, 34 disseminations in the form of reports, newsletters and conference presentations, as well as several notices in the mass media. Still, five more scientific papers are under development, of which four will be published by the first quarter of 2023. The AutoActive Research environment, consisting of ActivityPresenter, Matlab and Phyton toolboxes, has been developed and is released in GIT Hub. A scientific paper on the development was published in the Journal of Open Source Software in 2022. The environment can receive, synchronize and visualize time series sensor data and video. Regular meetings have been arranged between the developers and the reference group, with representatives from both user scenarios present, to ensure that the platform is developed in line with user requirements. In the sports case, six data collections have been completed on recreational and elite skiers in the laboratory at SenTIF, outdoors in Holmenkollen on roller skis, and on snow in Meråker, Trondheim and Gjøvik. These data have provided the basis for multiple scientific publications in different journals; Frontiers, Sensors, Plus One, Sports Biomechanics, International Journal of Sports Physiology and Performance and the International Journal of Sports Medicine. Three more papers are under development. Results have also been presented at six different conferences; the Nordic Winter Sport Conference in 2019 and 2022, Olympiatoppen research conference in 2019, the European College of Sport Science conference (ECSS) in 2020 and 2022, and the Third Annual Symposium On Sport Sciences in Aalborg University in 2021. A data collection has also been carried out on well-trained cyclists at Kristiania University College. This work has been presented at the ECSS conference in 2020 and 2022, and a scientific article was published in Frontiers in 2022. In the MS case, a large data collection including 55 patients and 20 healthy controls at the MS-Senteret Hakadal was completed in October 2021. Here, movement data was collected during several standard tests conducted at the start and the end of a rehabilitation stay. Preliminary results have been presented on the European Network for Rehabilitation in MS Conference (euRIMS) in 2020 and 2021 as well as on The annual conference of the European Committee for treatment and research in multiple sclerosis (ECTRIMS) in 2022. One scientific paper was published in Journal of Neurology in 2022, and two other papers will be published by the first quarter of 2023.

Resultatene fra AutoActive har vist at det er mulig å gjennomføre presise og komplekse datainnsamlinger ved bruk av bærbare sensorer, uavhengig av en laboratoriesetting. Dette kan ha stor betydning for videre forskning hvor man kan innhente data i en naturlig setting, noe som gir betydelig større mulighet til å forstå den enkelte pasient eller utøvers utfordringer og potensial for å opprettholde og/eller bedre egen funksjon og ytelse. Prosjektet har innhentet data fra to ulike case; idrett (langrenn) og multippel sklerose, MS (kronisk sykdom). Gjennom langrenns-caset er det vist at bruk av sensorer i felt kan benyttes til å gi umiddelbar feedback til utøvere, som gjør at de kan justere og tilpasse sin atferd for bedre prestasjon. Dette har betydelig overføringsverdi også for individualiserte rehabiliterings- og treningsforløp for pasienter. Resultatene fra prosjektet vil ha stor verdi i videre arbeid med sensorteknologi i felt, hvor målet bør være å få frem så presise data som mulig ved bruk av færrest mulig sensorer. I idretts-caset har prosjektet har jobbet tett med landslagene i langrenn, paralangrenn, roing og alpint. Det er gjennomført pilottesting og brukergruppemøter med Olympiatoppen og de ovennevnte idrettsforbundene med jevne mellomrom. Disse pilottestene og jevnlige brukergruppemøter anses som svært viktige for idretten, og prosjektet har hatt tett dialog med landslagstrenerne, idrettssjefen og mange av utøverne. Dette har gitt et unikt utgangspunkt for å utdanne og utvikle tjenestesystemene i disse idrettene, i tillegg gir det oss kontinuerlige tilbakemeldinger fra utøvere og trenere som hjelper videre forskning. For MS-caset har prosjektet synliggjort hvordan bruk av sensorteknologi kan kvantifisere bevegelse og funksjon. Dette vil redusere subjektiv fortolkning i vurderingen av hvert enkelt individ sin funksjon, og kan bidra til en bedre tilpasset og individualisert oppfølging. Prosjektet har også vist potensialet for å validere og eventuelt forbedre eksisterende kliniske tester. Kompetanseutviklingen gjennom utviklingen prosjektets analyseplattform, AutoActive Research Environment, har vært betydelig. I plattformen kan man innhente, sykronisere og visualisere data fra ulike sensorer. Dette har stor overføringsverdi til andre områder enn de to casene som har vært studert i prosjektet. Plattformen ligger åpent tilgjengelig og kan benyttes til analyse av bevegelsesdata og fysiologiske parametere innen en rekke andre anvendelser både i idrett og i helse- og rehabiliteringsfeltet. Det at plattformen ligger åpent tilgjengelig gir tilgang til andre forskningsmiljøer og mulighet for fremtidig samarbeid.

AutoActive is motivated by the need for better tools, methods and algorithms allowing extraction of reliable and useful information on human activity from heterogeneous sensor data. Present commercial systems can mainly give a direct mapping between a single measurement device and single parameters, for example GPS --> location and speed, and heart rate --> effort. However, to design and optimize such sensor solutions and algorithms is a long, empirical process that requires a broad range of competence. Access to this new level of information rely on - A physical and physiological understanding of the context and underlying processes, - A careful selection and combination of sensor devices - Data interpretation algorithms taking advantage of state-of-art data mining, machine learning and other multiparameter analysis methods. An open source software platform will be realised and used throughout the project to develop knowledge on how to collect and interpret data from multiple wearable sensor streams. General project results will be applied in two case studies to verify the project approach and methodology, as well as to demonstrate the potential of the technology. One case will be devoted to performance and technique assessment in sports, and one will be devoted to disease management for patients with multiple sclerosis. The project unites a multidisciplinary research team with partners from NTNU, Olympiatoppen, OUS, MS Senteret Hakadal, UiO, and SINTEF (leader), and will educate one PhD, at least 5 M.Sc and 2 part time Post.Docs.