The NewbornTime project was about improving newborn care by using artificial intelligence (AI) for activity and event recognition in video from the time during and after birth. Read about the project at www.uis.no/newborntime
Deprivation of oxygen to an infant during and after birth might lead to birth asphyxia, one of the leading causes of newborn deaths, cerebral palsy and neurodevelopmental impairment. According to guidelines, a newborn in need of help to start breathing should be resuscitated immediately after birth. Resuscitation activities include stimulating, clearing airways, and performing bag-mask-ventilation. In Norway, approximately 10% of term infants need stimulation and around 3% need bag-mask ventilation.
The NewbornTime project focused on generating an accurate timeline describing key events and activities performed on newborns. Such timelines can be used to evaluate compliance to guidelines and identify successful resuscitation activity patterns. It can further be useful in a de-briefing and quality improvement tool, and potentially real-time decision support in the future.
Infrared (IR) thermal video was collected from 8 different labor rooms as well as the operating theatre for C-sections at Stavanger University hospital (SUS). In addition, RGB (visible-light) video from newborn resuscitation stations were collected when newborns received resuscitation treatment. Artificial intelligence models, in the form of deep neural networks, were developed and trained for the precise detection of the time-of-birth (ToB) based on the thermal labor-videos, and for different resuscitation activities from the visible-light videos. The final dataset collected from SUS, as part of NewbornTime, includes 1012 thermal videos from labor and 274 visible-light videos from stabilization/ resuscitation. Among these, 239 newborns are represented in both video types. For the development of robust models, that could be adapted to different hospital settings, we included treatment videos from collaborative projects and simulations.
In the final phase of the project, the developed AI models were deployed as part of the NewbornTimeLine pilot, operating locally on secure, closed servers at SUS. This setup enabled the application of AI algorithms directly on sensitive clinical data, eliminating the need to transfer data from its original storage location and thereby enhancing data privacy and compliance.
The results from the project include the following: A digital consent and data collection system featuring dashboards for full oversight. A pilot consisting of two main parts: 1) NewbornTimeLine Activity Recognition - AI models for activity recognition from visible-light videos and models for ToB detection from thermal video with high accuracy. 2) NewbornTimeLine Dashboard - a web app and database that supports further research, efficient querying, and the generation of both individual and aggregated group statistics. One PhD has been completed within the NewbornTime project, and two early-stage career postdoctoral researchers have worked on the project in addition to two master's thesis students. Ten scientific papers have been published so far, with multiple additional papers currently in the pipeline.
Find an updated publication list at www.uis.no/newborntime
The project was a collaboration between the University of Stavanger (UiS), Stavanger University Hospital (SUS), Laerdal Medical and BitUnitor. UiS, SUS and Laerdal Medical have long experience in collaborative research on newborn care. The data collection was carried out at SUS, in addition to medical research studying the videos both manually and by looking at the generated timelines. BitUnitor developed a digital consent solution that was connected to a semi-automated GDPR-compliant data collection system and data platform developed by Laerdal Medical. UiS has been developing AI methods for event detection and activity recognition in video and led the project.
Outcomes and results:
- Unique dataset of videos from birth and newborn resuscitation (privat)
- AI models for time-of-birth detection from thermal video with (above) human level precision
- AI models for activity recognition during newborn resuscitation events from RGB video, focused on main stabilization and resuscitation activities with reasonable precision.
- Timeline generation from post-processed AI model outputs, conveniently stored in a database format
- Web based user interface for query and research on the timeline database
- Secure digital consent management system with dashboard giving full overview of consent collection status
- Semi-automated secure and GDPR compliant data collection system with dashboard giving overview of video collection status
- Pilot implementation of AI models running on edge producing timelines locally
- 10 scientific publications, 2 conference papers published not yet publicly available, several papers in the process
Impact:
- Publication of AI model architectures, preprocessing schemes for thermal videos, and adaptations necessary can be useful for other applications using video and thermal video, for automated event and activity recognition.
- The ToB AI model is highly promising and will be available through licensing for commercial companies and for research partners.
- Automated timelines from the AI models will be further explored in the Safer Births Scandinavia+ project
- Automated timelines can do data-driven objective auto-documentation of resuscitation events that can be taken into use at hospitals and additionally make data-collection in research more automated and scalable.
- Automated timelines, as well as all the knowledge and experience gained in working with video data insensitive time-critical situation, will be taken into use in data-driven debriefing in the Safer Births Scandinavia+ project, and can additionally be used for debriefing after simulation training
- Medical researchers plan to explore the timeline database and video dataset to gain more knowledge on newborn resuscitation, research on compliance with guidelines etc. Such type of evidence-based research on newborn resuscitation is highly sought for by international organizations like ILCOR.
- Potential medical impact of improved practices around newborn resuscitation is earlier onset of bag-mask ventilation which can save 1 life pr 1000 births in low resource setting
- Potential medical impact of improved practices around newborn resuscitation is reduced costs for the Norwegian government, who compensates families with 80mill NOK yearly due to death or sequelae of newborns, extendable to other countries.
Birth asphyxia is a primary cause of death in newborns as well as the main cause of cerebral palsy and other development disorders in children, and immediate resuscitation of the newborn is crucial to reduce the risk. The NewbornTime project will provide a tool for quality improvement of newborn resuscitation both at a macro level, challenging current guidelines, and at a micro level providing a debriefing and quality improvement tool. Ultimately, this can have a significant impact in reducing long-term damage and save lives.
The project aims to utilize video recordings from births and newborn resuscitation situations to develop an Artificial Intelligence (AI) based system, NewbornTimeline, for automatic timeline generation of time of birth as well as potential resuscitation activities like ventilation, stimulation, suction, and the number of health care providers involved. The system input will be based on thermal video from the delivery room and RGB (+ thermal) video from the resuscitation table.
NewbornTime will use thermal cameras in the delivery room and develop video processing algorithms to accurately detect the time of birth. Potential obstacles between newborns and cameras require multiple thermal cameras and real-world data to develop robust algorithms.
The project aims to explore semi-supervised learning of Deep Neural Networks (DNNs) for partly un-labeled and untrimmed videos for activity recognition during newborn resuscitation, as labeling of activities will be both time consuming and have privacy issues. There will also be a focus on solutions that can adapt to on-site environments, since variations between hospitals in different countries can be extensive. NewbornTime will develop a GDPR compliant and secure digital patient consent handling system and cloud-based storage for sensitive data. Such solutions can be transferred to other areas of video activity recognition on sensitive data for medical situations, as well as other non-sensitive data.
