The NewbornTime project is about improved newborn care by using artificial intelligence (AI) for activity and event recognition in video from the time during and after birth.
Deprivation of oxygen to an infant during and after birth might lead to birth asphyxia, one of the leading causes of newborn deaths, cerebral palsy and other long-term damage. According to guidelines, a newborn in need of help to start breathing should be resuscitated immediately after birth. Resuscitation activities include stimulating, clearing airways, and perform bag-mask-ventilation. In Norway, approximately 10% of term infants need stimulation and around 3% need bag-mask ventilation.
NewbornTime will produce a timeline describing events and activities performed on a newborn. Accurate time of birth will be detected using AI models from thermal videos collected in the delivery room. Activity recognition will be performed using AI in the form of deep convolutional neural networks (CNN) on thermal and RGB video from the resuscitation. The system will be designed to recognize multiple time-overlapping activities. Care will be given to make the AI models robust, reliable, general, and adaptive to be able to use it at different hospitals and settings. The timelines will be used to evaluate compliance to guidelines and identify successful resuscitation activity patterns. It can further be useful in a de-briefing and quality improvement tool.
The project is a collaboration between University of Stavanger (UiS), Stavanger University Hospital (SUS), Laerdal Medical and BitYoga. UiS, SUS and Laerdal has long experience in collaborative research on newborn care. They have documented promising results on detecting activities using resuscitation videos from a hospital in Tanzania. In NewbornTime the data collection will be performed at SUS. BitYoga and Laerdal will ensure smart GDPR compliant data-contracts and data-platforms. UiS will develop site-adaptive AI methods for activity recognition in video.
NewbornTime was officially launched in March 2021. The following results have been achieved by 30th November 2021:
Preparation and information:
- The project has received approval from REK (Regional ethical committee) and NSD (Norwegian center for research data).
- Data processing agreements between partners and subcontractors have been signed
- Information material for participants and employees is designed and includes information video, emails, websites for participants in Norwegian and English as well as a website with project information ( www.uis.no/newborntime )
- A user-group has been created and the first meeting completed
- Several interviews about the project have been given to the mass-media
- PhD research fellow, Jorge García-Torres Fernandez, employed starting 09.08.2021, PhD project title: «Event and activity recognition from thermal and optical videos during birth and newborn resuscitation»
- Postdoc candidate has signed contract and expected start-up is in January 2022.
Development of technical solutions:
- Video recording system is implemented at SUS. Thermal cameras are installed at 4 delivery rooms as well as the operating room for caesarean section, two cameras in each room. Thermal cameras are also installed in two resuscitation rooms, where optical cameras are located above the resuscitation stations.
- First version of the digital consent solution has been implemented
- Cloud-based data storage system is implemented with the following features:
a) Security by VPN and data encryption
b) Automatic collection of consent information and study ID from digital consent solution
c) Data is processed and uploaded in cloud storage in de-identified form after 48 hours only if consent has been given.
- Some simulation data has been collected in autumn 2021
- Collection of consent started 15.11.2021
- Data collection started on 15.11.2021.
Birth asphyxia is a primary cause of death in newborns as well as the main cause of cerebral palsy and other development disorders in children, and immediate resuscitation of the newborn is crucial to reduce the risk. The NewbornTime project will provide a tool for quality improvement of newborn resuscitation both at a macro level, challenging current guidelines, and at a micro level providing a debriefing and quality improvement tool. Ultimately, this can have a significant impact in reducing long-term damage and save lives.
The project aims to utilize video recordings from births and newborn resuscitation situations to develop an Artificial Intelligence (AI) based system, NewbornTimeline, for automatic timeline generation of time of birth as well as potential resuscitation activities like ventilation, stimulation, suction, and the number of health care providers involved. The system input will be based on thermal video from the delivery room and RGB (+ thermal) video from the resuscitation table.
NewbornTime will use thermal cameras in the delivery room and develop video processing algorithms to accurately detect the time of birth. Potential obstacles between newborns and cameras require multiple thermal cameras and real-world data to develop robust algorithms.
The project aims to explore semi-supervised learning of Deep Neural Networks (DNNs) for partly un-labeled and untrimmed videos for activity recognition during newborn resuscitation, as labeling of activities will be both time consuming and have privacy issues. There will also be a focus on solutions that can adapt to on-site environments, since variations between hospitals in different countries can be extensive. NewbornTime will develop a GDPR compliant and secure digital patient consent handling system and cloud-based storage for sensitive data. Such solutions can be transferred to other areas of video activity recognition on sensitive data for medical situations, as well as other non-sensitive data.