Back to search

IKTPLUSS-IKT og digital innovasjon

Ubiquitous cognitive computer vision for marine services

Alternative title: Marine tjenester basert på kognitivt datasyn

Awarded: NOK 15.5 mill.

Deep learning is a machine learning method that has revolutionised image analysis field during the last 10 years with accuracy approaching human performance. The methods data are driven and based on deep neural networks that learn the correspondence between data and content descriptions from examples. A key component for the success of deep learning is the availability of an extensive number of labelled examples, i.e., images that are marked with information on what they contain. In the COGMAR project, we have focused on analysis of complex image data, often with limited number of annotations. Vast amounts of marine data ranging from optical imagery and videos to acoustic surveys are today being collected. These data contain valuable information needed to ensure sustainable fisheries and harvest. In next generation marine services real-time analysis is anticipated and this will further increase the amount of data. The aim of the project is to contribute to the development of automatic solutions for extracting information from these big complex image data to bring exploitation of these data in the marine science to a new level, enabling extraction of new knowledge and continuous monitoring. We have developed several deep learning methods for analysis of different types of marine data: trawl camera, otolith images and acoustical data. For the trawl camera data, we have developed techniques that automatically detects each fish individual and classifies the species. Due to limited amount of training data, we have developed a simulation scheme that simulates fish in the trawl for a number of situations. These simulated data are used to train the network. The methodology for detecting and classifying fish from trawl camera, are planned to be included into Scantrol Deep Vision’s trawl camera system. For the otolith data, we have developed methods that automatically estimates the age of Greenland halibut individuals by analysing images of the otoliths. We have performed a study on how to understand how the network is reading the otolith images and found that the deep learning techniques reads the images completely different than humans. For the otolith data, we have found that the performance is worse with images from another laboratory. We have therefore developed methods that adapt new data sources to the network trained on Norwegian otolith images. The project results have also been included in the web portal DeepOtolith. For the acoustical data, we have developed methods to estimate the amount and fish species, with a particular emphasis on handling the data variation between different surveys and extended these to include context information such as depth and distance to seabed to increase performance. We have also developed methods that are able to exploit unannotated data for learning neural networks (so-called semi-supervised learning). The methodology for analysing acoustical data that have been developed in COGMAR have now been implemented in Kongsberg Maritime’s Blue Insight Platform.

Det faglige fotavtrykket fra prosjektet er stort, med blant annet 14 publikasjoner (og flere er i pipeline) i internasjonal vitenskapelige tidsskrift. Prosjektet har også medført at prosjektdeltakere har fått sentrale roller internasjonalt innen maskinlæring anvendt på marine data, blant annet i ICES-systemet. Prosjektet har styrket kompetansen til partnerne innen maskinlæring anvendt på marine data, samt stryket samarbeid mellom partnerne NR, HI og UiT. Dette har vært viktig for etableringen av SFIene CRIMAC og Visual Intelligence. CRIMAC og Visual Intelligence vil bidra til at resultater i COGMAR kan videreutvikles og dermed potensielt få enda større utnyttelse. Resultater fra prosjektet er og vil bli tatt i bruk av industrien. Kongsberg Maritime har implementert prosjektresultatene på analyse av akustiske data i sin Blue Insight plattform, og Scantrol Deep Vision planlegger å lansere Deep Vision plattformen for kommersielt fiskeri i løpet av 2022 og teste ut metodikken utviklet i COGMAR på dette systemet. Planen er at dette skal bli et kommersielt produkt for trålfiske, som kan bidra til identifisering av fangst og på den måten mer bærekraftig fiske. Havforskningsinstituttet vil benytte resultater fra prosjektet i sin satsing på autonome farkoster. Kongsberg Maritime leverer Sounder plattformen til HI i løpet av 2022. Sounder er en autonom overflatefarkost utrustet med ekkolodd til å overvåke fiskebestander. Planen er å analysere akustikkdata i nær sanntid ombord i plattformen.

Deep learning has been called the revolutionary technique that quietly changed machine vision forever, but is at present mainly applicable to standard RGB images of natural scenes or objects, or otherwise only for other types of imagery when a substantial amount of labelled data is available, which is seldom the case. This project aims at enabling this technology for computer vision problems anywhere, by developing easy-to-use cognitive solutions also for non-standard images and thereby extending the use of autonomous cognitive computer vision systems to new application areas. The methodology will be general and transferable to other domains such as medical imagery, remote sensing and various industrial applications. Within the project the aim is to solve key big data computer vision challenges in the marine sector. The overall concept of the project is to exploit the power of deep convolutional neural networks (CNNs) by developing new solutions for learning necessary to classify, localize and segment objects in non-standard, sparsely labelled, image data. Motivated by the methods' ability to generalize and the fact that unlabelled data is often inexpensive to acquire, our approach for solving this will be based on three main concepts; (i) cross-domain transfer learning, (ii) semi-supervised learning and (iii) data augmentation and simulation. Fisheries and aquaculture are major industries in Norway, and marine image data are acquired in a wide range of formats and modalities for various tasks. Automatic solutions for extracting information from these big non-standard image data will bring exploitation of these data in the marine science to a new level, enabling extraction of new knowledge and continuous monitoring of marine ecosystems. Solutions from the project will also contribute to innovation for industries manufacturing solutions for automated monitoring of fish and marine environments.

Publications from Cristin

No publications found

Funding scheme:

IKTPLUSS-IKT og digital innovasjon