Autonomy through stereo vision near the seashore

According to the traffic-rules at sea, all vessels shall at all times maintain a proper lookout by sight and other means. For autonomous ships, this poses a challenge because human vision is a highly refined instrument, having evolved over hundreds of millions of years, whose capabilities are unlikely to be surpassed by artfificial vision. Nevertheless, autonomous navigation systems can offer several advantages that surpass the capabilities of a human operator, such as persistent attention, fusion of vision with active sensors such as radar or lidar, and utmost precision in tracking, navigation and motion control. To strengthen the utility of computer vision in maritime surface autonomy, the Autosight project focuses on stereo vision. In 2023 and 2024, the Autosight project has used the autonomous passenger ferry milliAmpere 2 to record several datasets of relevance to the project's objectives. The project has addressed a number of issues of relevance for surface autonomy. We have developed a dual baseline solution for stereo vision, which makes it possible to combine the advantages of short and long baselines. We have studied maritime target tracking in stereo data, and compared this with target tracking via fusion of stereo and lidar. We have studied the use of optical flow and scene flow in the maritime environment, both via stereo and fusion of camera and lidar. We have used such flow techniques to improve the initialization of tracks in a target tracking system, and to enable better velocity estimation for extended object tracking. We have developed techniques for water segmentation in stereo data, using both machine learning and spatial probability models in combination. We have developed solutions to estimate free space on the water via stereo vision, and we have developed new representations of the autonomous vessel’s surroundings based on stereo data.

The project will establish knowledge about how stereo vision can be used to enable precise and safe operations for autonomous surface vessels in the close vicinity of the shore and other vessels, such as during docking operations of an autonomous ferry. Building upon recent breakthroughs of maritime autonomy and inspired by the advances in vision-based automotive autonomy, the project will critically compare and combine solutions based on both probabilistic models and machine learning. The project will develop tailor-made solutions to localization and extended object tracking in the harbor environment and demonstrate how these solutions can be used to strengthen the safety of an autonomous ferry.