If we are to develop robust robot-based automation, we need to develop solid visual and tactile based perception and thus equip robots with better perception and learning capabilities than they currently possess today.
Robots currently lack such visual and tactile perception skills and this is especially emphasized during manipulation of 3D compliant objects, which pose an additional challenge compared to manipulation of rigid objects.
The BIFROST project addresses these challenges by developing a visual-tactile perception, control, and learning framework in order to enable robots manipulate 3D compliant objects for a set of specific scenarios. By doing so, the project aims to generate new knowledge as a fundament for a future robot technology that may be capable to address in the future, e.g. challenging real-world robotic manipulation in domains characterized as critical society functions, such as those in food/seafood processing, involving compliant food object robotic manipulation.
Related to visual perception, we have been investigating 3D shape completion and reconstruction of volumetric objects, to enable a robot agent to make inference of the 3D objects shape during the manipulation stage when equipped with 3D vision. This work is a continuation of our preliminary work where we demonstrated a fast, accurate, GPU-enabled 3D reconstruction of volumetric objects based on high-resolution point clouds, which showed better accuracy and speed than the established real-time benchmarks.
Regarding Image-based tactile perception, we have been elaborating different concepts for gripper finger with image-based tactile sensors capable of characterizing semi-rigid and soft and objects when it comes to their compliancy. All elaborated prototype-designs combine high resolution tactile imaging and mount on different finger and gripper design configurations to facilitate dexterous grasping, while having the tactile sensing on in order to capture the deformation during manipulation. The research trials are focused on classification of objects based on the high-resolution tactile images acquired from grasping different object with different mechanical and texture properties and on testing different prototype mounts for tactile sensors on gripper fingers.
We have also designed and validated now what we call a general framework for grasping unknown objects by coupling deep reinforcement learning (DRL), Generative adversarial networks (GAN) and Visual Servoing (VS). The agent learns grasping in simulation environment using deep reinforcement learning, we used GAN for domain adaptation between the simulation and the real world, to bridge the sim-to-real gap, and we introduce image-based visual servoing to correct the grasping pose in the last phase prior to grasping the unknown objects – objects which the learned agent has never seen before neither in simulation, during training, nor the real world.
BIFROST involves the development of a novel visual-tactile perception and control framework for the advanced robotic manipulation of 3D compliant objects. The ability of robots to manipulate such objects remains key to the advancement of robotic manipulation technologies. Despite tremendous progress in robotic manipulation, modern robots are generally only capable of manipulating rigid objects. As with humans, in order to plan and perform complex manipulation tasks on 3D compliant objects, robots need to "see" and "feel by touch" the objects they manipulate, and to understand their shape, compliancy, environment and context. Current visual-only robotic manipulation suffers from an inability to perceive 3D information due to real-world physical occlusions and intra-object or self-occlusions.
BIFROST will enable the simultaneous perception of 3D object shapes and their compliancy by means of visual perception by an RGB-D sensor, augmented with touch through active exploration using an image-based tactile sensor for physically occluded objects, inaccessible to the visual sensor. Based on visual and tactile perception, BIFROST will achieve active manipulation and shape servoing of 3D compliant objects by which robot control tasks are generated in sensor space by mapping raw sensor observations to actions. In order to achieve the learning of complex manipulation tasks and active deformation, we will develop a high-level multi-step reasoning framework for the automatic selection of action types designed to achieve desired states and shapes.
As with the rainbow bridge of Norse mythology, BIFROST connects two worlds. Inspired by human innate perception, understanding, manipulation and learning, the aim is to develop similar capabilities in robots.
