Back to search

FRINATEK-Fri prosj.st. mat.,naturv.,tek

BIFROST - A Visual-Tactile Perception and Control Framework for Advanced Manipulation of 3D Compliant Objects

Alternative title: BIFROST - Et Visuell-Taktil persepsjons- og styringsrammeverk for avansert manipulasjon av føyelige objekter

Awarded: NOK 11.9 mill.

If we are to develop robust robot-based automation, we need to develop solid visual and tactile-based perception and thus equip robots with better perception and learning capabilities than they currently possess today. Robots currently lack such visual and tactile perception skills and this is especially emphasized during manipulation of 3D compliant objects, which pose an additional challenge compared to manipulation of rigid objects. The BIFROST project addresses these challenges by developing a visual-tactile perception, control, and learning framework in order to enable robots to manipulate 3D-compliant objects for a set of specific scenarios. By doing so, the project aims to generate new knowledge as a fundament for a future robot technology that may be capable to address in the future, e.g. challenging real-world robotic manipulation in domains characterized as critical society functions, such as those in food/seafood processing, involving compliant food object robotic manipulation. Regarding grasping we have finalized our 4DoF-grasping framework which combines Deep Reinforcement Learning, GAN, and IBVS in a single controller scheme. The framework is validated on four benchmarks, including one benchmark for deformable objects, and the framework achieves a remarkable closed-loop grasping success rate of over 90%. This makes our framework one of the grasping frameworks with the highest grasping success rates. Additionally, we have also developed a 6DoF grasping framework, based on reinforcement learning to be able to grasp challenging 6DoF objects that cannot be grasped with a 4DoF Scheme. Concerning advanced manipulation of deformable objects, we have developed a deep reinforcement learning approach to bring a specific object from the initial to the target shape as a result of prehensile move actions on the object. The results are promising for equipping robots with finer manipulation skills to be able to manipulate deformable objects without compromising their integrity. Regarding the advanced manipulation of deformable objects, we have additionally developed a control approach that relies on a coarse model of the soft object to be manipulated. This model is composed by a 3D mesh and we chose to represent the mechanical behavior of the object using a mass-spring model (MSM) because it provides real-time capability. Based on this coarse model, we derived the analytical expression of the controller that allows to indirectly move a feature point belonging to the soft object to a desired 3D position by acting with a robotic manipulator on a distant manipulated contact point. Since the MSM provides an approximation of the object behavior, which in practice may lead to a drift between the real object and its model, an online realignment of the model was performed by visually tracking the object's 3D deformation. In terms of image-based tactile perception and learning, we have developed an approach that from tactile perception is able to discern material properties, recognize textures, and determine softness, while with compliance, we are able to securely and safely interact with the objects and the environment around us. Combining these two abilities has culminated in a new and useful soft robotic gripper that is able to grasp a large variety of different objects and also perform simple manipulation tasks.

BIFROST involves the development of a novel visual-tactile perception and control framework for the advanced robotic manipulation of 3D compliant objects. The ability of robots to manipulate such objects remains key to the advancement of robotic manipulation technologies. Despite tremendous progress in robotic manipulation, modern robots are generally only capable of manipulating rigid objects. As with humans, in order to plan and perform complex manipulation tasks on 3D compliant objects, robots need to "see" and "feel by touch" the objects they manipulate, and to understand their shape, compliancy, environment and context. Current visual-only robotic manipulation suffers from an inability to perceive 3D information due to real-world physical occlusions and intra-object or self-occlusions. BIFROST will enable the simultaneous perception of 3D object shapes and their compliancy by means of visual perception by an RGB-D sensor, augmented with touch through active exploration using an image-based tactile sensor for physically occluded objects, inaccessible to the visual sensor. Based on visual and tactile perception, BIFROST will achieve active manipulation and shape servoing of 3D compliant objects by which robot control tasks are generated in sensor space by mapping raw sensor observations to actions. In order to achieve the learning of complex manipulation tasks and active deformation, we will develop a high-level multi-step reasoning framework for the automatic selection of action types designed to achieve desired states and shapes. As with the rainbow bridge of Norse mythology, BIFROST connects two worlds. Inspired by human innate perception, understanding, manipulation and learning, the aim is to develop similar capabilities in robots.

Funding scheme:

FRINATEK-Fri prosj.st. mat.,naturv.,tek