Back to search

IKTPLUSS-IKT og digital innovasjon

GentleMAN-Gentle and Advanced Robotic Manipulation of 3D Compliant Objects

Alternative title: GentleMAN-Skånsom og Avansert Robotisert Manipulasjon av 3D Føyelige Objekter

Awarded: NOK 16.0 mill.

Humans are naturally equipped with fantastic visual and tactile capabilities, outstanding manipulative dexterity, and an ability rapidly to learn new tasks in a way that continues to defy our understanding. We learn new and complex manipulation tasks by combining our senses with previous experience, and by learning from demonstration and self-exploration. If we want robots to perform complex manipulation tasks, we need to develop a new technology that equips them with vision to 'see' the scene and objects to be manipulated, tactile sensing to enable them to feel the objects, and a 'brain' that combines these senses to achieve new learning. The GentleMAN project addresses these challenges by equipping robots with advanced 3D vision, tactile sensing, and a 'brain' that uses artificial intelligence to enable them to learn new manipulation skills that reproduce human-like dexterity and fine motor skills. The 'brain' we are developing, using robot learning, will provide robots with new capabilities, enabling them to perform complex manipulation tasks working alongside humans. Research of the GentleMAN project focuses on the manipulation of compliant objects because of the challenges such objects generate in terms of effective manipulation. The increased use of robots for such manipulation tasks will be essential to the green transition and to sustainable future growth in the food, agriculture, ocean space and manufacturing sectors. Robots require artificial vision, and to date in this project we have met this need by equipping robots with 3D vision that rapidly generates (in real-time and GPU-enabled) an accurate, high resolution 3D reconstruction of the scene and the objects subject to manipulation. We have developed a reconstruction methodology that builds a 3D model of the compliant object using a 3D sensor mounted on a robot arm that moves along robot motion trajectories. This enables rapid capture of the high resolution object geometries that are essential to the selection of grasping and manipulation strategies. This work has resulted in a peer-reviewed journal paper. Inspired by the nimble tactile senses employed by humans when we grasp and manipulate objects, and our ability to learn from demonstration, we have developed a haptic-based, robot 'brain' based on learning from demonstration and tactile signals. This enables the robot gripper to be trained using an anthropomorphic robot system. By combining data from teleoperation with a haptic interface and human-based learning during the grasping process, the new ?brain? enables the robot autonomously to grasp new compliant objects with high levels of accuracy. The system consistently outperforms other recently-published baselines by a considerable margin. The work has also resulted in a peer-reviewed journal paper. The ability to master a given task by means of self-exploration is unique to human beings, as is the way we generalise acquired knowledge to solve similar tasks in similar contexts. The robot 'brain' being developed in the GentleMAN project has also achieved task mastery by learning by self-exploration and uses this knowledge to manipulate previously unseen objects. Learning takes places in a simulation environment that considerably shortens the learning process. Simulation also prevents the robot from exhibiting possible unwanted hazardous behaviour that may have negative consequences if learning took place in the physical world. Of particular interest here is that even if the robot has never previously seen a given object, either in simulation or the physical world, the 'brain' nevertheless enables the perfect grasping of such objects. The 'brain' is thus able satisfactorily to generalise when the robot is set to perform similar grasping and manipulation tasks on previously unseen objects. This work has been published in the proceedings of the ICRA-IEEE Conference in Robotics and Automation. To the best of our knowledge, this work is the first in the world to link Deep Reinforcement Learning (DRL) and Generative Adversarial Networks (GANs) for transfer learning of a robotic grasping agent in connection with previously unseen objects. In the next phase of the GentleMAN project, our work will focus on the development of new and advanced 3D vision and tactile sensing systems. The 'brain' will be supplemented with new learning models to enable it to handle the manipulation of compliant objects during interaction and under deformation, and to facilitate the collaborative operation of several robots for complex manipulation tasks.

GentleMAN will result in a novel robot control and learning framework enabling real-world manipulation of 3D compliant objects. This framework will be based on visual and force/tactile sensing modalities and multi-modal learning models by careful balance and tighter integration between the components responsible for object localization and pose estimation, based on visual information, and the one responsible for manipulation based on the force/tactile information. The robotic manipulation of 3D compliant objects remains a key, yet relatively poorly-researched, field of study. Currently, most approaches to robotic manipulation focus on rigid objects. These are primarily vision-based and require a 3D model of the object or attempt to build one. The interaction of a robot with 3D compliant objects is one of the greatest challenges facing robotics today, due to complex aspects such as shape deformation during manipulation, and the need for real-time perception of deformation and the compliancy of the objects. To these are added coordination of the visual, force and tactile sensing required to control and accomplish specific manipulation tasks. These challenges become even more acute if the objects are slippery, made of soft tissue, or have irregular 3D shapes. Such objects are common in the agriculture, manufacturing, food, ocean space, health and other sectors in both industrial and non-industrial settings. The GentleMAN addresses these challenges by focusing on providing robots with advanced manipulation skills that reproduce human-like movements and fine motor skills. Robots will thus learn intelligently how to induce and apply the necessary manipulative forces while generating occlusion-resilient vision control, real-time 3D deformation tracking and a shape-servoing strategy. A highly qualified and expert interdisciplinary consortium, consisting of SINTEF, NTNU, NMBU, INRIA, MIT and QUT has been assembled to conduct the proposed research.

Publications from Cristin

No publications found


IKTPLUSS-IKT og digital innovasjon