Back to search

IKTPLUSS-IKT og digital innovasjon

Neodroid - Creating a reality-ready robot brain in virtual reality

Alternative title: Neodroid - Utvikle en virkelighetsklar robothjerne i virtuell virkelighet (VR).

Awarded: NOK 8.0 mill.

Humans learn through a relatively slow process that unfolds over years, from the baby stage, through childhood and into adulthood. In Neodroid, we seek to develop a robot brain in an accelerated learning environment, so that a robot can learn in days or weeks what may take a human years to learn. The perfect accelerated learning environment is virtual reality (VR), where time can be sped up, and robots can move and learn faster and safer than in reality. The vision behind Neodroid is to teach robots how to perform visual-motor tasks, by training their brains in virtual reality. This is similar in philosophy to how Neo in the movie "The Matrix" learnt kung fu over the course of several hours within a simulator - in just a few seconds in the real world. Similarly, in Neodroid we will create a robot brain by training it - using deep learning algorithms - in virtual reality, much faster and safer than possible in the real world. The project has implemented a robot brain. The robot brain creates connections between what the 3D camera (robot eyes) sees and where the robot should place its gripper. The robot brain has been tested in VR, on a simple but difficult task; grasping virtual fish from a box. Testing continued with the robot brain in a real robot and with real fish. It worked surprisingly well. The robot managed to pick up real fish with a success rate of about 80%. This is surprising since the robot had never seen or touched a real fish before. The project is continued improving the robot brain in the areas were it made mistakes during the experiments on real fish. Specifically, this entails making it possible for the robot to think faster. Thinking in 3D takes a lot of computational power and there was a need to develop specific deep learning algorithms that can reduce the computational load, so that the robot can both think faster and perform tasks better. The improved robot brain was taught how to pick fish in an improved virtual environment, and then got to test its skills on real fish. The accuracy was almost perfect, and the computations were much quicker than before. Neodroid has shown that it is possible to create a robot brain in a virtual reality, capable of doing difficult tasks - in the real world - that require a combination of visual and motor skills.

Robothjernen sin AI-arkitektur forventes på sikt å ha betydning for forskningsfeltet, i at det muliggjør å løse komplekse oppgaver som krever en kombinasjon av visuelle og motoriske ferdigheter. Prosjektet har vært uvurderlig for kompetanseutvikling i SINTEF Ocean, innenfor AI og robotikk. Medarbeidere som har bygget kompetansen i prosjektet har allerede bidratt til å skape flere nye prosjekter med bakgrunn i den kompetansen, og økt det internasjonale samarbeid innenfor AI og robotikk. På sikt vil resultatene komme næringslivet til nytte, spesielt der hvor automasjon er for kostbart eller vanskelig å få til i dag. Dette gjelder spesielt innenfor bærekraftig utnyttelse av råvarer fra havet, som i dag ikke foredles fordi manuelt arbeid er for kostbart. Prosjektet har posisjonert SINTEF Ocean til å inneha den kunnskapen som er nødvendig for fremtidens bærekraftige automasjonssystemer basert på AI og robotikk.

The idea of Neodroid is to create a reality-ready robot brain in virtual reality (VR). We specifically focus on creating a robot brain capable of humanoid visual-motor ability. Visual-motor ability is the integration between visual perception and motor skills. More specifically, it is the ability to perform constructive tasks integrating both visual perception and motor skills. The motivation behind Neodroid is to enable robots to assist humans in performing such tasks. The primary objective of Neodroid is: - Develop deep learning architectures for creating a reality-ready robot brain in virtual reality. The secondary objectives are: - Implement a virtual reality environment for training a virtual robot. - Develop deep learning architectures for visual-motor tasks. - Implement human operation of a robot in virtual reality. - Demonstrate robot deep learning in virtual reality. - Demonstrate transfer-learning between virtual reality and the real world. The essence of Neodroid is a two-stage learning process in VR, followed by introduction to reality. The VR learning stages are first a 'baby' phase with environment-assisted learning, and a 'school' phase where a human enters VR to teach more complex tasks. After the school phase, the brain is transferred from the virtual to the real robot, so learnt visual-motor skills can be applied in the real world. The key challenges to enabling near-human level robot visual motor skill, using deep learning, are to generate a large set of training examples and to discover the best deep learning architectures. We believe that these challenges are best met by prototyping deep learning in VR, since this provides greater flexibility and a more cost-efficient and timely data acquisition and teaching process than using real robots and sensors. Once this architecture is found, it can be transferred to a real robot and refined.

Publications from Cristin

No publications found

Funding scheme:

IKTPLUSS-IKT og digital innovasjon