July 20, 2023 | Breakthrough in robotics research with adaptive AI

Robot grips intuitively and moves objects like a human

  • Agile Justin can perceive objects and coordinates the twelve joints in its hand to manipulate them dexterously.
  • The humanoid robot trains itself independently in a simulation.
  • Impressive developments in intelligent robotics.
  • Focus: Robotics, spaceflight, Industry 4.0, digitalisation

Agile Justin can grasp objects correctly – even if it does not know them yet. The humanoid robot can also turn these objects in its hand in any desired direction – without looking. This is commonplace for humans, but extremely difficult for robots. A team from the German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt; DLR) has now trained a robot to acquire these highly complex skills. To do this, the researchers used learning artificial intelligence (AI) and have achieved a breakthrough in AI for robotics.

Agile Justin has sensors in its two, four-fingered robot hands. It uses them to 'feel' objects. The robot recently demonstrated its skills at the automatica trade fair. Robotics researchers consider dexterity to be a major challenge. "Using new methods of learning artificial intelligence, we have succeeded for the first time in demonstrating abilities with the highly developed mechatronic DLR hand that come close to human dexterity. In particular, rotating and manipulating objects without camera inputs and with an hand pointing downward was a challenge," says Berthold Bäuml from the DLR Institute of Robotics and Mechatronics in Oberpfaffenhofen. Bäuml's AI research has been advancing the development of the humanoid robot Justin for many years. Justin catches balls thrown to him, for example, or recognises materials by sensing them with its fingers. Now, learning AI has significantly expanded Justin's capabilities.

"Turning the object is good; dropping it is bad"

The researchers used what is referred to as deep reinforcement learning. This means that Justin learns completely on its own, only by being told what is correct or incorrect. In this application, for example, it was instructed that 'turning the object towards the target is good', while 'dropping the object is bad'. Machine learning of skills takes only two to three hours in a simulation; then Justin masters the task. It develops the movement strategy for the fingers in a similar way to a human, through trial and error. "The robot can then directly execute its solutions in the real environment. Even we as researchers are often surprised. The possibilities of our new learning AI methods are fascinating," says Berthold Bäuml.

Thousands of examples for tens of thousands of things

The researchers also achieved another first in AI robotics. Justin is able to grasp arbitrary objects with its multi-fingered hands. For this, the researchers solved two major problems using AI. Firstly, using 50,000 examples, the robot learned what is referred to as a 'foundational model', which allows it to infer the shape of the back of an object from the visible front. Justin has thus acquired an understanding of three-dimensional shapes. Secondly, Justin can then find an optimal position for its hand and adjust its twelve finger joints appropriately. Humans grasp objects intuitively. In order to convey this intuition to the robot, another AI system was fed with thousands of examples for tens of thousands of objects. "The computations took a few days. But now the robot can recall the correct grip within fractions of a second," explains Bäuml. At automatica, Justin demonstrated its dexterity with objects handed to him by the visitors to the exhibition.

"The work on AI-based dexterous manipulation represents a worldwide scientific breakthrough. Never before have hands been able to manipulate such a large variety of objects so quickly and so precisely. This success was only possible through the combined use of the latest AI methods with model-based approaches and many years of robotics experience," says Alin Albu-Schäffer, Director of the DLR Institute of Robotics and Mechatronics. "Until now, it was often assumed that you needed a very large amount of data to learn dexterous object handling with robotic hands. In contrast to data for large language models, experimental data in robotics, even if generated by simulation, is very costly. The DLR research team has found a way to require less data by using more robotics knowledge and a new learning architecture. Using their new approach, the team can even train these models on a single computer and does not require a large computing cluster."

Possibilities for industrial production and healthcare

The next big step in robotics could be in industrial manufacturing. Multi-fingered robotic hands could assemble very complex and small parts if they master flexible, standardised intelligent manipulation. DLR researchers are working on this in the 'SmartHand' project, for example.

Robotic assistants are also an important topic in healthcare. In the future, they could assist caregivers and give people with reduced abilities back a higher degree of independence in everyday life. The intelligent manipulation skills developed at DLR are also fundamental to this. These skills apply, for example, to the safe grasping cups, bowls, boxes or other diversely shaped household objects.

About the DLR Institute of Robotics and Mechatronics

The DLR Institute of Robotics and Mechatronics in Oberpfaffenhofen develops robots that enable people to interact more effectively, efficiently and safely with the environment. These robots are intended to work in environments that are inaccessible or dangerous for humans. However, they are also intended to support and assist humans during work and in everyday life.

Berthold Bäuml's research group within the Institute relies on Deep Learning as the key principle for designing autonomous robots. They are intended to come close to human capabilities by combining sensor technology and motor skills with learning AI. The highly qualified members of the young AI research team come mainly from the Technical University of Munich, where Berthold Bäuml holds a lectureship. There are also research collaborations with renowned technology companies.


Katja Lenz

German Aerospace Center (DLR)
Corporate Communications
Linder Höhe, 51147 Cologne
Tel: +49 2203 601-5401

Lioba Suchenwirth

Public Relations
German Aerospace Center (DLR)
Institute of Robotics and Mechatronics
Münchener Straße 20, 82234 Oberpfaffenhofen-Weßling

Berthold Bäuml

German Aerospace Center (DLR)
Institute of Robotics and Mechatronics
Autonomy and Teleoperation
Münchener Straße 20, 82234 Oberpfaffenhofen-Weßling