Crossing to Mars: ISS robotics experiment Surface Avatar successfully completed

"The Surface Avatar experiment is a milestone in the collaboration between humans and robots in space. We have thus achieved all the technical prerequisites for controlling complex robotic missions on Mars, including towards a future permanent lunar research station," explains Prof Alin Albu-Schäffer, Director of the DLR Institute of Robotics and Mechatronics.

The core of the technology demonstration is to enable a person to command and deploy a team of different robots without intensive training. The Surface Avatar series of experiments launched in 2022 is being led by the Institute of Robotics and Mechatronics and is being carried out in collaboration with the European Space Agency (ESA) and the German Space Operations Centre (GSOC).

"Through the experiments with the astronaut in microgravity and the connection via relay satellites, we have mastered the technical hurdles in the remote control of robots. We are therefore confident that we will also be able to manage future scenarios, be it lunar gateways, lunar habitats or Earth to robots. With these experiments and technologies, we have built up unique expertise in Europe that is also useful for applications on Earth," says Thomas Krüger, Team Lead ESA Human-Robot Interaction Lab.

More robots, more difficult tasks

As a step up from the last Surface Avatar experiment in July 2024, not only have the tasks become more difficult and the robots more autonomous – another robot has also been added. In addition to the humanoid DLR robot Rollin' Justin, ESA rover Interact and the four-legged DLR robot Bert, the four-legged ESA robot Spot now completes the exploration team.

The Surface Avatar team demonstrated the robots' special teamwork and learning abilities in the next challenge: to explore a cave, Jonny Kim commanded the Interact rover, which in turn had loaded the robot Bert in its transport basket – the world's first robot-robot transport. Telecommanded by the astronaut in space, the rover travelled to the cave, carefully lifted the little quadruped out with its gripper arm and set it down on the ground.

But Bert had "injured" himself – one of his legs was stuck, so the four-legged friend was unable to explore the cave as intended. The experiment team on Earth had intended this malfunction so that the astronaut and robot would have to spontaneously solve an unforeseen problem – just like on a real space mission. To this end, Bert is equipped with "reinforcement learning", a machine learning method. In order to find a stable gait for Bert with three legs, Jonny Kim carried out a "training" programme: He had the robot try out different gaits and evaluated them until Bert had found a strategy that worked. After the successful learning session, the DLR robot completed its cave exploration. Here, the astronaut opted for manual control – Bert's camera eyes allowed him to see what the robot was seeing and control it remotely using the joystick on the Robot Command Terminal.

KI-supported preparation

The collaboration between NASA astronaut Jonny Kim on board the ISS and the robots on the simulated Mars habitat in Oberpfaffenhofen went so well that all tasks, including the additional options, were successfully completed in the tight time window of two and a half hours. For example, Jonny Kim steered the rover manually to pick up a sample container and place it in the transfer station. The US-American controlled the robots to drive around, look around and explore, touch and feel the environment. In an outstanding way, he used them as intelligent employees who planned and carried out mission tasks on site. They served as a physical extension of him from a distance – like avatars, in the spirit of the mission.

In preparation, the astronaut used the new AI assistant, the chatbot "Neal AI", which is based on a large language model (LLM) from the European provider Mistral AI. The development team at the Institute of Robotics and Mechatronics trained the chatbot specifically for the surface avatar mission. During the training phase, it answers Jonny Kim's questions about the functionalities of the robots, the operation of the control unit and the user interface. The idea behind this is that in future missions to Mars, the signal delay between Mars and Earth can be up to 40 minutes. During these communication breaks between the exploration team and the control centre, an AI assistant such as Neal AI could immediately help astronauts with questions.

Lace robotics from Germany and Europe

"Surface Avatar's final experiment was the perfect conclusion to the mission. It showed how robotics can support our astronauts in their exploration of space. Following this fruitful collaboration with ESA, I look forward to celebrating further joint successes in the future," summarises Principal Investigator Neal Y. Lii from the Institute of Robotics and Mechatronics summarises.

At the end of the Surface Avatar series, NASA astronaut Jonny Kim and DLR scientist Neal Lii shook hands in Oberpfaffenhofen – using the Rollin' Justin robot, which allowed both experiment partners to feel the force and movement of the handshake via force feedback. An emotional moment involving technology "made in Germany". Germany is a world leader in control technology to compensate for the time delay in space. So this tele-handshake was also a fitting tribute to the joint journey of DLR and ESA. Driven by developments in the field of artificial intelligence, robotics will continue to open up new dimensions for humans.

Further links