Multi-Arm Payload Manipulation via Mixed Reality

Florian Kennel-Maushart, Roi Poranne, Stelian Coros

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Multi-Robot Systems (MRS) present many advantages over single robots, e.g. improved stability and payload capacity. Being able to operate or teleoperate these systems is therefore of high interest in industries such as construction or logistics. However, controlling the collective motion of a MRS can place a significant cognitive burden on the operator. We present a Mixed Reality (MR) control interface, which allows an operator to specify payload target poses for a MRS in real-time, while effectively keeping the system away from unfavorable configurations. To this end, we solve the inverse kinematics problem for each arm individually and leverage redundant degrees of freedom to optimize for a secondary objective. Using the manipulability index as a secondary objective in particular, allows us to significantly improve the tracking and singularity avoidance capabilities of our MRS in comparison to the unoptimized scenario. This enables more secure and intuitive teleoperation. We simulate and test our approach on different setups and over different input trajectories, and analyse the convergence properties of our method. Finally, we show that the method also works well when deployed on to a dual-arm ABB YuMi robot.

Original languageAmerican English
Title of host publication2022 IEEE International Conference on Robotics and Automation, ICRA 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages7
ISBN (Electronic)9781728196817
StatePublished - 2022
Event39th IEEE International Conference on Robotics and Automation, ICRA 2022 - Philadelphia, United States
Duration: 23 May 202227 May 2022

Publication series

NameProceedings - IEEE International Conference on Robotics and Automation


Conference39th IEEE International Conference on Robotics and Automation, ICRA 2022
Country/TerritoryUnited States

All Science Journal Classification (ASJC) codes

  • Software
  • Artificial Intelligence
  • Electrical and Electronic Engineering
  • Control and Systems Engineering


Dive into the research topics of 'Multi-Arm Payload Manipulation via Mixed Reality'. Together they form a unique fingerprint.

Cite this