Human-Robot Interaction with Pose Estimation and Dual-Arm Manipulation Using Artificial Intelligence

Introduction

In the last few decades, a lot of research has been performed in the development of robots for Search and Rescue Operations. A majority of this research focuses on the search aspect and has achieved a significant success; however, research on the rescue aspect, especially when direct human-robot interaction is involved, is still at its initial stages. Based on the analysis of the previous state-of-the-art research in robots with victim extraction functions, a novel rescue robot design is proposed: Semi-Autonomous Victim Extraction Robot (SAVER).

Summary of Contributions

Fig. 1 Proposed Victim Extraction Procedure using SAVER.

For this ongoing project at RMLab, my part is to propose an autonomous system to safely manipulate the victim pose during extraction to the SAVER as shown in Fig. 1. My contributions in autonomous victim extraction using artificial intelligence can be summarized as follows:

1. Human pose estimation , performing real-time and accurate human pose estimation.
2. Safe and cooperative robotic manipulations , obtaining fast training process of the cooperative robotic manipulations with safety constraints.
3. Robotic manipulator modeling , efficiently learning robotic model using limited-size real-world data.
4. Integrated system , combining all the above techniques and is experimentally validated using a full-size manikin.

Overall System Architecture

Fig. 2 Overall system architecture.

The integrated sensing system, installed on top of the SAVER will capture the RGB Depth information of the surroundings. To improve the computation efficiency of human pose estimation, only RGB images are used to detect the human pose. Once a human is detected in the surroundings, the depth information is then integrated to localize the human joint positions in the 3D coordinates. The human pose is then fed into the decision-making neural network to generate trajectories to safely manipulate the human pose. The desired trajectories in the task-space are then mapped into joint-space for the low-level motor controllers to directly control robotic manipulators joints.

Related Publication:

1. [J1] Ren, H., Ben-Tzvi, P., "Learning Inverse Kinematics and Dynamics of a Robotic Manipulator using Generative Adversarial Networks", Robotics and Autonomous Systems, vol. 124, p. 103386, Feb 2020.
2. [J2] Ren, H., Ben-Tzvi, P., "Advising Reinforcement Learning Agents Towards Scaling in Continuous Control Environments with Sparse Rewards", Engineering Applications of Artificial Intelligence, vol. 90, p.103515, Apr 2020.
3. [P1] Ren, H., Ben-Tzvi, P., Dissertation Defense Presentation