Fig 3 - uploaded by Eiichi Yoshida
Content may be subject to copyright.
Source publication
This paper presents a novel method for retargeting human motions onto a humanoid robot. The method solves the following three simultaneous problems: the geometric parameter identification that morphs the human model to the robot model, motion planning for a robot, and the inverse kinematics of the human motion-capture data. Simultaneous solutions c...
Contexts in source publication
Context 1
... proposed method was tested with humanoid robot HRP- 4 [11]. An overview of the robot is shown on the left side of Fig. 3. Though the original body surface of HRP-4 is made of hard plastic covers, we replaced the covers with a soft suit in order to mimic a human body surface. The geometric properties of HRP-4 are also designed to be close to the measured average of humans. This similarity enables the robot to wear clothes or devices designed for humans. ...
Context 2
... the original body surface of HRP-4 is made of hard plastic covers, we replaced the covers with a soft suit in order to mimic a human body surface. The geometric properties of HRP-4 are also designed to be close to the measured average of humans. This similarity enables the robot to wear clothes or devices designed for humans. The right side of Fig. 3 shows the robot wearing an assistive device [8] for evaluating the device. Though the total DOF of the robot is originally 34 [11], our robot has a roll joint at the waist. The finger joints were fixed for the whole retargeting process; therefore, the total robot DOF for our experiments was 31. The placements of the joints that we use ...
Similar publications
Markerless motion capture and understanding of professional non-daily human movements is an important yet unsolved task, which suffers from complex motion patterns and severe self-occlusion, especially for the monocular setting. In this paper, we propose SportsCap—the first approach for simultaneously capturing 3D human motions and understanding fi...
Citations
... Literature [15] conceptualized a punishment mechanism combining recurrent neural networks with variable gain neural dynamics design (VG-NDD) theory as the underlying architecture, and the proposed motion trajectory planning method has higher computational accuracy and efficiency in practical feedback. Literature [16] designed a robot system to simulate human motion, which can combine human motion data and physical constraints to achieve an accurate simulation of human motion and can be applied in the industry of humanoid robot-assisted operation. ...
... .(16) is used to determine whether to use the "equipotential line tangent circle" escape method for path planning if it meets Eq.(16).The tangent point is solved as follows, assuming that the slope of the tangent line is k . The equation of the tangent line can be listed from the point-slope equation, as well as the center and radius of the circle as follows: is the obstacle coordinates. ...
... .(16) is used to determine whether to use the "equipotential line tangent circle" escape method for path planning if it meets Eq.(16).The tangent point is solved as follows, assuming that the slope of the tangent line is k . The equation of the tangent line can be listed from the point-slope equation, as well as the center and radius of the circle as follows: is the obstacle coordinates. ...
In order for a robot to complete a given task, it must first be made to autonomously reach a specified target location, so optimizing the robot path trajectory planning is a prerequisite for the use of robots. In this paper, for the two problems of the traditional artificial potential field method of target unreachable and local optimization, we first improve the repulsive potential field function so that the robot’s gravitational force and repulsive force are zero at the target position and then construct the robot kinematic analytical model by setting the virtual target point away from the local minimum value point. On this basis, the improved algorithm is used to compare simulation experiments in three environments with the real trajectory planning test in the showroom. In the climate “near the obstacle of the target point” and the pure U-shaped area environment, the robot of the traditional APF algorithm cannot reach the target point. However, the algorithm in this paper reaches the target point in all three environments, with a time taken of only 21, 33, and 42 seconds, respectively. In the real trajectory planning of the showroom, the improved algorithm in this paper reaches the target point quickly and accurately, with a total path trajectory length of 77.7835m, a time of 43 seconds, and a total turning angle of 793°. This paper provides an effective method for planning robot path trajectories in a complex and variable environment.
... Additionally, we used retargeted Torobo motions and source human motions reported by Kang et al. [18] for evaluation. This method retargeted human motions from the KIT bimanual manipulation dataset [19] to Torobo using an IK-based method [20]. ...
This study proposes an approach to human-to-humanoid teleoperation using GAN-based online motion retargeting, which obviates the need for the construction of pairwise datasets to identify the relationship between the human and the humanoid kinematics. Consequently, it can be anticipated that our proposed teleoperation system will reduce the complexity and setup requirements typically associated with humanoid controllers, thereby facilitating the development of more accessible and intuitive teleoperation systems for users without robotics knowledge. The experiments demonstrated the efficacy of the proposed method in retargeting a range of upper-body human motions to humanoid, including a body jab motion and a basketball shoot motion. Moreover, the human-in-the-loop teleoperation performance was evaluated by measuring the end-effector position errors between the human and the retargeted humanoid motions. The results demonstrated that the error was comparable to those of conventional motion retargeting methods that require pairwise motion datasets. Finally, a box pick-and-place task was conducted to demonstrate the usability of the developed humanoid teleoperation system.
... Model-based control methods rely on the mathematical model of the mechanical structure to design an optimal tracking controller. Due to the inherent complexity of mathematical models of robots, this control strategy is mainly adopted in more simplified robotic mechanisms [7][8][9][10][11][12]. On the other hand, datadriven approaches are suitable for multilegged robots with different morphology [13,14]. ...
... A study by [8] introduced dynamic filters in the retargeting process of the whole-body data collected from 17 inertial sensors to prevent motions that could cause the humanoid robot to fall. Other studies by [9,12] solved an optimization problem and used inverse kinematics strategies to track human-motion-capture data using biped robots. Ref. [12] developed a whole-body teleoperation architecture that enables intuitive and natural interaction between the human operator and the humanoid robot at the configuration space level, whereas [9] divided the retargeting problem into morphing the human model to the robot model, motion planning for the robot, and then inverse kinematics of human-motion-capture data. ...
... Other studies by [9,12] solved an optimization problem and used inverse kinematics strategies to track human-motion-capture data using biped robots. Ref. [12] developed a whole-body teleoperation architecture that enables intuitive and natural interaction between the human operator and the humanoid robot at the configuration space level, whereas [9] divided the retargeting problem into morphing the human model to the robot model, motion planning for the robot, and then inverse kinematics of human-motion-capture data. Unlike the previously mentioned studies, ref. [11] developed a differentiable optimal control (DOC) framework that interfaces with various sources of motion data and allows for the retargeting of motions from animals or animations onto quadruped robots with significantly different proportions and mass distributions. ...
In this study, we introduce a novel framework that combines human motion parameterization from a single inertial sensor, motion synthesis from these parameters, and biped robot motion control using the synthesized motion. This framework applies advanced deep learning methods to data obtained from an IMU attached to a human subject’s pelvis. This minimalistic sensor setup simplifies the data collection process, overcoming price and complexity challenges related to multi-sensor systems. We employed a Bi-LSTM encoder to estimate key human motion parameters: walking velocity and gait phase from the IMU sensor. This step is followed by a feedforward motion generator-decoder network that accurately produces lower limb joint angles and displacement corresponding to these parameters. Additionally, our method also introduces a Fourier series-based approach to generate these key motion parameters solely from user commands, specifically walking speed and gait period. Hence, the decoder can receive inputs either from the encoder or directly from the Fourier series parameter generator. The output of the decoder network is then utilized as a reference motion for the walking control of a biped robot, employing a constraint-consistent inverse dynamics control algorithm. This framework facilitates biped robot motion planning based on data from either a single inertial sensor or two user commands. The proposed method was validated through robot simulations in the MuJoco physics engine environment. The motion controller achieved an error of ≤5° in tracking the joint angles demonstrating the effectiveness of the proposed framework. This was accomplished using minimal sensor data or few user commands, marking a promising foundation for robotic control and human–robot interaction.
... The dynamic implications of ZMP could be expressed as the ground reaction forces that can be balanced with gravity and inertial forces of the robot when using ZMP as the action point. The ZMP stability criterion can be described as that when ZMP is maintained within the support domain, the bipedal robot can maintain stability, where the support domain represents the minimum polygonal domain that can accommodate all contact points between the sole of robot and the ground [56]. The descriptions of ZMP is defined as follows: ...
Bipedal gait control has always been a very challenging issue due to the multi‐joint and non‐linear structure of humanoid robots and frequent robot–environment interactions. To realise stable and robust bipedal walking, many aspects including robot modelling, gait stability and environmental adaptivity should be considered to design the gait control method. In this paper, a general description of bipedal gait and the corresponding evaluation indicators are introduced. Moreover, the existing bipedal gait control methods are classified into model‐based gait, stability criterion‐based gait and learning strategy‐based gait and a comprehensive review is conducted. Finally, the existing challenges and development trends of bipedal gait control are presented.
... To enhance the similarity of the robot motion to the human one, the retargeting of the elbow motion with a lower priority in the optimization problem is suggested in [69]. To overcome the manual morphing problem, Ayusawa and Yoshida [70] suggested solving an optimization problem to identify geometric parameters of the morphing function. ...
Teleoperation of humanoid robots enables the integration of the cognitive skills and domain expertise of humans with the physical capabilities of humanoid robots. The operational versatility of humanoid robots makes them the ideal platform for a wide range of applications when teleoperating in a remote environment. However, the complexity of humanoid robots imposes challenges for teleoperation, particularly in unstructured dynamic environments with limited communication. Many advancements have been achieved in the last decades in this area, but a comprehensive overview is still missing. This survey article gives an extensive overview of humanoid robot teleoperation, presenting the general architecture of a teleoperation system and analyzing the different components. We also discuss different aspects of the topic, including technological and methodological advances, as well as potential applications.
... For example, Learning-from-observation (LFO) converts MoCap motion sequences into reference motion primitives that can be tracked by a Zero Moment Point (ZMP) controller, which enables a humanoid robot to graciously perform traditional Japanese dance routines [54]. Ayusawa and Yoshida [3] proposed a strategy to refine motion primitives to create more dynamically realistic movements. ...
We present SLoMo: a first-of-its-kind framework for transferring skilled motions from casually captured "in the wild" video footage of humans and animals to legged robots. SLoMo works in three stages: 1) synthesize a physically plausible reconstructed key-point trajectory from monocular videos; 2) optimize a dynamically feasible reference trajectory for the robot offline that includes body and foot motion, as well as contact sequences that closely tracks the key points; 3) track the reference trajectory online using a general-purpose model-predictive controller on robot hardware. Traditional motion imitation for legged motor skills often requires expert animators, collaborative demonstrations, and/or expensive motion capture equipment, all of which limits scalability. Instead, SLoMo only relies on easy-to-obtain monocular video footage, readily available in online repositories such as YouTube. It converts videos into motion primitives that can be executed reliably by real-world robots. We demonstrate our approach by transferring the motions of cats, dogs, and humans to example robots including a quadruped (on hardware) and a humanoid (in simulation). To the best knowledge of the authors, this is the first attempt at a general-purpose motion transfer framework that imitates animal and human motions on legged robots directly from casual videos without artificial markers or labels.
... To enhance the similarity of the robot motion to the human one, the retargeting of the elbow motion with a lower priority in the optimization problem is suggested in [69]. To overcome the manual morphing problem, [70] suggested solving an optimization problem to identify geometric parameters of the morphing function. ...
Teleoperation of humanoid robots enables the integration of the cognitive skills and domain expertise of humans with the physical capabilities of humanoid robots. The operational versatility of humanoid robots makes them the ideal platform for a wide range of applications when teleoperating in a remote environment. However, the complexity of humanoid robots imposes challenges for teleoperation, particularly in unstructured dynamic environments with limited communication. Many advancements have been achieved in the last decades in this area, but a comprehensive overview is still missing. This survey paper gives an extensive overview of humanoid robot teleoperation, presenting the general architecture of a teleoperation system and analyzing the different components. We also discuss different aspects of the topic, including technological and methodological advances, as well as potential applications. A web-based version of the paper can be found at https://humanoid-teleoperation.github.io/.
... Frontiers in Robotics and AI frontiersin.org through data augmentation using motion retargeting (Di Fava et al., 2016;Ayusawa and Yoshida, 2017) between different bodies and also dynamic simulations under different conditions. This network will be exploited for symbolization presented in the following section. ...
This article presents perspective on the research challenge of understanding and synthesizing anthropomorphic whole-body contact motions through a platform called “interactive cyber-physical human (iCPH)” for data collection and augmentation. The iCPH platform combines humanoid robots as “physical twins” of human and “digital twins” that simulates humans and robots in cyber-space. Several critical research topics are introduced to address this challenge by leveraging the advanced model-based analysis together with data-driven learning to exploit collected data from the integrated platform of iCPH. Definition of general description is identified as the first topic as a common basis of contact motions compatible to both humans and humanoids. Then, we set continual learning of a feasible contact motion network as the second challenge by benefiting from model-based approach and machine learning bridged by the efficient analytical gradient computation developed by the author and his collaborators. The final target is to establish a high-level symbolic system allowing automatic understanding and generation of contact motions in unexperienced environments. The proposed approaches are still under investigation, and the author expects that this article triggers discussions and further collaborations from different research communities, including robotics, artificial intelligence, neuroscience, and biomechanics.
... In this work, the morphing policy is found manually offline. To overcome the manual morphing problem, Ayusawa et al. [27] suggested solving simultaneously the problems of geometric parameter identification of the morphing function and the inverse kinematics. ...
This thesis aims to investigate systems and tools for teleoperating a humanoid robot. Robotteleoperation is crucial to send and control robots in environments that are dangerous or inaccessiblefor humans (e.g., disaster response scenarios, contaminated environments, or extraterrestrialsites). The term teleoperation most commonly refers to direct and continuous control of a robot.In this case, the human operator guides the motion of the robot with her/his own physical motionor through some physical input device. One of the main challenges is to control the robot in a waythat guarantees its dynamical balance while trying to follow the human references. In addition,the human operator needs some feedback about the state of the robot and its work site through remotesensors in order to comprehend the situation or feel physically present at the site, producingeffective robot behaviors. Complications arise when the communication network is non-ideal. Inthis case the commands from human to robot together with the feedback from robot to human canbe delayed. These delays can be very disturbing for the human operator, who cannot teleoperatetheir robot avatar in an effective way.Another crucial point to consider when setting up a teleoperation system is the large numberof parameters that have to be tuned to effectively control the teleoperated robots. Machinelearning approaches and stochastic optimizers can be used to automate the learning of some of theparameters.In this thesis, we proposed a teleoperation system that has been tested on the humanoid robotiCub. We used an inertial-technology-based motion capture suit as input device to control thehumanoid and a virtual reality headset connected to the robot cameras to get some visual feedback.We first translated the human movements into equivalent robot ones by developping a motionretargeting approach that achieves human-likeness while trying to ensure the feasibility of thetransferred motion. We then implemented a whole-body controller to enable the robot to trackthe retargeted human motion. The controller has been later optimized in simulation to achieve agood tracking of the whole-body reference movements, by recurring to a multi-objective stochasticoptimizer, which allowed us to find robust solutions working on the real robot in few trials.To teleoperate walking motions, we implemented a higher-level teleoperation mode in whichthe user can use a joystick to send reference commands to the robot. We integrated this setting inthe teleoperation system, which allows the user to switch between the two different modes.A major problem preventing the deployment of such systems in real applications is the presenceof communication delays between the human input and the feedback from the robot: evena few hundred milliseconds of delay can irremediably disturb the operator, let alone a few seconds.To overcome these delays, we introduced a system in which a humanoid robot executescommands before it actually receives them, so that the visual feedback appears to be synchronizedto the operator, whereas the robot executed the commands in the past. To do so, the robot continuouslypredicts future commands by querying a machine learning model that is trained on pasttrajectories and conditioned on the last received commands.
... Wang et al. presented a generative framework for motion retargeting using a single depth sensor [23]. Ayusawa et al. introduced a motion retargeting method for humanoid robots based on simultaneous morphing parameter identification and motion optimization [4]. Choi et al. designed a motion retargeting pipeline composing of motion retarget descriptions, optimization, inverse kinematics and trajectories post-processing [7]. ...
Motion retargeting from human demonstration to robot is an effective way to reduce the professional requirements and workload of robot programming, but faces the challenges resulting from the differences between human and robot. Traditional optimization-based methods is time-consuming and rely heavily on good initialization, while recent studies using feedforward neural networks suffer from poor generalization to unseen motions. Moreover, they neglect the topological information in human skeletons and robot structures. In this paper, we propose a novel neural latent optimization approach to address these problems. Latent optimization utilizes a decoder to establish a mapping between the latent space and the robot motion space. Afterward, the retargeting results that satisfy robot constraints can be obtained by searching for the optimal latent vector. Alongside with latent optimization, neural initialization exploits an encoder to provide a better initialization for faster and better convergence of optimization. Both the human skeleton and the robot structure are modeled as graphs to make better use of topological information. We perform experiments on retargeting Chinese sign language, which involves two arms and two hands, with additional requirements on the relative relationships among joints. Experiments include retargeting various human demonstrations to YuMi, NAO and Pepper in the simulation environment and to YuMi in the real-world environment. Both efficiency and accuracy of the proposed method are verified.
