Fig 3 - uploaded by Rüdiger Dillmann
Content may be subject to copyright.
Source publication
This paper deals with easy programming methods of dual-arm manipulation tasks for humanoid robots. Hereby a programming by demonstration system is used in order to observe, learn and generalize tasks performed by humans. A classification for dual-arm manipulations is introduced, enabling a segmentation of tasks into adequate subtasks. Further it is...
Context in source publication
Context 1
... dual arm manipulation by demonstration requires a classification of coordinated actions in order to enable the system to fulfill a reasonable segmentation of the observed task. Following the conclusions of [12] two arm manipulations can be classified according to fig. 3 in coordinated and uncoordinated ...
Similar publications
Robots that share their workspace with people, like household or service robots, need to take into account the presence of humans when planning their actions. In this paper, we present a framework for human-aware planning that would make the robots capable of performing their tasks without interfering with the user in his every day life. We focus i...
Citations
... However, these methods often fall short in dynamic or intricate environments as they lack the flexibility required for adaptive task execution. In contrast, learning-based strategies, such as Reinforcement Learning (RL) and Imitation Learning (IL), provide more adaptive solutions by enabling robots to learn control policies from either human-designed rewards [5,6] or human demonstrations [7,8]. However, these methods struggle to generalize to zero-shot scenarios, where robots need to execute tasks without prior specific training. ...
... node_7: name: put "red_bell_pepper" onto "cutting_board" arm_num: 1 edge: [6] type: release description: Place the red bell pepper onto the cutting board to prepare for cutting. node_8: name: cut "red_bell_pepper" with "knife" arm_num: 1 edge: [2,7] type: tool use description: Cut the red bell pepper using the knife while it is on the cutting board. node_9: name: put "knife" onto "cutting_board" arm_num: 1 edge: [5,8] type: release description: Place the knife back onto the cutting board after use. ...
Dual-arm robots offer enhanced versatility and efficiency over single-arm counterparts by enabling concurrent manipulation of multiple objects or cooperative execution of tasks using both arms. However, effectively coordinating the two arms for complex long-horizon tasks remains a significant challenge. Existing task planning methods predominantly focus on single-arm robots or rely on predefined bimanual operations, failing to fully leverage the capabilities of dual-arm systems. To address this limitation, we introduce DAG-Plan, a structured task planning framework tailored for dual-arm robots. DAG-Plan harnesses large language models (LLMs) to decompose intricate tasks into actionable sub-tasks represented as nodes within a directed acyclic graph (DAG). Critically, DAG-Plan dynamically assigns these sub-tasks to the appropriate arm based on real-time environmental observations, enabling parallel and adaptive execution. We evaluate DAG-Plan on the novel Dual-Arm Kitchen Benchmark, comprising 9 sequential tasks with 78 sub-tasks and 26 objects. Extensive experiments demonstrate the superiority of DAG-Plan over directly using LLM to generate plans, achieving nearly 50% higher efficiency compared to the single-arm task planning baseline and nearly double the success rate of the dual-arm task planning baseline.
... However, these movement primitives greatly limit the tasks achievable by the method as they often require costly demonstrations or labor-intensive hard-coded motions for each task-specific primitive. Additionally, learning from demonstrations in bimanual settings is difficult as teleoperating two high-degree-freedom robots or collecting kinesthetic demonstrations on both arms simultaneously is challenging and sometimes impossible for a single human and may require specialized hardware [16,11,31,12,8,13]. Recent works have demonstrated more effective interfaces for data collection in a bimanual setting, but these interfaces are limited to specific hardware instantiations and would still require large amounts of expert data to learn a high dimensional policy [14]. ...
Key to rich, dexterous manipulation in the real world is the ability to coordinate control across two hands. However, while the promise afforded by bimanual robotic systems is immense, constructing control policies for dual arm autonomous systems brings inherent difficulties. One such difficulty is the high-dimensionality of the bimanual action space, which adds complexity to both model-based and data-driven methods. We counteract this challenge by drawing inspiration from humans to propose a novel role assignment framework: a stabilizing arm holds an object in place to simplify the environment while an acting arm executes the task. We instantiate this framework with BimanUal Dexterity from Stabilization (BUDS), which uses a learned restabilizing classifier to alternate between updating a learned stabilization position to keep the environment unchanged, and accomplishing the task with an acting policy learned from demonstrations. We evaluate BUDS on four bimanual tasks of varying complexities on real-world robots, such as zipping jackets and cutting vegetables. Given only 20 demonstrations, BUDS achieves 76.9% task success across our task suite, and generalizes to out-of-distribution objects within a class with a 52.7% success rate. BUDS is 56.0% more successful than an unstructured baseline that instead learns a BC stabilizing policy due to the precision required of these complex tasks. Supplementary material and videos can be found at https://sites.google.com/view/stabilizetoact .
... Workpiece alignment includes tasks found in industry assembly of which task such as peg-in-hole can be improved by means of compliance. Articulated motion involves opening task such as jar [15] and valve turnings [16], [17]. Table 1 shows the review summary of common application carried out by contact manipulation. ...
Robots play an important role all over the world in the industrial field in terms of dealing with various materials, performing duties quickly and high accuracy as well as in the pharmaceutical/chemical field since robots can handle some issues better than human. Robotic manipulation is the term used to describe how robots deal with many objects. This article reviews the contact and non-contact manipulation with their most common robotic applications and focusing on the non-contact manipulation since it can do specific tasks which contact manipulation cannot do. Reviewing of the robotic non-contact manipulation which include the metal-based robotic manipulation such as magnetic and electrostatic levitation and the non-metal-based robotic manipulation method such as aerodynamic and acoustic levitation have been made in this article. The main core of this article is acoustic levitation which considered one of the most important levitation and manipulation method due to the recent advancements that made it proper to deal with any kind of materials. Here, we review the renowned three methods of acoustic levitation that already using in robotic non-contact manipulation which are standing wave, near-field and single-beam acoustic levitation. Then, we present a performance characteristic of the reviewed methods and the summary of controlling strategies, followed by conclusion and future recommendations for acoustic levitation methods as a non-contact manipulation.
... Uncoordinated motions are independent motions without considering other arms. In this case, each arm is programmed without considering the other, such as executing independent tasks in sequential order [5], [6]. Coordinated-motion tasks are those in which two or more arms need to be synchronized with each other to complete complex tasks [5], [6]. ...
... In this case, each arm is programmed without considering the other, such as executing independent tasks in sequential order [5], [6]. Coordinated-motion tasks are those in which two or more arms need to be synchronized with each other to complete complex tasks [5], [6]. Coordinated-motion tasks are also divided into symmetrical and asymmetrical tasks. ...
... Time synchronization errors could be due to: 1) different robot controllers, 2) different kinematic algorithms, 3) processing time differences, etc. The performance of symmetrical coordinated-motion tasks has been evaluated in various ways: 1) software simulation [5], [6], and 2) independent external sensors, such as laser trackers [9], [10], telescoping ball bars [11], 12], and motion-tracking systems [13], [14]. A laser tracker interferometer consists of a laser interferometer, a gimbal system, and a reflector. ...
Most existing robot performance evaluation methods focus on single robotic arms performing independent motion tasks. In this paper, a motion gauge is proposed to evaluate the symmetrical coordinated-motion performance between two robotic arms. For this evaluation, the proposed device monitors the relative distance between the two robotic arms in real-time, which is used to evaluate the coordinated-motion errors with respect to accuracy, and repeatability between the two arms. The proposed metrology device is composed of two linear displacement sensors sliding on a linear rail, two ball-and-socket magnetic couplers for mounting to robotic arms, and a wireless communication module for data transmission. For validation, the proposed system monitored the two robotic arms programmed to simulate symmetrical coordinated motions.
... Many tasks where manipulation in contact is required also involve tools. They can either be rigidly attached to the robot arm or can be grasped by the robot for use; in this survey we do not differentiate between these cases, except by noting that grasping a tool always creates uncertainty regarding Wiping or polishing [18,19,20,21,22,23,24,25,26,27] Grinding or similar [28,29,30,31,32,33,34,35,36,37] Scooping [38,39] Peg-in-hole variants [40,41,42,43,44,45,17,46,47,48,49,50] [51,52,53,54,55,56,57,58,59,60,61,62,63,64] [ 65,66,67,68,69,45,70,71,72,73,74] Articulated motions [75,76,77,78,79,80,81,82,83,84,85,86,87] Hydraulic [88,89,90,91,92,93,94] ...
... The majority of everyday articulated objects that afford single arm manipulations can be modeled as one degree of freedom mechanisms (doors, drawers, handles) even though doors with handles can be seen as a combined two degrees of freedom mechanism. Also many opening tasks, such as jars [78] and valve turnings [79,80], are articulated motions. ...
... Considering industrial assembly tasks, [131] did a thorough analysis of industrial assembly tasks occurring within Toyota car manufacturing, but did not report cases where simultaneous motions of two arms were required; thus, the second arm in many tasks acts as a fixture. However, in a classic articulated rotation tasks such as opening a jar, rotating a pepper mill or just manually tightening nuts and bolts [78,132,133] the task may be performed faster if both arms move in harmony, reducing the number of required grip changes. Also in tasks such as melon scooping while one arm holds and the other one scoops [38], drawing on a board held by another robot [108] or peg-in-hole where also the "hole" is held [43,46], it can be advantageous if also the holder arm can explicit compliance, even if no active motions are performed; this is also how a human performs such tasks [43]. ...
In this survey we present the current status on robots performing manipulation tasks that require varying contact with the environment, such that the robot must either implicitly or explicitly control the contact force with the environment to complete the task. Robots can perform more and more manipulation tasks that are still done by humans, and there is a growing number of publications on the topics of 1) performing tasks that always require contact and 2) mitigating uncertainty by leveraging the environment in tasks that, under perfect information, could be performed without contact. The recent trends have seen robots perform tasks earlier left for humans, such as massage, and in the classical tasks, such as peg-in-hole, there is more efficient generalization to other similar tasks, better error tolerance, and faster planning or learning of the tasks. Thus, in this survey we cover the current stage of robots performing such tasks, starting from surveying all the different in-contact tasks robots can perform, observing how these tasks are controlled and represented, and finally presenting the learning and planning of the skills required to complete these tasks.
... Cela fait environ 40 ans que le besoin d'un robot à deux bras capable de travailler en coordination pour manipuler (Figure (a) 1.5), assembler des pièces (Figure (b) 1.5), et réaliser des compétences humaines souvent effectuées avec deux bras, a été décrit [80]. Zollner dans [153] distingue deux types de tâches coordonnées de manipulation. Le premier type est représenté par la manipulation d'un seul objet par plusieurs bras appelé coordination symétrique (Comanipulation). ...
Cette thèse a pour objectif principal de développer une loi de commande pour contrôler un robot à deux bras afin de remplir les tâches de manipulation et de garantir les aspects de sécurité pour l'homme et le robot. Pour y parvenir, nous avons présenté un aperçu des études qui ont été menées dans ce cadre. Ensuite, nous avons procédé à la modélisation cinématique géométrique et dynamique du robot. Nous avons utilisé le logiciel SYMORO+ pour calculer son modèle dynamique. Au terme d'une présentation détaillée de la méthode d'identification des paramètres des robots manipulateurs, nous avons appliqué celle-ci de manière concrète sur notre propre robot. Ceci nous a permis d'obtenir un vecteur de paramètres capable de garantir une matrice d'inertie définie positive pour toute configuration d'articulation du robot, tout en assurant une bonne qualité de reconstruction du couple pour des vitesses articulaires à la fois constantes et variables. Grâce à cette identification dynamique, nous avons pu réaliser un simulateur précis du robot sur Matlab/Simulink. Les forces externes du robot ont été estimées à partir du module dynamique identifié et une validation expérimentale a été décrite. Pour faire appliquer des lois de commande sur un robot industriel, il a été nécessaire de réfléchir à une communication externe sur le robot. L'approche adoptée a été présentée ainsi que sa réalisation sur le robot. Cette dernière, nous a permis de programmer en python toutes nos problématiques à mettre en œuvre sur le robot. Après avoir validé toutes les étapes préalablement citées nous sommes partis vers l'implémentation de la commande sur le robot. La tâche principale que nous avons adoptée est le dénudage des câbles. A cet effet, nous avons choisi de réaliser une commande hybride force/position dans laquelle les forces externes sont contrôlées afin de garantir l'absence de dommages sur les câbles à dénuder. Le robot qui a servi à réaliser toutes les expériences est le robot IRB14000 YuMi de Abb. Toutes les expériences réalisées dans le cadre de ce projet ont été décrites et présentées. Ce travail de thèse a été réalisé dans le cadre du projet Robotix Academy financé par le fonds européen de développement régional INTERREG V-A Grande Région.
... The final modes are triggered by the task planner as soon as the task interpreter has correctly recognized the command coming from the dialog manager. At present, the robotic system learns tasks and flows of actions in an off-line manner by programming by demonstration or by teleoperation (Robler & Hanebeck, 2004;Zollner, Asfour, & Dillmann, 2004). ...
Cognitive computing is the use of computerized models to simulate the
human thought process in complex situations where the answers may be
ambiguous and uncertain. Using self-learning algorithms that use data mining, pattern recognition, and natural language processing, the computer can
mimic the way the human brain works. So it can be incorporated with
Human�Robot Interaction (HRI) for assisting humans in various forms.
The study of interactions between humans and robots is thus fundamental
to ensure the development of robotics and to devise robots capable of
socially interacting intuitively and easily through speech, gestures, and facial
expressions. HRI is an integrative field that has provided considerable
improvement in various applications, including human�computer interaction
(HCI), robotics and artificial intelligence, the service robots combined with
cognitive computing to design intelligent robotic services. This robotics provides novel services in terms of companionship for the elderly and disabled
and does household chores in the home environment. It lays at the crossroad
of many subdomains of AI and, for effect, it calls for their integration:
modeling humans and human cognition; acquiring, representing, manipulating in a tractable way abstract knowledge at the human level; reasoning on
this knowledge to make decisions; and eventually instantiating those decisions into physical actions both legible to and in coordination with humans.
Unique features of Cognitive Computing for Human�Robot Interaction:
Principles and Practices
... In recent years, researchers applied learning based approach to bimanual manipulation. using imitation learning from demonstrations [62,17,54,60] and reinforcement learning [30,1,8,10,18]. For example, Amadio et al. [1] proposed to leverage probabilistic movement primitives from human demonstrations. ...
We address the problem of solving complex bimanual robot manipulation tasks on multiple objects with sparse rewards. Such complex tasks can be decomposed into sub-tasks that are accomplishable by different robots concurrently or sequentially for better efficiency. While previous reinforcement learning approaches primarily focus on modeling the compositionality of sub-tasks, two fundamental issues are largely ignored particularly when learning cooperative strategies for two robots: (i) domination, i.e., one robot may try to solve a task by itself and leaves the other idle; (ii) conflict, i.e., one robot can easily interrupt another's workspace when executing different sub-tasks simultaneously. To tackle these two issues, we propose a novel technique called disentangled attention, which provides an intrinsic regularization for two robots to focus on separate sub-tasks and objects. We evaluate our method on four bimanual manipulation tasks. Experimental results show that our proposed intrinsic regularization successfully avoids domination and reduces conflicts for the policies, which leads to significantly more effective cooperative strategies than all the baselines. Our project page with videos is at https://mehooz.github.io/bimanual-attention.
... As seen in [24], dual arm manipulation tasks can be classified into coordinated (both end effectors should work together by taking into account the feedback from the other in a closed loop) or uncoordinated (open loop). Additionally, while performing coordinated tasks, the movements of the arms can be symmetric or asymmetric. ...
Automating the action of finding the opening side of a box is not possible if the robot is not capable of reaching and evaluating all of its sides. To achieve this goal, in this paper, three different movement strategies to bi-manipulate a box are studied: overturning, lifting, and spinning it over a surface. First of all, the dynamics involved in each of the three movement strategies are studied using physics equations. Then, a set of experiments are conducted to determine if the real response of the humanoid robot, TEO, to a box is consistent with the expected answer based on theoretical calculus. After the dynamics validation, the information on the forces and the position in the end effectors is used to characterize these movements and create its primitives. These primitive movements will be used in the future to design a hybrid position–force control in order to adapt the movements to different kinds of boxes. The structure of this control is also presented in this paper.
... The upper level generates an optimal sequence of motion for user's requirements using timed Petri nets. In [4], an easy programming methods of dual arm manipulation tasks for humanoid robots have been used. It is shown how the generated programs are mapped on and executed by a humanoid robot. ...
In this study, Petri net model is developed for controlling a mobile robot that could be applied in library environment to help organizing books to its belonging bookshelf. The study includes creating an algorithm using Matlab coding based on Petri net mathematical modelling. The approach starts with creating a Petri net model to find next directions of the robot. These directions are embedded in the Petri net model and there is a highest probability of reaching a set of target areas. The graphical representation of Petri net helped to model the system mathematically and thus helped to generate a code depending on the input data (path data). The path data is given to the system in a separate file containing a matrix with a given size (representing the path) where a non-zero number represents the presence of path. The created software uses Matlab code as a sensor; called path sensor, which consists of row and column information properly combined with the Petri net model. The coding system, which executes path sensor mission, divides the path into [3x3] sized matrices, each of which is analyzed by the system and give the next direction as an output. The software outcome returns the row and column subscripts of each non-zero item and is used to find the next direction coordinates. This study, also shows the realization of three robots’ coding system. Each robot possesses a codingbased sensor to follow a path despite crossing paths by setting priorities to robots for crash avoidance in Matlab coding simulation. The robots are performing 8 different tasks that is a total of following 8 different paths. The robot application is suggested to be at library where the three robots have the principle task of transporting books from a starting point to the assigned bookshelf, having 8 possible paths (each path leading to a unique bookshelf type) to follow in order to reach the aim.
