Available via license: CC BY 4.0
Content may be subject to copyright.
Digital Twin Implementation of Autonomous Planning
Arc Welding Robot System
Xuewu Wang*, Yi Hua, Jin Gao, Zongjie Lin, and Rui Yu
Abstract: Industrial robots are currently applied for ship sub-assembly welding to replace welding workers
because of the intelligent production and cost savings. In order to improve the efficiency of the robot system, a
digital twin system of welding path planning for the arc welding robot in ship sub-assembly welding is proposed
in this manuscript to achieve autonomous planning and generation of the welding path. First, a five-dimensional
digital twin model of the dual arc welding robot system is constructed. Then, the system kinematics analysis
and calibration are studied for communication realization between the virtual and the actual system. Besides, a
topology consisting of three bounding volume hierarchies (BVH) trees is proposed to construct digital twin
virtual entities in this system. Based on this topology, algorithms for welding seam extraction and collision
detection are presented. Finally, the genetic algorithm and the RRT-Connect algorithm combined with region
partitioning (RRT-Connect-RP) are applied for the welding sequence global planning and local jump path
planning, respectively. The digital twin system and its path planning application are tested in the actual
application scenario. The results show that the system can not only simulate the actual welding operation of the
arc welding robot but also realize path planning and real-time control of the robot.
Key words: arc welding robot; digital twin; topology; path planning
1 Introduction
Welding robots have extensive use in production lines
of the automotive industry, the equipment
manufacturing industry, and other industries due to
their ability to reduce labor costs, shorten product
delivery cycles, and ensure the stability of welding
quality. Welding robots are required to not only meet
the basic action requirements but also consider the
welding accuracy and stability, seam tracking
capability, and anti-jamming capability in the welding
process. Traditional welding robots determine the
position of the welding seam and the welding path
through manual teaching, which relies heavily on the
engineer’s experience and lacks thorough consideration
of the above production requirements. Moreover,
traditional teaching methods require replanning
welding paths when handling workpieces with varying
geometric characteristics, which makes welding robots
less flexible. With the development of the Internet of
Things (IoT), digital twin technology has been widely
used in enterprises undergoing a digital transformation
as an effective method to address the challenges of
cyber-physical convergence in the manufacturing
industry[1]. In a digital twin system, virtual entities
(VE) created by digital technology can simulate the
behavior of physical entities (PE) on actual production
lines through real-time data. Additionally, the virtual
entities can generate solutions to drive the real physical
system through techniques such as dynamic trajectory
prediction approach[2] and iterative decision
• Xuewu Wang, Yi Hua, Jin Gao, and Zongjie Lin are with the
School of Information Science and Engineering, East China
University of Science and Technology, Shanghai 200237,
China. E-mail: wangxuew@ecust.edu.cn.
• Rui Yu is with the Institute for Sustainable Manufacturing, and
also with the Department of Electrical and Computer
Engineering, University of Kentucky, Lexington, KY 40506,
USA. E-mail: rui.yu@uky.edu.
* To whom correspondence should be addressed.
※This article was recommended by Associate Editor Jianhua Liu.
Manuscript received: 2023-03-22; revised: 2023-05-17;
accepted: 2023-05-24
COMPLEX SYSTEM MODELING AND SIMULATION
ISSN 2096-9929 05/06 pp 236−251
Volume 3, Number 3, September 2023
DOI: 10.23919/CSMS.2023.0013
© The author(s) 2023. The articles published in this open access journal are distributed under the terms of the
Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).
optimization under rule constraints. If the digital twin
technology is employed for welding operations,
collision-free optimal welding paths can be
autonomously planned and tested in virtual space. The
real welding system can also be controlled in real-time
using the digital twin system. Hence, digital twin
technology is the key to improving the reliability and
flexibility of welding.
This manuscript proposes a digital twin system of
welding path planning for the arc welding robot in ship
sub-assembly welding, referred to as “the dual arc
welding robot digital twin system” in the following
text. Due to the fact that the dual robot has a great
working space and can weld the workpiece
symmetrically, the heat produced by welding is evenly
distributed on the workpiece, and the welding
distortion is effectively reduced[3]. This manuscript
focuses on implementing a digital twin autonomous
welding path planning system for the arc welding
robot. In this digital twin system, welding seam
extraction, global welding sequence planning, and local
jump path planning are utilized to obtain a better
collision-free welding path for the workpiece. The
primary work and innovations of this study are
summarized as follows.
(1) A dual arc welding robot digital twin system
based on a five-dimensional digital twin model
proposed in Ref. [4] is designed to achieve control and
feedback between virtual and real twins.
(2) A topology for constructing virtual entities is
proposed. Based on this topology, a welding seam
extraction algorithm and collision detection algorithm
in virtual space are proposed.
(3) The genetic algorithm (GA) is applied for
planning global welding sequences, and the Rapidly-
exploring Random Tree (RRT) algorithm combined
with region partitioning is used to implement the local
jump path planning.
This paper is organized as follows. Section 2
describes the related works on robotic digital twin
systems, analyzes the maturity of these digital twin
systems, and presents two requirements that need to be
noticed while designing a complete digital twin system
for welding robots. Section 3 describes the components
of the proposed digital twin system. Section 4 describes
the calibration of the autonomous planning system for
the arc welding robot, which is a prerequisite for the
application of digital twin technology. Section 5
presents a new topology for constructing virtual entities
and designs methods for extracting welding seams and
collision detection based on this topology. Section 6
describes the algorithms for the global planning of
welding sequences and local jump paths. Section 7
displays the digital twin system architecture, welding
seam extraction results, and autonomously generated
welding paths. Finally, Section 8 concludes this paper
with the direction of future research.
2 Related Work
In recent years, digital twin technology has been
studied and applied in the fields of aerospace[5, 6], smart
cities[7], energy production[8], special equipment failure
prediction[9, 10], and biomedicine[11, 12]. The practical
applications of the digital twin in these fields are
becoming increasingly mature. In contrast, the digital
twin technology applied to robotics has certain
deficiencies in terms of virtual-real interaction, data
twin drive, and application services. One of the reasons
is that robotics is a complex system with multi-
disciplines, multi-physics, and multi-domain
characteristics, involving numerous components,
elements, and production line data in specific
application scenarios[13]. Tao et al.[1] established six
levels of digital twin maturity: simulating reality with
virtuality, reflecting reality with virtuality, controlling
reality with virtuality, anticipating reality with
virtuality, optimizing reality with virtuality, and
virtual-real symbiosis. Based on the above criterion,
this section provides an overview of research related to
the robotics digital twin.
It is inefficient and expensive for intelligent grasping
robots to mark a wealth of successful and unsuccessful
grasps in the real world. To address this problem, Hu
et al.[14] proposed digital twin technology to create
multiple virtual robots and conduct grasping
experiments in a simulated environment. This approach
enables the collection of a plethora of simulation data
through numerous grasping experiments on various
objects. As a supplement to the real dataset, the virtual
simulation dataset is input into the GGS-CNN network
for offline learning, which improves the accuracy of
grasping objects for the robot. The primary function of
this digital twin system is to generate simulation data,
achieving the first level of digital twin maturity as
defined by Tao et al.[1] The digital twin system[15]
synthesized all production data from the robotic
assembly lines to compensate for the real-world
uncertainty of the environment as well as human
XuewuWangetal.:DigitalTwinImplementationofAutonomousPlanningArcWeldingRobotSystem 237
behavior unpredictability. As a result, the need for
offline programming and validation of robot motions is
eliminated when there are changes in the
manufacturing process within this digital twin system.
Reflecting reality with virtuality is achieved in this
system. Wang et al.[16] analyzed the motion movement
of a curved-arm gantry robot and constructed a digital
gantry robot twin model. This twin-robot model can
control real gantry robots for sorting, loading, and
unloading tasks. This system reaches the level of
controlling reality with virtuality. Yan[17] established a
digital twin system for welding robots, which provides
a service for predicting actual welding quality. When
the working conditions change, the digital twin can be
used to select different welding parameters for testing
and quickly find a better welding solution. To a certain
extent, this system realizes anticipating reality with
virtuality. Hu[18] combined digital twin technology with
epistemological and information-theoretic methods in
the visual grasping robot system. The mutual
information generated by the visual model during the
perception of the real world can both reduce the
uncertainty of visual perception and enhance the
adaptation to the random posture of parts on the
conveyor as they enter the visual range. The robot
performs tasks autonomously and optimizes in real-
time. In a digital twin for geometry assurance, an
efficient method for optimizing spot welding sequence
was proposed[19]. The overall quality and efficiency of
the welding process are improved. To some extent,
these systems achieve optimizing reality with
virtuality. Virtual-real symbiosis, as the ideal target for
digital twins, provides accurate prediction, decision
making, and optimization services across the entire
lifecycle[1].
As mentioned above, most research on robotic digital
twin systems focuses on digital twin construction and
information interaction between physical entities and
virtual twins. However, cyber-physical fusion for
application services such as decision-making, control,
and optimization is insufficient. In addition, building a
cell-level digital twin of a single robot is insufficient to
meet the increasingly complex requirements of today’s
intelligent manufacturing industry. Based on the above
discussion, in order to develop a system-level dual arc
welding robot digital twin system that can guide
welding operations, several key requirements need to
be considered:
(1) The virtual space and the real space should
maintain a low latency data transmission and
continuous control within a permissible error.
(2) The parametric model which has detailed
geometric information should be utilized to perceive
the real space.
3 Dual Arc Welding Robot Digital Twin
System
The software development tool Open CASCADE[20] is
used to build the dual arc welding robot digital twin
system. A key feature of a mature robotic digital twin
is its ability to incorporate a multi-dimensional model.
In line with this, this manuscript constructs a five-
dimensional system-level digital twin model of the dual
arc welding robot, which is described as follows:
MDWR =(PEDWR,VEDWR,PSDWR,DDDWR ,CNDWR)(1)
PEDWR
VEDWR
PSDWR
DDDWR
PEDWR
VEDWR
CNDWR
where denotes the physical entities of the dual
arc welding robot, is the corresponding virtual
entities, is the integrated path planning service
of the dual arc welding robot system, consists
of various data of and , and is
the connection for data interaction among various
components of this system. The five-dimensional dual
arc welding robot digital twin system is shown in Fig. 1.
VEDWR
VEDWR
GC
PEDWR
PC
MC
BC
The is the mapping of physical entities in the
virtual world constructed by this system. The
consistency between virtual and physical entities plays
an important role in digital twin implementation[21]. As
shown in Eq. (2), should have a strictly
consistent geometric structure with and
maintain a consistent posture within a certain error.
The virtual robots and the external axes, as mobile
virtual twin units, should follow the same kinematic
constraints as the physical entities, as well as
adhere to the same boundary and collision constraints
.
VEDWR =(GC,PC,MC,BC)(2)
VEDWR
PEDWR
VEDWR
Under these constraints, is consistent with
on multiple time and spatial scales. Therefore,
can perform welding seam extraction and path
planning and then quickly check the rationality of the
generated results.
PSDWR
is composed of the welding seam extraction
module, motion analysis module, collision detection
module, path planning module, and 3D online
monitoring module. The welding seam extraction
module is used to generate welding seam information
and welding tasks. The motion analysis module is to
238 ComplexSystemModelingandSimulation,September2023, 3(3): 236−251
create a reasonable welding path according to the
specified welding task. The cooperation of the collision
monitoring module and the path planning module is
used to generate a collision-free and better motion path
that meets the process requirements. The 3D online
monitoring module mainly obtains the twin data in real
space and updates the display of the virtual entities on
the software.
DDDWR
DDDWR
DPE
DVE
DPS
DWP
Since the main function of the digital twin system of
the dual-arc welding robot is to provide welding path
planning and realize real-time bilateral control,
is mainly the twin data related to the path. In Eq. (3),
includes PE data ( ), VE data ( ), PS data
( ), and welding process data ( ).
DDDWR =(DPE,DVE,DPS,DWP)(3)
DPE
DVE
DPS
includes the real-world 15-dimensional robot
motion data and the position data of the workpiece in
space. The 15-dimensional robot motion data for the
two arc welding robots consist of 12 joint angle data
and 3 external axis angle data, while the workpiece
position data are a homogeneous transformation
matrix. is the geometric model data describing
geometric dimensions, assembly relationships, and
posture of virtual entities. is a dataset including the
extracted class and position data from the welding
seams, the solution of the welding sequence planning,
the bounding box data of arc welding robots and
DWP
workpieces, and the 15-dimensional joint angle data
obtained by inverse kinematics of the discrete
collision-free path point set. is the welding
process data, including welding current, welding
voltage, welding speed, and wire feed speed. These
data determine the quality of welding.
CNDWR
is the transmission channel used to realize
the information interaction between the above parts,
mainly the data interaction channel between the real
and virtual space. In order to make the dual-arc
welding robot system bidirectionally controllable, the
communication between real space and virtual space is
realized through Socket, using TCP/IP communication
protocol, and the data formats are double precision
format. Figure 2 represents the uploading of the real
space state data and the downloading of the control
data generated by the virtual space.
After elucidating the architecture of the dual arc
welding robot digital twin system, the DT-driven path
planning can be divided into six steps in real space and
virtual space, as shown in Fig. 3.
Step 1: Digital twins are created, constrained by the
robot’s kinematics, external axis behavior, and scene
constraints. The assembly relationships of the scene
entities are aligned with the real space, and then all
workpiece position data on the workbench are
measured by the laser scanning sensor. The initial
spatial posture data of the robot are obtained by
Path service
System software
3D
monitoring
Path
planning Decision-making
Digital data
Interaction
Kinematics
Collision
detection
Seam
extraction
Feedback solution
Model data import
Posture update
Update
Virtual entities
Drive
Feedback
PSDWR
Physical entities
PEDWR
CNPS
CNPD
DDDWR
CNSD
CNVD
CNPV
VEDWR
CNVS
Fig. 1 Five-dimensional dual arc welding robot digital twin system.
XuewuWangetal.:DigitalTwinImplementationofAutonomousPlanningArcWeldingRobotSystem 239
performing forward kinematics in virtual space with
the transmitted 15 initial joint values.
Step 2: The welding seam extraction module is used
to obtain data on the class and space location of the
welding seam. In this welding seam extraction module,
the workpiece parametric models can truly reflect the
assembly structure and shape characteristics, which
simplifies the welding seam data acquisition.
Step 3: Bounding box data for the robot and
workpiece are obtained using the topology and shape
features of their respective CAD models. These data
are then used to build the collision detection module,
which is crucial for ensuring the safety of the welding
process.
Step 4: Based on Step 3, a global welding sequence
is planned for the welding seam data from Step 2. After
the welding sequence is planned, a local path planning
algorithm is used to plan a collision-free welding path
consisting of discrete points between the jump points of
each welding seam.
Virtual space Real space
Uploading CNDWR
Downloading CNDWR
Robot’s pose data
15 dimensional joint data
Workpieces’ location and type data
Socket
Fig. 2 Process of data transmission.
Step 1
Virtual space Real space
Construction of
VEDWR
PEDWR
DT constraints
Kinematics
Behavior
Scene boundary
Computer graphics
Model topology
Shape feature
Capturing features
Welding seam extraction
Collision detection
Welding sequence planning Local jump path planning
Data processing Inverse kinematics
Decision data Operating state
Socket communication Communication module
Control module
Laser module
Welding module
Hardware moduledSoftware compositions
Process control
Position matrix
Feedback pose
Motion data
Data collection
Point cloud processing
DT system
Welding seam position Collision data
Gantry
Robot 1
Robot 2
Workpiece
Workbench
Laser scanning
sensor
Motion data
Path discretization points
Bilateral control
Step 2
Step 3
Step 4
Step 5
Fig. 3 Steps of DT-driven path planning.
240 ComplexSystemModelingandSimulation,September2023, 3(3): 236−251
Step 5: Acquire the data of robot by solving the
discretization points kinematically. Then, the motion
data are transferred to the physical entity in real space
via a socket communication. At the same time, the
robot’s built-in controller derives the position of the
end welding torch and feeds it back to the virtual space
to update the digital twin’s representation of the robot’s
position and orientation.
4 System Calibration
MC
PC
GC
BC
To plan and control the dual arc welding robot in the
real system to move and complete a series of welding
tasks, the virtual system must ensure that the virtual
entities of the digital twin system satisfy the constraint
relations described in Eq. (2). Firstly, the kinematic
analysis of the robot is required to ensure that the
kinematic constraints ( ) of the virtual system are
consistent with the real system. Then, the coordinate
system calibration between virtual and real space is
carried out to ensure that robots meet the positional
consistency ( ). Geometric consistency ( ) is
satisfied at CAD modeling time. The boundary
collision constraint ( ) is set using operational
parameters such as the working range of the gantry, as
shown in Table 1.
After completing the above steps, the real twin also
can drive the virtual dual arc welding robot with real
data. The control module collects the end position data
of the arc welding robot every 200 ms and sends it to
the virtual twin in real time via the communication
module. The virtual twin is visualized through the
software’s 3D monitoring module.
4.1 Kinematic analysis of arc welding robot
The robot needs to satisfy the kinematic relations when
welding. In order to properly control the physical
entity, the virtual entity of the dual-arc welding robot
needs to perform kinematic analysis to have consistent
kinematic relations with the real robot. The Denavit-
Hartenberg (D-H) parameters of the arc welding robot
are shown in Table 2. The correlation between the
robot angle and the end position of the welding torch
can be obtained by performing the kinematics analysis
according to Table 2.
4.2 Coordinate setting
e
Aligning the origins of the virtual and real coordinate
systems ensures that the virtual and real robot systems
are consistent in the definition of the world coordinate
system[16]. The coordinate systems in both the virtual
and real systems are defined in the case where external
axis angles are zero. The coordinate system is
established with the center point of the rotation axis
(E3) as the coordinate origin of the external axis
coordinate system and the forward direction of the
external axis (E2) as the positive direction of the y-axis.
This coordinate system is defined as the world
coordinate system of the whole system as shown in
Fig. 4. The transformation matrix of the external axis
coordinate system to the world coordinate system is
shown in Eq. (4).
To
e(∆x,∆y, θ)=
cos θ−sin θ0∆x
sin θcos θ0∆y
0 0 1 0
0 0 0 1
(4)
∆x
∆y
θ
OTo
e
→e
where , , and denote the values of the external
axes E1, E2, and E3, respectively. The transformation
between the world coordinate and the external axis
coordinate is as follows: .
r1
r2
Te
r1
Te
r2
e
Te
r1
→r1
Te
r2
→r2
and denote the reference coordinate systems of
the two robots. According to the real robot installation
position, the transformation matrices and are
shown in Eq. (5). The coordinate relationship between
the external axis coordinate and the robot base
coordinate is , then the coordinate relationship
between the world coordinate and the robot base
Table 1 Operation parameter setting.
Comment Name Value Unit/type
x-axis boundary xBoundary −2000−2000 mm
y-axis boundary yBoundary −3500−3500 mm
Workbench height wrkHeight 1820 mm
Detection accuracy detAccuray 1×10−36 mm
Decimals count decimalCnt 2 int
Meshing quality Quality Precise enum
Note: int: integer; enum: an enumeration type with the following
options: Coarse, Normal, and Precise.
Table 2 D-H parameters of the arc welding robot.
i
Joint
ai
(mm)
αi
(rad)
di
(mm)
θi
(rad)
Range (°)
1 750 −π/2 450
θ1
−170−170
2 640 π 0
θ2−
π/2 −100−160
3 195 −π/2 0
θ3+θ2
−190−268
4 0 π/2 −700
θ4
−200−200
5 0 −π/2 0
θ5
−140−140
6 0 π −75
θ6
−270−270
ai
αi
di
θi
Note: : link length, : link twist, : link offset, and : joint
angle.
XuewuWangetal.:DigitalTwinImplementationofAutonomousPlanningArcWeldingRobotSystem 241
OTo
e
→e
coordinate is .
Te
r1=
0−1 0 0
1 0 0 −500
0 0 −1 0
0 0 0 1
×
1 0 0 0
0−100
0 0 −1 0
0 0 0 1
=
0 1 0 0
1 0 0 −500
0 0 −1 0
0 0 0 1
,
Te
r2=
0 1 0 0
−1 0 0 −500
0 0 −1 0
0 0 0 1
×
1 0 0 0
0−100
0 0 −1 0
0 0 0 1
=
0−1 0 0
−1 0 0 500
0 0 −1 0
0 0 0 1
(5)
(xw,yw, θz)
(xw,yw)
xoy
θz
In order to guarantee the consistency of the
workpiece placement in the virtual and the real space,
the laser scanning sensor needs to scan the whole
workbench to obtain the accurate position data of the
workpiece concerning the world coordinate system
after each placement of the workpiece. The real space
position data are transferred to the virtual
space for calibration. The term is the position
of the workpiece in the plane of the world
coordinate system, and is the rotation angle around
the z-axis. The transformation matrix of the workpiece
coordinate system to the world coordinate system is
shown in Eq. (6).
To
w=
cos θzsin θz0xw
−sin θzcos θz0yw
0 0 1 −∆h
0 0 0 1
(6)
5 Welding Seam Extraction and Collision
Detection
In the proposed dual arc welding robot digital twin
system, the design of welding seam extraction and
collision detection algorithms is built on the virtual
entity topology.
5.1 Topology of virtual entities
In order to improve the efficiency of welding seam
extraction and collision detection, this paper proposes a
hierarchical topology structure for constructing a
virtual entity, and it is composed of three bounding
volume hierarchies (BVH) trees. As shown in Fig. 5,
the first level BVH tree is formed from oriented
bounding boxes of each virtual entity. The second level
BVH tree is composed of oriented bounding boxes of
all subparts on the assembly tree of each virtual entity.
Most collision detection can be finished with bounding
box calculation in first two level BVH trees. The third
level BVH tree consists of sensitive entities and is
called the sensitive entity layer. This manuscript
defines the smallest operable unit as the sensitive
entity, and the sets of smallest operable units constitute
the sensitive entity layer. In sensitive entity layer, time
consuming operations such as precise collision
detection and welding seam extraction are performed
by Boolean operation. For instance, precise collision
detection first decomposes two complex virtual objects
into multiple sub-entities, and the sub-entities create a
sensitive entity set and perform entity intersection
detection. Sophisticated precision collision detection
can only be performed at the sensitive entity layer. In
welding seam extraction, it is necessary to perform the
topological intersection of the sensitive entity layer,
Zo
XoXr1
Pr1
Yr1
Zr1Zr2
Yr2
Pr2
Xr2
Po
Yo
Ze
Pe
XeYe
Fig. 4 System coordinate system setting.
242 ComplexSystemModelingandSimulation,September2023, 3(3): 236−251
where the smallest operable units are located, in order
to obtain the precise position of the welding seam.
5.2 Welding seam extraction
Ship sub-assembly welding involves a diverse range of
workpieces and non-uniform distribution of welding
seams. To address this, the system offers two modes of
welding seam extraction: automatic generation and
manual selection.
3rd
The input data for the automatic welding seam
generation mode are the CAD model of the workpiece,
and the output data are the welding seam data for that
workpiece. The detailed process is shown in Fig. 6. The
input CAD model is first separated into rib plates and
base plates, where the rib plate entity is designated as
the sensitive entity for welding seam extraction. The
level BVH tree of the workpiece topology is
composed of rib plates and baseplate entities. In the
sensitive entity layer, preliminary welding seam data
are obtained through the entity intersection of the rib
and base plates. However, the preliminary data may
contain short edges that are not required to be welded,
creating noise in the final results. In this case, Eq. (7) is
applied to remove these unnecessary elements:
Ei∈Dseam,if (L(Ei)>LTHLD )and i∈DS (7)
Ei
DS
Dseam
L(·)
Ei
LTHLD
Ei
LTHLD
where represents the edge in the preliminary dataset
. The term denotes the valid seam data.
Similarly, is used to calculate the length of , and
is the threshold length. The short edges are
filtered out by comparing the length of each with
.
Manual selection is a complementary mode to
automatic generation. The user can manually select the
welding seams in the space according to customized
1st level BVH tree: Virtual entity OBB
2nd level BVH tree: Subpart OBB
3rd level BVH: Sensitive entity
Triangle
Subpart OBB Subpart OBB
Sensitive entity
(e.g., triangle)
Virtual entity OBB Virtual entity OBB
Virtual space
Fig. 5 Topology of virtual entities.
Load CAD model Split the workpiece Intersect the entities Get preliminary data Output the result
Fig. 6 Automatic generation strategy of workpiece welding seam.
XuewuWangetal.:DigitalTwinImplementationofAutonomousPlanningArcWeldingRobotSystem 243
requirements. Inspired by the idea of triangular mesh
surface intersection[22], the virtual workpieces’ rib
plates and base plates are triangulated at first. The
sensitive entity is set as a triangle in this mode.
Afterwards, intersection triangles between the rib plate
and base plate are selected via the mouse click event.
The intersection segment obtained by the intersection
calculation is the welding seam, as shown in Fig. 7.
∆1
∆2
π1
π2
Li
∆1
Li
Li
In Fig. 7, the rib plate’s triangle-1 and the base
plate’s triangle-2 are selected and extended into
and planes, respectively. The intersection line is
obtained by solving parametric equations of the two
planes. To determine the start and end points of the
welding seam, three vertices in are substituted into
the linear equation of the intersection line . The
process provides two points on the intersection line ,
corresponding to the start and end points of the welding
seam.
In addition to the welding seam position, the axis
and centerline direction are also essential
characteristics of the welding seam. As shown in
XwYwZw
Xw
ES
Fig. 8, the welding seam coordinate system is set as
. The is consistent with the weld direction
, expressed as
Xw=ES
∥ES∥,ES =xwp2−xwp1,ywp2−ywp1,zwp2−zwp1
(8)
Zw
∆1
∆2
Yw
As shown in Eq. (9), the welding centerline is
determined by normal vectors and , while is
determined by the right-handed rule.
XwYwZw
√2w
Zw
XTYTZT
ZT
Zw
ZT
ZT
XTYTZT
As shown in Fig. 9, considering the actual welding
process, the welding seam coordinate system is
lifted along the direction. The coordinate
system at the end of the welding torch is set as .
The posture of the welding torch is related to the
welding seam. The end of the welding torch should
coincide with the origin of the lifted welding seam, and
coincides with the welding centerline . The angle
between and the horizontal plane is set up with an
adjustable range of ±15° to cope with different welding
requirements. Figure 8 describes the relationship
between welding torch posture and welding seam
characteristics. When does not need to be adjusted,
the torch end coordinate system follows Eq.
(10). As this system is the dual arc welding robot for
welding, the earlier process needs to be repeated for the
other side of the welding seam.
Zw=n1+n2
∥n1+n2∥(9)
XT=Xw,
ZT=−Zw,
YT=XT×ZT
(10)
5.3 Collision detection
Collision detection is essential in planning a collision-
free welding path. This system needs to detect robot
joints’ self-collision, the collision between two robots,
and the collision between the robot and the workpiece.
π2
π1
Li
WP1
WP2
1
2
Fig. 7 Single welding seam extraction process.
WP1
WP2
15°
15°
π2
π1
YT
ZT
XT
OT
Yw
Zw
Ow
Xw
path
OwOwOwOwOwOw
ES
Fig. 8 Relationship between welding torch posture and welding seam characteristics.
244 ComplexSystemModelingandSimulation,September2023, 3(3): 236−251
These collision detections can be abstracted as collision
detection between two entities.
The collision detection algorithm for two entities is
built on the proposed topology of virtual entities, with
sensitive entities set up as sub-entities of the virtual
entities to be collision detected. The algorithm is
hierarchical, and the next level is activated only after
detecting collisions in the previous level BVH tree. The
algorithm flow is depicted in Fig. 10, which is divided
into 3 stages:
(1) The separating axis theorem (SAT)[23] is used to
judge whether the oriented bounding boxes (OBB) of
two virtual entities intersect or contain each other. If
not, the two entities do not collide and output the result.
Otherwise, second-stage detection is needed.
2nd
(2) The second-stage detection is carried out in the
-level BVH tree. The Euclidean distance between
the centroid of Entity-A’s OBB and the centroids of
Entity-B’s all subparts OBB is calculated in its
assembly tree. Priorities are sorted by centroid
Euclidean distance, with the shortest distance being the
highest priority. Then whether the oriented bounding
box of Entity-A intersects or contains the oriented
bounding boxes of the first three closest subparts of
Entity-B is judged. The subparts whose bounding
boxes intersect with A’s bounding box are stored in the
candidate queue where exact collision detection is
required. If the number of elements in this queue is
zero, it means that there is no collision, and then the
algorithm exits.
(3) Precise collision detection requires queuing out
the candidate queue and calling the Open CASCADE
entity intersection method to determine whether the
collision with Entity-A occurs. If none of the elements
in the candidate queue collides with Entity-A, the two
entities do not collide. Precise collision detection is
time-consuming, but it is only needed when two
objects are very close. In most cases, collision
detection in the first two levels of the BVH tree is
perfectly adequate.
6 Three-Dimensional Path Planning
6.1 Global welding sequence planning
Global welding sequence planning plays an important
role in improving the welding efficiency of arc welding
robots. Welding sequence planning involves
determining two parameters: the welding sequence and
direction, which can thus be transformed into a
generalized traveling quotient problem (TSP) with
direction[23]. The algorithm operates on the following
assumptions: (1) Each welding seam is welded only
once. (2) The welding task involves only straight
seams. (3) The rotary axis E3 can only be rotated to 0°,
90°, 180°, and −90°. Based on this, global welding
sequence planning based on GA is utilized in the
w
w
WP1
Original welding seam
Normal vector
Welding seam after lifting
WP2
2w
√
n2
n1
Fig. 9 Lifting of welding seam.
Start
Acquisition of entities topology
Calculation of the distance between A’s
OBB and B’s subpart OBB
Prioritize by distance
Traversing candidate queue for precise
entity intersection detection
Output collision detection result
Yes
Yes
End
No
No
Step 2
Step 3
Step 1
Intersection
detection of two whole
entities’ OBB
Intersection detection
between A’s OBB and B’s high priority
subparts OBB
Fig. 10 Process of collision detection algorithm.
XuewuWangetal.:DigitalTwinImplementationofAutonomousPlanningArcWeldingRobotSystem 245
digital twin platform for dual arc welding robots.
d(·)
Since it is difficult to obtain the actual robot running
distance between welding points, and the actual
running distance of the robot in local path planning is
positively correlated with the Euclidean distance
between welding points, the Euclidean distance of two
endpoints is used in the welding sequence
planning. Meanwhile, it is partial to consider only the
motion of the torch. The rotation of the rotary axis will
also increase the ticks of welding. Hence, the final
welding sequence model is shown in Eq. (11).
min L=min
i∈S
j∈S
dmic1
i+(1−mi)c2
i,mjc1
j+(1−
mjc2
jwi,j+M
i∈S|mi−mi+1|,
s.t.
i∈S
wi,j=1,∀j∈S;
j∈S
wi,j=1,∀i∈S;
i∈s
j∈s
wi,j⩽|s|−1,∀s∈S,s,∅;
mi∈{0,1},∀i∈S;
wi,j∈{0,1},∀i,j∈S
(11)
L
i
j
i
j
S
c1
i
c2
i
i
d(·)
wi,j
mi
mi=
c1
i→c2
i
mi=
c2
i→c1
i
where is the path length, and denote the -th and
-th welding seams in the welding seam set ,
respectively. and are the two endpoints of the -th
welding seam, and represents the Euclidean
distance between the two endpoints. The term is the
flag between two welding seams. 1 means that the two
welds are connected, and 0 means that there is no
connection. Likewise, indicates the weld direction,
0 indicates the positive direction of welding
( ), and 1 indicates the negative direction
( ). Moreover, M denotes the penalty value for
changes in weld directions, which is set to 1000 in this
paper.
The design of the encoding scheme is the key step in
optimizing welding sequence using genetic algorithm.
The double coding and decoding scheme of the weld
sequence and weld direction are explained in Fig. 11.
The integer coding is adopted for the weld sequence
O→
c1
3→c2
3→c1
5→c2
5→c2
2→c1
2→c1
4→c2
4→c2
1→c1
1→O
and the binary coding for the weld direction. The
decoded weld sequences of the example dual coding
chromosomes including 5 welding seams are
,
where point O is the initial position of the robot.
The detailed steps of applying genetic algorithm for
global welding sequence planning are as follows:
(1) Initialization: The sequence and direction of
initial welding seams are randomly generated as the
initial population, and the corresponding fitness value
is calculated.
(2) Generation of mating pool: Duplicate
individuals in the initial population are first removed.
Then, 4 individuals are grouped together to compare
the fitness values, and the two individuals with higher
fitness value are put into the mating pool. The above
steps are repeated until the mating pool contains the
desired number of individuals.
(3) Crossover operation: Circular and uniform
crossover operators are adopted for the weld sequence
and weld direction, respectively.
(4) Mutation operation: The weld sequence is
changed by the 2-opt method, while the weld direction
is changed by a single point.
(5) Selection: The parent and offspring populations
are merged, and duplications are deleted. Then, the top
N individuals are selected into the next population
according to the fitness value ranking. If the number of
next population is less than N, new individuals are
randomly generated to fill the vacancy.
(6) Judgement of termination condition: The
optimal solution is output if the maximum number of
iterations is reached. Otherwise, go back to Step (2).
6.2 Local path planning based on RRT-Connect-
RP
After planning the welding sequence and welding
direction, the arc welding robot needs to move from the
exit point of the previous welding seam to the approach
point of the next welding seam. Local path planning
aims to find a better and collision-free transition path
between two points based on the actual scenario until
Weld sequence:
Weld direction:
Decoding:
0
O→c3
1→c3
2→c5
1→c5
2→c2
2→c2
1→c4
1→c4
2→c1
2→c1
1→O
0 01 1 1
0
Workpiece A001
Workpiece A002
Positive direction
Negative direction
14253
Fig. 11 Double coding and decoding scheme.
246 ComplexSystemModelingandSimulation,September2023, 3(3): 236−251
all welding tasks are completed. The sampling-based
RRT algorithm can satisfy the needs of local path
planning. However, the RRT algorithm has the
shortcomings of overly random sampling and single-
step size setting. Therefore, this manuscript chooses to
partition the region of the workpiece and proposes the
RRT-Connect-RP algorithm.
qgoal
3rd
In conventional RRT-Connect[24], all sampling points
are randomly generated, and setting directly to the
starting point of the next weld would result in a large
number of invalid paths being acquired. Especially
when searching the region below the height of the
workpiece, the torch needs to do precise collision
detection with the workpiece on the level BVH tree
and can easily collide with the workpiece. Therefore,
the RRT-Connect-RP algorithm divides the region
where the workpiece is located into two parts, using the
height of the workpiece as the boundary:
ψ=ψ1,0⩽h⩽hw;
ψ2,h>hw
(12)
hw
h
ψ1
ψ2
ψ2
qgoal
ψ1
where is the height of the workpiece, denotes the
height from the end of the welding torch to the base
plate of the workpiece, is the collision-prone region,
and is the region where the robot can move freely.
The two endpoints of each welding seam are extended
to both sides and lifted to as jump points. Using
jump points as in RRT-Connect-RP can
significantly reduce invalid sampling in the area.
2l
ψ2
ψ1
Wp_detect
Tcoll
Tcoll
The RRT-Connect-RP expands nodes according to
different regions. The fixed value is utilized as the
expansion step size if in the region, which can
accelerate the path search. The extension step of the
region is related to whether it is doing precise collision
detection , and the number of collisions .
Precise collision detection implies that the node is very
close to the obstacle and the step size needs to be
reduced. is used to control the magnitude of the
StepSize
reduced step size. The formula for calculating
is shown in Eq. (13). Probabilistic target-biased
sampling is also applied to reduce the number of
iterations.
StepSize =
2l,xinit ∈ψ1;
1+(−1)Wp_detect ×e−3
Tcoll+1l,xinit ∈ψ2
(13)
The strategy of the extension nodes is shown in
Fig. 12. The region partition and the generation of
jump points are shown in Fig. 13a. The local trajectory
between the start and end points generated by RRT-
Connect-RP is shown in Fig. 13b.
7 System Implementation
Figure 14 depicts the platform of the dual arc welding
robot digital twin system, which includes a monitoring
module displaying the real system in the bottom right
corner of the interface. The virtual entities in the virtual
system have size, position, and motion constraints
consistent with the real system, and all coordinate
systems have been calibrated in advance. In this digital
twin system, bidirectional control of virtual and real
ψ2ψ1
xgoal
xstart
<l
≈l
2l
Fig. 12 Extension nodes strategy.
End of the weld
Transition trajectory
(a) Jump points generation (b) Local trajectory
q1,start q2,start
q1,goal q2,goal
Jump point
ψ2
ψ1
hw
Fig. 13 Local path planning by RRT-Connect-RP.
XuewuWangetal.:DigitalTwinImplementationofAutonomousPlanningArcWeldingRobotSystem 247
robots, welding seam extraction of the workpiece,
global welding sequence planning, and local path
planning based on the RRT-Connect-RP algorithm are
implemented.
7.1 Welding seam extraction results
Figure 15 shows the results of welding seam extraction
for the three workpieces. The welding seam extraction
based on the topology of virtual entities successfully
extracted all straight seams without any failures and
accurately planned the welding posture for each seam.
The relative position of welding seams on the
workpiece is recorded by the data management module.
Thus, only one welding seam extraction is required for
one type of workpiece. When the workpiece position
changes, the endpoints of each weld can be calculated
by the relative position.
7.2 Global welding sequence planning results
Figure 16 depicts the layout of five workpieces with a
total of 32 welding seams in virtual space. Two dual
arc welding robots are employed to handle symmetric
welding tasks simultaneously, while each robot
O→c1
15 →c2
15 →c1
16 →c2
16 →c1
8→
c2
8→c1
7→c2
7→c2
10 →c1
10 →c2
9→c1
9→c2
11 →c1
11 →c2
3→
c1
3→c2
4→c1
4→c2
6→c1
6→c1
5→c2
5→c1
2→c2
2→c2
1→c1
1→
c2
12 →c1
12 →c1
13 →c2
13 →c1
14 →c2
14 →O
independently works on 16 welding seams. Therefore,
only 16 welding seams and 32 endpoints are considered
in the welding sequence planning. The genetic
algorithm is used to plan the welding sequence with a
population size of 200 and 150 iterations. The resulting
welding sequence is
. The total length
of the path is 22 070.46 mm, as shown in Fig. 17.
7.3 Welding path planning results
Following the welding sequence planned in Section
7.2, the RRT-Connect-RP algorithm is used to plan the
local jump path. The welding trajectory in virtual space
is presented in Fig. 18a. The offline simulation module
is employed in the system to validate the simulated
path in the virtual space. Subsequently, the online
control module establishes a twin connection to the real
system and transmits the 15-dimensional joint data to
robots in real-time. The simulation in virtual space is
illustrated in Fig. 18b, while Fig. 18c shows the digital
c
mma
p
i
Simulation panel
Status monitoring
Control panel
StepSize
CAD
modeling Seam
gement
Fig. 14 Platform of the dual arc welding robot digital twin system.
Weld seam
Selected triangle
Endpoint of weld seam
Fig. 15 Welding seam extraction results.
248 ComplexSystemModelingandSimulation,September2023, 3(3): 236−251
twin system controlling the physical entity to complete
the welding task.
In order to verify the effectiveness of the proposed
digital twin system for the real robot, a continuous
trajectory following and data feedback experiment was
constructed. The continuous control trajectories for
robots and discrete endpoints that the real system fed
back in every 200 ms are compared in Fig. 19. The
continuous control trajectories were the pre-planned
collision-free paths generated by the digital twin
system. From the experimental result, the real dual arc
welding robot could follow the control trajectory
generated by the proposed system. The dual arc
welding robot digital twin system allows for welding
path planning and simulation as well as continuous
control of real robots.
8 Conclusion
To improve work efficiency in the ship sub-assembly
welding robot system, a digital twin system for
autonomous welding path planning of the dual arc
welding robot is developed. In the system, the topology
of virtual entities is proposed first. The welding seam
extraction and collision detection algorithms are then
designed based on the topology. In addition, intelligent
optimization algorithms are applied to the welding path
planning of the dual arc welding robot, including the
global welding sequence planning based on the genetic
algorithm and the local jump path planning based on
the RRT-Connect-RP algorithm. Through the result
analysis, the dual arc welding robot digital twin system
achieves consistency and synchronization between the
virtual entities and physical entities. At the same time,
the cyber-physical integration of welding path
planning, decision optimization, and robot control is
realized.
Acknowledgment
This work was supported by the National Natural
Science Foundation of China (Nos. 62076095 and
61973120) and National Key Research and
Development Program (No. 2022YFB4602104).
Y
Z
Fig. 16 Workpieces layout in virtual space.
−400
−600
−800
−1000
−1200
Z (mm)
Y (mm)
X (mm)
−1400
−1600
−1800
2000
1500
1000
500
0
−500
−1000
−1500
−2000
−2000 −1500 −1000 −500 0500
21
12
13
14
15
11
3
5
4
6
9
7
8
10
16
Start point
Weld points
Jump trajectories
Weld seams
O
1000 1500
Fig. 17 Result of welding sequence planning.
XuewuWangetal.:DigitalTwinImplementationofAutonomousPlanningArcWeldingRobotSystem 249
References
F. Tao, C. Y. Zhang, Q. Qi, and H. Zhang, Digital twin
maturity model, (in Chinese), Computer Integrated
Manufacturing Systems, vol. 28, no. 5, pp. 1267–1281,
2022.
[1]
N. Zhao, G. Lodewijks, Z. Fu, Y. Sun, and Y. Sun,
Trajectory predictions with details in a robotic twin-crane
system, Complex System Modeling and Simulation, vol. 2,
no. 1, pp. 1–17, 2022.
[2]
P. Gu and Y. Xu, Research on path planning of
synchronous welding of dual robot based on QPSO, (in
Chinese), Industrial Instrumentation & Automation, vol. 5,
pp. 78–82, 2015.
[3]
F. Tao, W. R. Liu, M. Zhang, T. Hu, Q. Qi, H. Zhang, F.
Sui, T. Wang, H. Xu, Z. Huang, et al., Five-dimension
digital twin model and its ten applications, (in Chinese),
Computer Integrated Manufacturing Systems, vol. 25,
no. 1, pp. 1–18, 2019.
[4]
W. R. Liu, F. Tao, J. F. Cheng, L. C. Zhang, and W. M.
Yi, Digital twin satellite: Concept, key technologies and
applications, (in Chinese), Computer Integrated
Manufacturing Systems, vol. 26, no. 3, pp. 565–588, 2020.
[5]
Z. Xu, F. Ji, S. Ding, Y. Zhao, Y. Zhou, Q. Zhang, and F.
Du, Digital twin-driven optimization of gas exchange
system of 2-stroke heavy fuel aircraft engine, J. Manuf.
Syst., vol. 58, pp. 132–145, 2021.
[6]
X. Li, H. Liu, W. Wang, Y. Zheng, H. Lv, and Z. Lv, Big
data analysis of the Internet of Things in the digital twins
of smart city based on deep learning, Future Gener.
Comput. Syst., vol. 128, pp. 167–177, 2022.
[7]
Y. Guo, F. Yang, S. Ge, Y. Huang, and X. You, Novel
knowledge-driven active management and control scheme
of smart coal mining face with digital twin, (in Chinese),
Journal of China Coal Society, doi: 10.13225/j.cnki.jccs.
2022.0223.
[8]
Y. Wang, F. Tao, M. Zhang, L. Wang, and Y. Zuo, Digital
twin enhanced fault prediction for the autoclave with
insufficient data, J. Manuf. Syst., vol. 60, pp. 350–359,
2021.
[9]
L. Liu, X. Zhang, X. Wan, S. Zhou, and Z. Gao, Digital
twin-driven surface roughness prediction and process
parameter adaptive optimization, Adv. Eng. Inform.,
vol. 51, p. 101470, 2022.
[10]
S. S. Akash and M. S. Ferdous, A blockchain based
system for healthcare digital twin, IEEE Access, vol. 10,
pp. 50523–50547, 2022.
[11]
J. Pang, Y. Huang, Z. Xie, J. Li, and Z. Cai, Collaborative
city digital twin for the COVID-19 pandemic: A federated
learning solution, Tsinghua Science and Technology,
vol. 26, no. 5, pp. 759–771, 2021.
[12]
X. Zhang, B. Wu, X. Zhang, J. Duan, C. Wan, and Y. Hu,
An effective MBSE approach for constructing industrial
robot digital twin system, Robotics Comput. Integr.
Manuf., vol. 80, p. 102455, 2023.
[13]
W. Hu, C. Wang, F. Liu, X. Peng, P. Sun, and J. Tan, A
grasps-generation-and-selection convolutional neural
network for a digital twin of intelligent robotic grasping,
Robotics Comput. Integr. Manuf., vol. 77, p. 102371,
2022.
[14]
N. Kousi, C. Gkournelos, S. Aivaliotis, C. Giannoulis, G.
Michalos, and S. Makris, Digital twin for adaptation of
robots’ behavior in flexible robotic assembly lines,
Procedia Manuf., vol. 28, pp. 121–126, 2019.
[15]
W. Wang, W. Ding, C. Hua, H. Zhang, H. Feng, and Y.
Yao, A digital twin for 3D path planning of large-span
curved-arm gantry robot, Robotics Comput. Integr.
Manuf., vol. 76, p. 102330, 2022.
[16]
S. Yan, Construction and evolution of digital twin for
welding robots, (in Chinese), MSc dissertation, School of
Mechatronics Engineering, Harbin Institute of
Technology, Harbin, China, 2021.
[17]
YZ
X
(a) Welding trajectory (b) Simulation in virtual space (c) Welding process
Fig. 18 Welding path results.
−400
−600
−800
−1000
−1200
Z (mm)
X (mm)
Y (mm)
−1400
−1600
−1800
1200 1000 800
600 1000 500 0−500 −1000
Robot1 continuous control trajectory
Robot1 discrete feedback point
Robot2 continuous control trajectory
Robot2 discrete feedback point
Fig. 19 Comparison of continuous control trajectories and
discrete feedback points.
250 ComplexSystemModelingandSimulation,September2023, 3(3): 236−251
F. Hu, Mutual information-enhanced digital twin promotes
vision-guided robotic grasping, Adv. Eng. Inform., vol. 52,
p. 101562, 2022.
[18]
R. S. Tabar, K. Wärmefjord, R. Söderberg, and L.
Lindkvist, Efficient spot welding sequence optimization in
a geometry assurance digital twin, Journal of Mechanical
Design, vol. 142, no. 10, p. 102001, 2020.
[19]
Open CASCADE, https://www.opencascade.com/, 2022.[20]
H. Zhang, Q. Qi, W. Ji, and F. Tao, An update method for
digital twin multi-dimension models, Robotics Comput.
Integr. Manuf., vol. 80, p. 102481, 2023.
[21]
T. Möller, A fast triangle-triangle intersection test, J.
Graph. Tools, vol. 2, no. 2, pp. 25–30, 1997.
[22]
S. Gottschalk, Separating axis theorem, Tech. Rep. TR96-
024, Department of Computer Science, UNC Chapel Hill,
Chapel Hill, NC, USA, 1996.
[23]
J. J. Kuffner and S. M. LaValle, RRT-connect: An
efficient approach to single-query path planning, in Proc.
2000 ICRA. Millennium Conference. IEEE Int. Conf.
Robotics and Automation. Symposia Proceedings (Cat.
No. 00CH37065), San Francisco, CA, USA, 2002, pp.
995–1001.
[24]
Xuewu Wang received the PhD degree in
automation from China University of
Mining and Technology in 2003, the MS
degree in welding from Lanzhou
University of Technology in 2000, and the
BS degree in welding from Harbin Institute
of Technology in 1995. He has been with
East China University of Science and
Technology since 2003 where he is working as an associate
professor. His research interests are intelligent welding robot,
intelligent optimization algorithm, and welding automation.
Yi Hua received the BS degree from
Xinxiang University in 2021. He is
currently pursuing the MS degree at East
China University of Science and
Technology. His main research interests
include system simulation and
optimization, intelligent manufacturing,
and digital twin.
Jin Gao received the BS degree in
automation from Qingdao University of
Science and Technology, China in 2020.
He is currently pursuing the MS degree at
East China University of Science and
Technology. His research interests include
multi-objective optimization algorithms,
welding robot path planning, and modeling
and optimization.
Zongjie Lin received the BS degree from
Nanjing University of Technology, China
in 2021. He is currently pursuing the MS
degree at East China University of Science
and Technology. His main research interest
is wire and arc additive manufacturing.
Rui Yu received the BS degree in
information engineering from Xi’an
University of Posts &
Telecommunications, Xi’an, China in 2014
and the MS degree in electrical
engineering from University of Kentucky,
Lexington, KY, USA in 2019. He is
currently pursuing the PhD degree in
electrical engineering at University of Kentucky, Lexington,
KY, USA. His research is concerned with intelligent sensing and
control of welding processes.
XuewuWangetal.:DigitalTwinImplementationofAutonomousPlanningArcWeldingRobotSystem 251
