Human-Robot Collisions Detection for Safe Human-Robot Interaction Using One Multi-Input-Output Neural Network

Abstract

In this paper, a multilayer feedforward Neural Network based approach is proposed for human-robot collision detection taking safety standards into consideration. One multi-output neural network is designed and trained using data from the coupled dynamics of the manipulator with and without external contacts to detect unwanted collisions and to identify the collided link using only the intrinsic joint position and torque sensors of the manipulator. The proposed method is applied to the collaborative robots, which will be very popular in the near future, and is implemented and evaluated in 3D space motion taking into account the effect of the gravity. KUKA LWR manipulator is an example of the collaborative robots and it is used for doing the experiments. The experimental results prove that the developed system is considerably efficient and very fast in detecting the collisions in the safe region and identifying the collided link along the entire workspace of the three joints motion of the manipulator. Separate/Uncoupled neural networks, one for each joint, are also designed and trained using the same data and their performance are compared with the coupled one.

Full text

Soft Computing
ــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــ
METHODOLOGIES AND APPLICATION
Human-Robot Collisions Detection for Safe Human-Robot
Interaction Using One Multi Input-Output Neural Network
Abdel-Nasser Sharkawy 1, 2,
*
, Panagiotis N. Koustoumpardis2, Nikos
Aspragathos2
Publisher’s version: https://link.springer.com/article/10.1007/s00500-019-04306-7
Abstract In this paper, a multilayer feedforward Neural Network based approach is
proposed for human-robot collision detection taking safety standards into consideration. One
multi-output neural network is designed and trained using data from the coupled dynamics of
the manipulator with and without external contacts to detect unwanted collisions and to
identify the collided link using only the intrinsic joint position and torque sensors of the
manipulator. The proposed method is applied to the collaborative robots, which will be very
popular in the near future, and is implemented and evaluated in 3D space motion taking into
account the effect of the gravity. KUKA LWR manipulator is an example of the collaborative
robots and it is used for doing the experiments. The experimental results prove that the
developed system is considerably efficient and very fast in detecting the collisions in the safe
region and identifying the collided link along the entire workspace of the three joints motion
of the manipulator. Separate/Uncoupled neural networks, one for each joint, are also designed
and trained using the same data and their performance are compared with the coupled one.
Keywords Coupled System, Robot Collision Detection, Collided Link Identification, Safe
Human-Robot Interaction, Coupled/Uncoupled Neural Network, Levenberg-Marquardt
1. Introduction
When the robots and humans share the same workspace, safety is very important due to the
operator proximity to the robot that can lead to potential injuries. Standards, which provide
the safety considerations and requirements, must be taken into account during human-robot
interaction (HRI). The safety requirements for the integration of industrial robots and
industrial robot systems are provided by ISO 10218-1 and ISO 10218-2 [1,2]. The basic
*
Abdel-Nasser Sharkawy
eng.abdelnassersharkawy@gmail.com
Panagiotis N. Koustoumpardis
koust@mech.upatras.gr
Nikos Aspragathos
asprag@mech.upatras.gr
1 Mechanical Engineering Department, Faculty of Engineering, Sou th Valley University, Qena 83523,
Egypt
2 Department of Mechanical Engineering and Aeronautics, University of Patras, Rio 26504, Greece

2
hazards and hazardous situations are described, and the requirements to eliminate or reduce
the risks associated with these hazards are presented. ISO/TS 15066 [3] specifies the safety
requirements for the collaboration between the industrial robot systems and the work
environment. It provides the requirements and guidance on collaborative industrial robot
operation according to [1,2], and the limits for quasi-static and transient contact. Safety
investigation was presented in [4] where the pain tolerance limit was determined. It is
recommended that safety must be considered in designing any control system for HRI.
1.1. Related Work
In the relevant literature, systems for safe human-robot collaboration were proposed based on
collision avoidance or detecting the collision. Collision can be avoided by monitoring the
environment using sensor based approaches. Based on depth sensors and vision, Flacco et al.
[5] proposed a method to evaluate distances between the robot and possibly moving obstacles
including humans. These distances were used to generate repulsive vectors which were
processed in order to obtain smooth and feasible joint velocity commands towards avoiding
obstacles. Schmidt and Wang [6] proposed an active collision avoidance via path
modification and robot control in real-time with zero robot programming. Anton et al. [7]
used a natural user interface (NUI) that allows the operator to intervene in the robot tasks.
Based on color sensors, Kitaoka et al. [8] proposed an approach for obstacle avoidance, where
the robot obtained color information and distance information of objects from images
captured by a stereo camera system. Lenser and Veloso [9] presented a fast method to avoid
unknown obstacles by a mobile robot using a simple camera, where the vision module
detected objects, both known and unknown, around the robot. The unknown objects were
detected by paying attention to occlusions of a floor of known colors. The results from the
previous approaches are promising if there are no occlusions and problems arise in detecting
the operator/obstacle and robots, when the sensor is too far or too close to the working space
so multiple sensors should be placed to monitor the workspace from multiple directions.
Number of sensors for collision avoidance is also considered. Single-sensor variants were
used in [10,11], whereas multi-sensor approaches were proposed in [12,13]. These methods
can be used to avoid the collision but modifications in the robot body are necessary for the
sensors installation.
To improve the safety system in HRI, apart from the level of collision avoidance, a
second level of collision detection and reaction is required if the first level protection fails.
Some researchers sought to develop methods to detect the collisions. These methods are
classified into two main categories; model-based and data-based.
For model-based approaches, disturbance observer was considered. Haddadin et al. [14]
presented two collision detection schemes and five reaction strategies using a lightweight
robot, which was designed particularly for cooperative and interactive tasks. In the first
collision detection scheme, the generalized momentum was required whereas in the other
scheme, the measured joint torque was compared with the one estimated by the robot model.
Cho et al. [15] proposed a collision detection method and developed three reaction strategies,
the oscillation mode, torque-free motion mode, and forced stop mode, which can be used in
various collision scenarios, based on generalized momentum and joint torque sensors. Their
experiments were done using 7-DOF service robot arm, developed by them. Jung et al. [16]
proposed a set of band-width filters in disturbance observer for human-robot collisions
detection. The collisions were detected by investigating the frequency characteristics of each
part of the manipulator and possible disturbances. Impedance and Admittance control based

3
approach was also taken into account. Morinaga and Kosuge [17] presented a nonlinear
adaptive impedance control law without using external sensors and based on the difference
between the actual input torque to the manipulator and the reference input torque, which is
calculated using the estimated parameters of the manipulator dynamics. Based on force/torque
sensor, Shujun Lu et al. [18] presented a model-based collision detection method using two
six-axis force/torque sensors; one on the base and the other on the wrist, for 1-DOF and 2-
DOF manipulators. The main disadvantages of model-based methods that an explicit dynamic
model is required, which is not always available, and there is uncertainty.
Data-based approaches are also proposed for human-robot collisions detection. In these
methods, data are used for training a system then the trained system is used for estimating
collisions and detecting them. Approaches based on fuzzy logic and neural network (NN)
systems were considered. Dimeas et al. [19,20] introduced two methods, one based on
intelligent fuzzy identification and another based on time series for one and two joints motion.
Fuzzy systems were implemented to estimate the external collision torques using the position
errors, the measured joint torques and the commanded joint velocities as the inputs. Their
Fuzzy system detected the collisions quickly and accurately by having lower threshold value.
The time series system estimated the collision torque by only using the measured joint
velocity signals and having lower collision detection time but its threshold was higher than
the fuzzy system. Their systems were implemented for 1-DOF and 2-DOF manipulators
ignoring the dynamic coupling between the joints during motion. Also, the generalization of
the methods outside of the training range motion was not considered/presented. Also, in [18],
a collision detection approach was developed based on neural network training, for one and
two joints motion, using base and wrist force/torque sensor. The NN was trained and designed
using the short history of joint angles and the readings from the wrist and base force/torque
sensors as the inputs to estimate the collision forces and contact positions. Although, the
results illustrated the validity of the developed collision detection scheme but two external
sensors are required and the generalization of the NN was not considered by testing it outside
of the training range motion.
In our previous papers [21,22], a multilayer feedforward NN was presented for collisions
detection using one joint motion and two joints in planar horizontal motion respectively. In
the second paper, one NN coupling the two joints was designed and trained taking into
consideration the dynamics coupling of the manipulator, for the collision detection and also
the identification of the collided link. The efficiency of the trained NN was high and the
generalization ability was presented.
1.2. The Main Contribution
In this paper, an approach based on the training of a multilayer feedforward neural network
(NN) is proposed, for human-robot collisions detection and collided links identification.
Before the design of the multilayer feedforward NN, the joints dynamics coupling of a
serial manipulator is investigated and taken into account. Rich signals such as joints position
error, commanded joints velocity, measured joints torque, and gravity are selected to be the
inputs to the NN, whereas its outputs are the estimated external torques of the joints. The
designed NN is trained using Levenberg-Marquardt (LM) algorithm, where a priori
knowledge of the dynamic model of the robot is not required. For the training of the NN, the
measurement of the collision force mapped to the joints torque and the gravity estimation are
required. The trained NN estimate the external torque, detect the collision, and using its
output the collided link can be identified. ISO standards are taken into consideration to be
sure that the collisions are detected by the proposed method in the safe region of the human-

4
robot interaction. As KUKA LWR robot is an example of the collaborative robots, which
have intrinsic joint position and torque sensors, it is used in this work for carrying out the
experiments and check the proposed method. Also, the estimation of external torque, given by
the Kuka Robot Controller (KRC), is used only for training the NN, but it can be replaced by
an external force/torque sensor. Extensive experimentation shows that the trained NN is very
effective in the entire workspace of the manipulator joints space motion despite that it is
trained in a part of it. Comparison between the performance of the uncoupled neural networks
(uncoupled NNs) and the proposed coupled one is presented and discussed.
The proposed method can be applied to any robot having joint torque sensors, or where
the estimations of the joints torques are available. The benefit from estimating the joint
torques from the motor currents can also be used. In more detail, this point is discussed later
in section 4 where the coupled NN is designed.
2. The Proposed Collision Detection Method
In this section, the proposed method is outlined. The dynamic coupling between the joints of
3-DOF serial manipulator is investigated analytically and experimentally to estimate the
effects of the inertia and Coriolis torques in the correct collision detection and particularly in
collided link identification. According to the results of this investigation, one multilayer
feedforward NN is designed coupling the three joints and trained using the Levenberg-
Marquardt algorithm. The proposed method can be applied to any serial manipulator.
However, it is presented here for only three degrees of freedom with rotational joints,
articulated robot, for the clarity and understandability and since the wrist joint axes are
crossed in the same point, the wrist could be considered as one body from the point of view of
the safety. In addition, the three joints have high possibility to collide. The results of the
initial experimental investigation are used to identify the necessary inputs by examining the
information richness of the investigated signals, in order to design properly the NN inputs.
The training data with and without collisions are collected in a range of the entire joint
configuration space but the trained NN is tested inside and outside of this range to prove its
generalization ability. The detection of the collision is based on the comparison of the
external collision torques of the joints estimated by the NN with the defined thresholds that
also proposed by the NN. The collision thresholds are defined as the maximum of the
absolute values of the approximation error, between the external torque used for NN training
and the one estimated by the NN, plus the average of the estimated external torque of the
contact-free motion for the first joint, whereas the maximum of the absolute values of the
approximation error of the contact-free motion for the other two joints. The approach for the
definition of the collision thresholds takes into account mainly the human safety and it is
really conservative as it is discussed in the main part of the paper, where the defined
thresholds are compared with the safety ISO standards for HRI. The trained NN is extensively
tested to illustrate its ability for the correct collision detection and collided link identification
and statistics for the NN performance are presented. Three uncoupled NNs are trained for the
three joints, one NN for each joint, and their performance is compared with the coupled one.
Since the proposed method is developed for collaborative robots and KUKA LWR robot is an
example of this type of robots, it is used for the data collection and the experimental
demonstration of the efficiency of the proposed approach.
We do not use the NN as a classifier/pattern recognition to directly detect
collision/identify colliding link because of the following problems: 1) the challenge to find
the minimal set of features [23,24] which can result in satisfactory predictive performance

5
and without loss of the intrinsic information contained in the original data. 2) lack of the
rigorous [24]. 3) the misclassification costs [24] which can carry significantly different
consequences on the different groups/classification results [24,25]. 4) most practical systems
require combining the NN with other techniques for pre- and post-processing [26].
The rest of this manuscript is organized as follows; Section 3 investigates the dynamic
coupling between the three joints of the manipulator. The design of the proposed coupled NN
is presented in Section 4, whereas Section 5 shows the training and validation of the coupled
NN for safe HRI. Section 6 presents the experimental evaluation of the effectiveness, the
performance and the generalization for the trained NN whether inside the joints range used
for the NN training or outside of this range. In Section 7, the performance of the uncoupled
trained NN, for each joint, is discussed. Also, a comparison between the performance of the
uncoupled/independent NNs and the coupled one is presented.
3. Dynamic Coupling of the Joints
It is well known that the manipulator is a dynamically coupled system, where joint motion or
external force (collision) to any link affects the torques of the three joints considering 3-DOF
manipulator with rotational joints (articulated robot as shown in Fig. 1). Dynamic coupling of
the joints is crucial for the proposed collision detection method and for identification of the
collided link considering the effects of the inertial forces appeared due to collision.
The dynamics for three joints motion of the manipulator are given by [27]
 (1)
where the vectors  and  represent positions, velocities, and accelerations of the
three joints of the manipulator respectively.
Fig. 1. Three-DOF manipulator with rotational joints where external forces (collisions) are applied
during the motion of joints. The symbol “X” means that the force is perpendicular and towards to the
vertical plane of the manipulator.













6
The inertia matrix , Coriolis and centrifugal matrix and
gravity vector of the three-links manipulator from (1) are calculated [27], and
presented in Appendix A. The inertial, Coriolis, and the gravitational forces show the
dynamic coupling between the joints since these forces applied to of any link, affects the
actuator torques of the three joints and also the measured joints torques . If
the collision forces are high, then the actual motion of the joints is affected and so the inertia
or Coriolis forces could be estimated as external force  as it will be discussed and
shown later in Fig. 3 when they affect the torques, the measured joint torque and external
torque, of joint 3. These external forces affect the provisional inputs of the proposed NN
design, as it is discussed later, and particularly the measured joints torques, since the effect on
the velocities and accelerations of the links can be detected in the measured joint torque. It
should be noticed that the case presented in Appendix A with this selection of the joints, the
inertial forces of link 2 and link 3 have no effect on the measured torque of joint 1 but in other
manipulator anatomies might have effect.
In this paper, Kuka LWR manipulator, as a collaborative robot, is used for carrying out
all experiments and investigations. Its three joints namely A1, A3, and A5 joints are used, as
joint 1, joint 2, and joint 3 respectively. It should be stated that these joints are selected to
make the experiments easier and as we will see later these joints help to show clearly the
proposed collision detection method (easy to fix the external force sensor whether on link 1 or
2 or 3 when the proposed method will be checked as we will see later in section 6).
To further investigate the coupling, 18 experiments are executed using Kuka LWR robot.
Only joint A1 (joint 1) is dedicated to move with three different constant speeds, one constant
speed per experiment. During this motion, an obstacle (box)
†
with different weights is placed
in the path of the robot to investigate the effect of the collision with this obstacle on the
measured joint torques and the external torques that estimated by Kuka Robot Controller
(KRC), of the joints 1, 2 and 3 for three different configurations of the manipulator as shown
in Fig. 2. The joints angles of the Kuka robot for the three configurations are presented in
Table 1. A weight (0.445 kg) is attached to the last link (end-effector) to increase its inertia.
Although the robot can handle payload of 7kg, in the experiments this small weight (0.445
kg) is used as we just show the dynamic coupling as well as to perform the safely
experiments. In Appendix A, the maximum amplitudes of the measured joint torques
 and the external torques are presented at the time
when the obstacle collides with the manipulator to summarize the effect on the torques of the
joints 1, 2, and 3 for the 18 experiments. Fig. 3 shows the results of three experiments out of
18 with the specified three configurations of the links when the speed is 1.0 rad/s and the
obstacle weight is 1 kg (10 N). The effect of the collision on the measured/external torques of
the joints is represented by small blank circle as shown in Fig. 3.
†
Any other body could be used to do the collision with the robot since our main concept from these
experiments is just to show the dynamic coupling between the manipulator joints.

7
(a) (b) (c)
Fig. 2. An obstacle is placed in the path of the motion of joint A1 of the KUKA LWR robot (other
joints are fixed) with three different configurations (a) The first configuration of the robot, (b) the
second configuration, and (c) the third configuration. Pictures are taken from the top view.
Table 1 The joints angles of the kuka robot for the three configurations.
Configuration
Joints Angles (deg)
Comments
A1
A2
A3
A4
A5
A6
E1
First, Fig. 2(a)
0.0
0.0
0.0
0.0
0.0
0.0
90.0
---------------
Second, Fig. 2(b)
0.0
0.0
0.0
0.0
90.0
0.0
90.0
The end-effector is
perpendicular to the
other links
minimizing the effect
of the inertial forces.
Third, Fig. 2(c)
0.0
0.0
0.0
0.0
-90.0
0.0
90.0
(a)

8
(b)
(c)
Fig. 3. The effect of the collision between the obstacle and the robot during its motion on the measured
joint torques and the external torques of the three joints. (a) The first configuration of the robot, (b) the
second configuration, and (c) the third configuration.
From Fig. 3 and Table A.1 that presented in Appendix A, it is noted that with the first
configuration, the collision between the obstacle and the robot has high effect on the
measured torques and the external torques of the three joints, whereas for the second and the
third configuration of the robot, the effect is high on the torques of joints 1 and 2 and small
effect (approximately close to zero) on the measured/externals torques of joint 3 since the
inertial forces applied to the end-effector approximately have no effect or very small one.
With high speeds and heavy weights, the effect on the measured/external torques increases for
the three joints. This resulted effect on the measured torques and the external torques of the
joints during the collision proves the coupling that occurs between the three joints. The

9
collision between the obstacle and the robot affects the positions, velocities, and the
accelerations of the links, then this effect can be detected in the measured and estimated
external joints torques by KRC. The effect on the measured joint torque of joint 3 results from
the inertial, Coriolis, and gravitational forces of the end-effector which could be interpreted as
external force, so a careful definition of the collision threshold should be done to avoid false
collided link identification.
In addition, as noted from Table 1 and equations (A.1) to (A.3), presented in Appendix A,
the dynamic coupling is depending on the configurations of the manipulator links. In the first
configuration ( ), the inertial, Coriolis, and gravitational
forces of the links are affected and high compared with the ones in the second (
) or the third  configuration. Therefore,
the measured and estimated external joints torques are affected. It should be stated that in our
experiments with the obstacle we rely on the inertial forces to show the dynamic coupling
rather than the static force concerning the friction. However, we do not use a surface with
high friction to avoid any crash with the robot as well as for higher safety.
As depicted from the results, the effect of the collision on the measured torques and the
external torques of the joints 1 and 2 for the three configurations is high compared with the
effect on the torques of joint 3, which is low or it can be neglected. From that it can be
concluded that the collided link is link 2. The above investigation can conclude that the
collision in any link can appear in all the other links however if the external force is applied to
link  affects strongly the joints 1 to , while the effect of the external force on the joint 
is very small, which is close to the assumption of quasi-static motion taken in [28]. If the
collision thresholds are carefully selected taken into account the presented investigation, then
the external collision torques vector 



 up to joint is
expected to be above collision threshold while the estimated external collision torques


 after the th joint are very small and negligible. Therefore, the collided link
can be successfully identified. However, in some cases could confuse since the dynamic
coupling is configuration dependent and sometimes when there is collision on link, maybe
some of the external torques of joints 1 to  are very small and do not exceed the collision
threshold. Also, if the external torques of joint 3 from Table A.1 are compared with the
collision threshold that proposed by the model-based approach (0.5452 Nm), which is
presented later in section 6, it is found that the external torque in case of the high speeds (0.53
Nm) is very close to the collision threshold, which means that in other cases maybe this
external torque could exceed the threshold. Therefore, this could cause conflict in identifying
the collided link. Therefore, further investigations are advisable/needed for the identifying the
collided link criteria. It should be noted that this criteria, for identifying the collided link, are
applied easily if a single link is collided at a time, since the case where more than one
collision at a time is very rare or not occurring in practice. Also, the case where multi-links
are collided at the same time is not tackled in the paper (might be unresolved and rare case to
happen).
It should be stated here that the previous experiments, done to show the dynamic coupling
between the joints, use one joint motion which is a simple case and as presented from the
results the collision that occurs on link 2, affects the torques of the three joints which shows
the dynamic coupling. This coupling is taken into account for the design of the NN as it is
presented in the next section.

10
4. The Coupled Neural Network
Multilayer neural network is a powerful tool for the non-linear systems identification. It can
adapt itself by changing the network parameters in a surrounding environment and can easily
handle imprecise, fuzzy, noisy, and probabilistic information [29]. The NN derives its
computing power through its massively parallel distributed structure and its ability to learn
and therefore to generalize. It was proven that the neural network could approximate any
complex (large-scale) linear or non-linear function given appropriate training data [30,31]. It
also offer useful properties and capabilities such as nonlinearity, input–output mapping and
adaptivity, function approximation and generalization [30]. NN has a kind of universality [32]
e.g. approximation of smooth batch data containing the input, output and possibly gradient
information of a function [33], approximating the derivatives of a function [34]. NN has a
vital role in identification of the dynamic systems and fault detection since it not only can be
used to detect the occurrence of the fault but it also provides a postfault model of the robotic
manipulator. This postfault model can be effectively used to isolate and identify the fault and,
if possible, for accommodation of the failure [35].
The NN is selected/proposed for human-robot collision detection and then identify the
collided link due to many reasons. The methods relying on the dynamic model require
heavy/complex inverse dynamics computation [36]. These methods suffer from the high
uncertainty and inaccuracy in the model parameters identification [18,37–39]. This
uncertainty and the inaccuracy in the model cause estimation error which increases the false
alarms possibility. Therefore, effectiveness of the collision detection method is negatively
affected as well as the smoothness and the continuation of the robot movement [37]. The
building of a model and parameter identification is time consuming and should be done for
each manipulator [39]. The dynamic parameters of the model,,, and, are
not available/known in most of the current robots. The NN is noise free [18] and it does not
copy given data since it can approximate any function and it has the ability to generalize. The
NN can approximate the correct output for new inputs which are not used for the training
process. Also, the NN can work effectively with the collisions which are highly random. In
addition, the inverse dynamics problems of the robot is solved using the NN as presented in
[40–42]. All these characteristics/advantages of the proposed coupled NN are
proved/investigated during the next sections; section 5 and section 6.
From the previous dynamic equations study and the experimental work as well as our
previous papers [21,22], where a planar motion was considered, the main inputs for the neural
network coupling the three joints that give the good performance and the minimum mean
squared error (MSE) are the current position error 
 between the desired and actual joint
position, the previous position error 
, the actual joint velocity  and the measured
joint torque  where. In this paper, space motion is considered, therefore, the
effect of the gravity  applied to each link is used as input to the NN too. The actual joint
velocity is given by KRC, but it can be derived from the numeric differentiation of the
measured joint position variables as in [21]. We do not use the joint acceleration as an input
to the NN due to high noise and as presented in [43] using the joint accelerations does not
improve the performance.
The proposed NN can be applied to the service and collaborative robots that have joints
torque sensors such as KUKA LWR, HC10, and FRANKA EMIKA [44–46], or where the
estimations of the joints torques are available indirectly [47]. Currently, most of the industrial
manipulator do not have joints torque sensors, however the proposed method could be
generalized and applied in any other robot if the measurements or estimation of the joints

11
torques are available. For the robots that do not have intrinsic joint torque sensors, the motor
currents can be used to estimate these joint torques [48,49] and this feature is available in
most types of the robots. Also, the estimation of the actual joint torques, based on a nonlinear
adaptive impedance control law, proposed in [17] could be used. In the near future, the trends
in robotics are to have joint torque sensors.
Fig. 4. Experimental setup with Kuka LWR manipulator.
A sinusoidal motion with time-variable frequency is commanded to the three joints of
KUKA LWR robot. The collisions are performed randomly by a human hitting the robot with
his hand on the end-effector (link3), the link 2, and the link 1 as shown from Fig. 1 and Fig. 4
with the directions of the external collision forces where. Fig. 5 and Fig. 6
present the waveforms of the selected inputs to the NN without and with collisions
respectively.
From Fig. 5 and Fig. 6, it is clear that sudden changes and spikes in the variables
appeared in case of collisions compared with the corresponding variables without collisions.
In Fig. 5, the small spikes in the position error diagrams, marked by “d”, represent the
changes in the direction of the joint motion. In Fig. 6, the spikes in position error, actual joint
velocity, and measured joint torque diagrams, which are signed by the small blue circles,
represent collisions on the link 1. The spikes signed by the small blank rectangles show
collisions on the link 2, and the spikes signed by the small blank circles mark collisions on the
end-effector. The other small spikes in the position error diagrams, which are similar to the
spikes referred by “d” in Fig. 5, are representing the changes of direction of motion of the
joints. Also, the small spikes in the actual joint velocity diagrams whether in Fig. 5 or Fig. 6,
which are not referred by any symbol, are coming from the inertial forces, as it was discussed
in [21]. It should be stated here that the ranges of the applied accelerations are [0,
1.0740], [0, 2.1584], and [0, 0.7107] for joints 1, 2, and 3
respectively.
Joint 1 (A1)
Joint 2 (A3)
Joint 3 (A5)













12
(a) Joint 1 (b) Joint 2 (c) Joint 3
Fig. 5. The waveforms of the main inputs for the NN without collisions
‡
.
(a) Joint 1 (b) Joint 2 (c) Joint 3
Fig. 6. The waveforms of the main inputs for the NN with collisions
§
.
From the previous discussion in section 3, where the coupling are proved from the
dynamic equations and collision with the obstacle, it can be deduced here that the motion of
these three joints is a coupled system, where any external force applied to any link affects the
position, velocity, acceleration, and torque of the joints. If a collision occurs on the end-
‡
All the horizontal axes of the diagrams are repres enting “Time (s)”.
§
All the horizontal axes of the diagrams are repres enting “Time (s)”.

13
effector, the effect appears clearly on the position error, the actual joint velocity, and the
measured joint torque of the three joints. But when there is a collision on link 2, the effect on
the position error, the joint velocity, and the measured joint torque of joints 1 and 2 appears
clearly, whereas a small effect occurs on these three variables
**
of joint 3 due to the inertial
force applied to the link (this effect does not appear here because it is small and because of
the scaling of the diagrams). When there is a collision on link 1, the effect on the position
error, the joint velocity, and the measured joint torque of joint 1 appears clearly, whereas a
small effect occurs on these three variables of joints 2 and 3 because of the inertial forces
applied to the links (this effect does not appear also here because it is very small). It should be
noted that the spike “s” in the position error, the joint velocity, and the measured joint torque
of joint 2 (Fig. 6(b)), means that the collision occurs on link 1 and the perpendicular distance
between this force (collision) and joint 1 axis is very small. Due to that, the effect of this
collision approximately does not appear on the position error, or the joint velocity, or the
measured joint torque of joint 1 (approximately zero effect) but this collision has effect on
these three variables of joint 2 depending on the configuration of the robot at this moment.
The corresponding spikes to “p1” and “p2”, which are in the position error and measured
joint torque of joint 1 (Fig. 6(a)), are not appeared clearly in the actual joint velocity diagram
of joint 1 due to the scaling of this diagram. Also, the collisions occurring whether on the end-
effector or link 2 and their effects do not appear clearly on the position error, the actual joint
velocity, and the measured joint torque diagrams of joint 1, this happens because of the
scaling of these diagrams. Although, the scaling of the corresponding diagrams used for each
joint is different, the small effects that stated and discussed above will not appear also if the
same scaling is used.
From the previous figures and discussion, we can summarize that the inputs of the NN are
rich signals which are important to the NN performance, as we will see later. Also, the
proposed NN accept one point from each of the previous diagrams at a time. It should be
noted also that the approach, introduced in the previous two paragraphs for identifying the
occurring collisions whether on the end-effector, link 2, or link 1, is done manually by
ourselves just to show the importance of the dynamic coupling between the joints for the
design/implementation of the proposed coupled NN and this approach is not the
practical/effective way compared with the proposed NN.
It is known that, the feedforward neural network is the simplest and non-complex type of
artificial neural networks [30,50]. One multilayer feedforward neural network, in which the
three joints are coupled/combined, is designed and trained. Three layers are used to compose
the structure of the NN as shown in Fig. 7; the input layer, the non-linear hidden layer
(hyperbolic tangent activation function) and the linear output layer that calculates the external
torques 


 of joints 1, 2 and 3 respectively. The estimated external
torques , by KRC, are used only for training the network. Alternatively, the
external collision force can be measured by any external force sensor and then transformed to
the joint torques via the Jacobian.
The training errors  and  are given by

 
 
 (2)
**
These three variables in this paragraph are indicating to the position error, actual joint velocity, and
measured joint torque.

14
Fig. 7. The multilayer feedforward neural network coupling the three joints.
5. NN Training and Validation for Safety
Levenberg-Marquardt training algorithm is used to train the network and do the work in fast
and stable way. Levenberg-Marquardt algorithm is a type of the second-order optimization
techniques that have a strong theoretical basis and provide significantly fast convergence and
it is considered an approximation to Newton’s Method [51,52]. Compared with other training
methods, LM learning is used because it can trade-off between the fast learning speed of the
classical Newton’s method and the guaranteed convergence of the gradient descent [51,53].
LM algorithm always suits to large data sets and converges in less iterations and in shorter
time than the other training algorithms.
The proposed collision detection method is implemented using the KUKA LWR IV
manipulator shown in Fig. 4. KUKA LWR robot is characterized by an extremely light
anthropomorphic structure with 7 revolute joints and driven by compact brushless motors via
harmonic drives. All the joints are equipped with position sensors on the motor and link sides,
and with joint torque sensor. The KR C5.6 lr robot controller unit, together with the so-called
Fast Research Interface (FRI) [54], is able to provide (at a 1 msec sampling rate) the link
position , velocity , and joint torque  measurements, the gravity , and an estimation of
the external torques.
A sinusoidal motion, with variable and constant frequencies, is commanded
simultaneously on the three joints of KUKA LWR robot when there is effect of the gravity.
The collected data are used for training the NN where the variable frequencies of the three
joints motion are linearly increasing (experiments 1, 2 from Table B.1 in Appendix B). These
frequencies produce angular velocity up to 0.685 rad/s for joint 1, 0.934 rad/s for joint 2 and












 

 







15
0.56 rad/s for joint 3. It is also known that the larger data, which represent different cases,
used for training the NN, the better performance and generalization of the trained NN will be.
We aspire to have better trained NN performance in approximately all different cases
(whether variable or constant frequencies motions), so the three constant frequencies are used
(experiments 3 from Table B.1 in Appendix B). It should be noted here from the
specifications of KUKA LWR robot that every joint A1, A3, and A5 can move alone with
speed until 1.96 rad/s [55], but when these joints move simultaneously in the range from -90
deg to +90 deg for each joint, the maximum permitted speeds are found experimentally as
0.81 rad/s for joint 1, 1.15 rad/s for joint 2, and 0.65 rad/s for joint 3. The experiments for the
NN training process are restricted to be in the joints range from -90 deg to 90 deg and not all
the entire workspace of the joints to show the performance/generalization of the trained NN
outside of the training process range and in different conditions, as well as the possibility to
reduce the training time and the human burden. The generalization ability of the trained NN is
discussed later in section 6. During our experiments, there is no possibility for robot self-
collision because of the joint limits.
The training data are divided into three sets: in the first set the robot joints perform the
motion with variable frequencies without any external force (collision) applied to the robot
body. In the second set the same motion is performed with the user performing random
collisions suddenly and stochastically by the human hand at different locations on the three
moving links. The third set contains data from the three experiments that done using the three
constant frequencies. In each experiment, the user performs collisions suddenly and
stochastically with his hand at known locations on the three moving links with different
configurations each time. During the experiments the robot is commanded to move with
position control mode and no reaction strategy is implemented. The performed collisions are
applied momentarily and the safety of the human operator during this experiment is taken into
consideration. The protocol for these experiments is illustrated in Appendix B.
Using the data with and without collision, the NN with 15 inputs is designed and trained.
The total number of input-output pairs collected from the experiments and used are 349572 at
a frequency of. From these data 90% are used for training, 5% for validation
and 5% for testing and this split-up occurs randomly using Matlab Toolbox. It should be
noted that it is not restricted to use these 349572 input-output pairs exactly, since this number
is found after doing our experiments. Therefore, less than or more than this number of input-
output pairs can be used for training the NN according to the collected data from the
experiments provided that the performance of the NN is high and overfitting is avoided. In
our case, the NN performance is high by using this number of input-output pairs for its
training as it is presented in section 6. Since Matlab Toolbox is used for the NN training
process, it is found experimentally that the NN has high/better performance after using 90%
from the data for the training. The structure of the designed NN is determined in the following
way; after trying many different number of hidden neurons (40, 50, 70, 90, 120, 150, and 170)
with also many different weights’ initializations, it is found that the best number of hidden
neurons are 150 and the number of iterations are 1000 that give us the main factors; the
minimum MSE (0.034677) and the adequate collision thresholds, that will be presented in the
next subsection. The training process is very fast and stable, towards getting the very small
error and MSE, and implemented offline using Matlab on AMD Ryzen 5 1600 Six-Core
processor. The NN training convergence is shown in Appendix C. It should be noted that the
methodology followed for the NN structure determination and training was discussed in detail
in our previous paper [21], see “Appendix A” in that paper. As the NN training occurs offline
so it is no problem if the training happens using the computer of KUKA robot or any other

16
external device such what we use (AMD Ryzen). Moreover, this does not affect the
performance of the NN since after the NN training is completely finished, the trained NN is
implemented online on KUKA robot controller to be checked and evaluated as we will see
later in section 6. The main point here to use the AMD Ryzen system is just to finish the
training quickly as it has high computational power.
The trained NN is evaluated with the same data-set that is used for the training to get an
insight about the approximation. The difference between the external
torquesgiven by Kuka Robot Controller (KRC) and the external torques



estimated by the trained NN is illustrated in Fig. 8. It is clear that the
approximation error of the collision external torque for each joint is higher than the error of
contact-free motion. In case of contact-free motion, the average of the absolute error values is
very small and is 0.1060 Nm, 0.1333 Nm, and 0.0864 Nm for joint 1, joint 2 and joint 3
respectively, whereas in case of the collision, the average of the absolute error values is
higher and is 0.1350 Nm, 0.1765 Nm, and 0.1012 for joints 1, 2, and 3 respectively.
(a) Joint 1 (b) Joint 2 (c) Joint 3
Fig. 8. Neural Network approximation error.
5.1. Determination of Collision Thresholds
Since the safety is crucial factor in HRI, so the collisions should be detected and the collided
link should be identified in order to achieve a safe human-robot collaboration. In the HRI
literature, collision thresholds were defined taking into account the human safety as well as
the minimization of the false collisions detections for smooth and continuous HRI. Collision
thresholds were determined based either on contact force or on joint torque. Based on contact
force, Shujun Lu et al. [18] defined the collision threshold as a value below the contact force
that parameterizes the unified pain tolerance limit of a human, which was determined in [4].
Based on joint torque, Haddadin et al. [14] defined the threshold as 10% of the maximum
nominal torque of the robot. Dimeas et al. [20] proposed the threshold as the maximum
approximation error, between the external torque mapped from an external force sensor and
the one given by the fuzzy system, in contact-free motion. In [17], the collision threshold was
identified based on the characteristics of the normal distribution. In this paper, the collision
threshold for joint 2 and joint 3 is defined in [21,22] as the maximum of the absolute values of

17
the approximation error, between the external torque given by KRC and the one given by the
NN, of the contact-free motion (Fig. 8, upper diagrams) but for joint 1 it is chosen as the
maximum of the absolute values of the approximation error plus the average of the estimated
external torque of the contact-free motion. By providing the dataset without any collisions to
the trained NN, the resulted thresholds values are ,
 and  for joints 1, 2 and 3 respectively and the collision
is assumed, when
 for ith joint. It is found also that our proposed thresholds are
close to 10% of the maximum nominal torques of the robot when they are compared by the
thresholds proposed by Haddadin et al. [14].
5.2. Collision Detection, Collided Link Identification, and Safety Considerations
ISO standards provide the safety requirements for the collaborative industrial robot systems in
the work environment, where the maximum permissible forces for quasi-static and transient
contact, maximum transferred energy, and robot speed limits during transient contact are
defined. From ISO/TS 15066 [3], the worst case for the safety reasons is proposed as 65 N
which is the maximum permissible force can affect the face (make minor injury or pain) of
the human during quasi-static contact. For the configurations of KUKA LWR manipulator
which are the worst case of the human, this force can cause a maximum external torque of
 for joint 1,   for joint 2, and   for joint
3. In this paper, the collision thresholds for the three joints  are much
lower than the maximum external torques calculated by the
maximum permissible force provided by ISO since  represents 7% of, 
represents 5% of, and  represents 10.4% of. This means that the human-
robot collaboration is in the safe side, since the collisions are detected in the safe region
where the contact is detected before causing minor injury or even pain and once the collision
is detected, any retraction strategy can be built based on our results or the robot could be
stopped in the worst case. In addition to the previous point of the ISO, it should be taken into
account for safe human-robot collaboration, no significant delay should be found between
registering a torque and providing a subsequent robot reaction, and also only a single collision
happens/occurs at a time since the case where more than one collision at a time is very
rare/impossible case to happen. It should be noted here that the above defined thresholds
represent critical values as discussed in the previous paper [21], since if the thresholds are
chosen to be below the defined ones, many false positives (FP) collisions alerts will be
provided even any actual collision did not happened and this affects the smoothness and the
continuity of the HRI. On the other hand if the thresholds are chosen to be above the defined
ones, the collision detection time will increase and may be the collisions occur close to the
joints which give external collision torques close to these thresholds which case affects badly
the safety of the human-robot collaboration. As shown from the results of our previous paper
[21], where the collision was detected using 1-DOF manipulator and the best collision
threshold was chosen, the number of the false positive collisions was low (8%) which means
the effect on smoothness of the collaboration is very small/negligible.
To examine more carefully the approximation errors, the external torque resulted from
KRC and the external torque estimated by the NN for the three joints are compared as shown
in Fig. 9. The spikes signed by small dots are representing the collisions on the link 1, the
spikes with small blank rectangles are representing the collisions on the link 2 and the spikes
signed by small blank circles are representing the collisions on the end-effector. These results

18
show the coupling between the joints as discussed previously since any collision that occurs
to a link, affects the external torques of the three joints.
As discussed previously in section 3, the collision applied to link, affects the joints 1 to 
(above threshold) whereas the effect of this external force (collision) on the joint  is very
small and can be negligible (under threshold) so the collided link can be identified easily.
From our results shown in Fig. 9, the collided link can be identified easily, based on the
discussion presented in section 3, using the following algorithm:
When 
AND 
AND 
  THEN “link 3 (end-effector)” is collided.
When 
AND 
AND 
  THEN “link 2” is collided.
When 
AND 
AND 
  THEN “link 1” is collided.
It should be noted here that the spikes referred by the symbol “d” in the joint 2 and joint 3
diagrams (Fig. 9(b), Fig. 9(c)) are the effects of the collisions on link 1 on the external torques
of the joints 2 and 3. From our investigation of the coupling and from the literature, these
spikes should be small and should not exceed the collision threshold for joint 2 or joint 3, but
this is a faulty result of the NN that affects its performance measure, as it is presented in
section 6. The previous criteria, for identifying the collided link, is applied easily if a single
link is collided at a time. In the case of simultaneous multi-link collisions, the proposed
method can easily understand that there is a collision but it cannot determine if one or two
collisions are happened. In fact, the case of simultaneous multi-link collisions is very rare.
(a) Joint 1 (b) Joint 2 (c) Joint 3
Fig. 9. The external collision torques from KRC and NN for the three joints.
6. Effectiveness of the Coupled Trained NN
In this section, the proposed method is evaluated and tested online in the experimental setup
by commanding the KUKA robot to perform three joints motion simultaneously with constant
joint velocity that is normal in robot applications. The effectiveness of the trained NN is
tested out of the training joints range to show its generalization performance. An external

19
force sensor (ATI F/T Nano 25) is used to compare its reading that is converted into joint
torques via Jacobian with the external torques estimated by the trained NN and KRC.
The trained NN is evaluated online with joints speed equal to 0.5 rad/s for three cases of
the collisions (experiments 4 from Table B.1 in Appendix B). In every case, the force sensor
is fixed in a different link (link1, link2, and end-effector) and the collisions occur only to the
force sensor perpendicularly to the link surface. Fig.10 shows a case when the force sensor is
fixed at the end-effector (link 3). Fig. 11 to Fig. 13 illustrate the comparison between the three
external collision joint torques from KRC, the trained NN and the signal of the external force
sensor that mapped to the joints torque.
Fig. 10. Collision on the force sensor fixed on the end-effector.
(a) Joint 1 (b) Joint 2 (c) Joint 3
Fig. 11. The three external collision joint torques resulted from KRC, NN and using the force sensor
with collision on the end-effector only.

20
(a) Joint 1 (b) Joint 2 (c) Joint 3
Fig.12. The three external collision joint torques resulted from KRC, NN and using the force sensor
with collision on link 2 only.
(a) Joint 1 (b) Joint 2 (c) Joint 3
Fig. 13. The three external collision joint torques resulted from KRC, NN and using the force sensor
with collision on link 1 only.
Fig. 11 shows the results of a collision on the end-effector, where the three external joints
torques are affected by this collision (above the threshold) and the approximation between the
corresponding external torques is good. Before and after collision, the external joint torques
measured by the external sensor are approximately zero and during the sudden collision are

21
increased quite rapidly. This wave form illustrates the collision phenomenon most accurately,
since the robot dynamics (such as the inertial forces, Coriolis and centrifugal forces, and
frictions) do not affect these measurements as it is the case in the estimated external joint
torques by KRC. The measurements by the force sensor are mapped to the external joints
torques via Jacobian. In Fig. 12, a collision on link 2 is illustrated, where the external torques
of joints 1 and 2 are highly affected by the collision, whereas joint 3 external torque has small
effect at time when this collision occurs and this effect sometimes does not appear clearly if
the collision force is small. The approximation between the three external torques is good.
From Fig. 13 where a critical collision on link 1 is shown, the joint 1 external torque is highly
affected by the collision, whereas small effect appears to the external torques of joints 2 and
3. The good approximation between the external joint torques estimated by KRC and
measured by external force sensor with the estimated by the trained NN proves the success of
the proposed approach.
The trained NN detects the collisions very fast. The actual collision detection time by the
proposed method is calculated as the elapsed time from the detection of the collision by the
external force sensor to the moment when the estimated external torque by the trained NN
exceeds the collision threshold. The actual collision detection time by the proposed method
and the model-based approach for joints 1, 2 and 3 is presented and compared in Table 2. The
model-based approach is referred to the estimation of the external torques by KRC. In fact,
these external torques were obtained by the build-in model of the KUKA robot that uses a
state feedback controller [56]. From Table 2, although the proposed collision thresholds by
the trained NN are slightly higher than the ones proposed by the model-based approach, the
collision detection time by the trained NN is lower which prove that the collisions are
detected very fast by the proposed NN. Also, these results from the trained NN are promising
for safe HRI. The latencies of the sensors and other hardware and their relation with this
collision detection time, which is presented in Table 2, are presented in Appendix D.
From the results shown from Fig. 11 to Fig. 13, it can be inferred that the trained NN is
easily identifying the collided link using the same algorithm presented in subsection 5.2,
since when the external collision torques of joint 1 only exceed the collision threshold (Fig.
13(a)), it means that the collisions occur on the link 1, whereas when the collision torques of
joints 1 and 2 exceed the threshold (Fig. 12(a), Fig. 12(b)), it means that the collisions occur
on link 2. When the collision torques of the three joints exceed the threshold (Fig. 11), it
means that the collisions occur on the link 3 “end-effector”. As discussed in section 3 that in
some cases maybe a conflict occurs in identifying the collided link, since the dynamic
coupling is configuration dependent. Sometimes, when there is collision on link, maybe
some of the external torques of joints 1 to  are very small and do not exceed the collision
threshold. Also, in case of the high speeds, maybe some of the external torques of joints 
to  exceed the collision threshold. Therefore, it will be difficult to identify exactly the
collided link. In these cases, further investigations are advisable/needed for identifying
correctly the collided link.
It should be noted here that the deviations resulted from the external joint torque mapped
by the force sensor during the collision occurrence come from factors such as the accuracy of
the attachment of the force sensor on the manipulator links, the accuracy of the perpendicular
distance between the force and the joint, force sensor bias and calibration. However, these
deviations are very small and do not affect the measurements.

22
Table 2 Collision detection time obtained by the proposed method and by the model-based
approach for each joint using the force sensor as the reference signal for the correct detection.
To confirm the validity and effectiveness of the proposed method under a wide range of
operating conditions and to acquire a performance measure, another 75 trials of collisions
(true negative, TN) are evaluated with various magnitudes, points of collision, directions and
different velocities of the motion (experiments 5 from Table B.1 in Appendix B). Table 3
provides the performance in terms of the number of the correctly detected collisions (true
positive, TP), the number of false negatives (FN), which is the number of collisions not
detected by the method and the number of the false positives (FP), which are the collision
alerts provided by the method when there is not an actual collision. It should be noted that we
perform many collisions (75) in different conditions to be sure that the proposed method is
effective. Therefore, using less than or more than this number of collisions can be used to
check the coupled trained NN, but if this number is high, it will be better for deep
investigation for the efficiency of the trained NN.
Method
Joint
Threshold
(Nm)
Collision Detection Time (msec)
Collision on
end-effector
Collision on link
2
Collision on link
1
The
Coupled
Trained
NN
Joint 1
1.6531
6.0
28.0
24.4
Joint 2
1.5258
13.6
16.2
----------
Joint 3
0.8109
6.1
-----------
----------
Model-
based
approach
Joint 1
1.5237
11.9
28.2
27.1
Joint 2
1.1460
22.0
15.4
----------
Joint 3
0.5452
10.0
-----------
----------
External
Force
sensor
Joint 1
The signal from the external force sensor is taken as the reference
one.
Joint 2
Joint 3

23
Table 3 Summary of the performance of the proposed method obtained for the different
collision scenarios and angular speeds of the joints.
Collision
Scenario
Joint
Trained Neural Network Method
Performance Measure (%)
TN
TP
FN
FP

 *100%
Average
for the
three joints
Collisions on
the end-
effector
Joint 1
25
19
6
3
64 %
84 %
Joint 2
25
25
0
2
92 %
Joint 3
25
25
0
1
96 %
Collision on
link 2
Joint 1
25
16
9
2
56 %
72 %
Joint 2
25
24
1
2
88 %
Joint 3
-----††
-------
--------
--------
---------
Collision on
link 1
Joint 1
25
21
4
3
72 %
72 %
Joint 2
-------
--------
--------
--------
----------
Joint 3
-------
--------
--------
--------
----------
Average Percentage
Case
1
86.7
%
13.3
%
8.7 %
78.0 %
Case
2‡‡
95.3
%
4.7%
8.7 %
86.6%
From Table 3, it is noted that when the collisions occur on the end-effector, the proposed
method succeeds to detect the collisions effects on the estimated external torques of the joint
2, joint 3 (high rate, 92 % and 96 % respectively), and joint 1 (low rate, 64 %). This is
justified because when the collision occurs on the end-effector, the effect appears clearly on
the external torques of joints 2 and 3 but sometimes it has small effect on joint 1 torque based
on the configurations of the links where the perpendicular distance between this external force
(collision) and joint 1 axis is very small or in another meaning, this collision occurs close to
the axis of joint 1. This low effect cannot exceed the collision threshold of the joint 1 so it
cannot be detected. This small effect of the collision is not detected by joint 1 but in the same
time the collision is detected by joint 2 or joint 3 so there is no effect on the safety of the
††
The symbol “-------” means that the proposed method does not detect any effect of collision on the
joints after th joint such +1, +2 and so on when the collision happens on link “” and this is very
good which prove the success of the proposed method. The output of the NN in this case is the external
joint torque and it does not exceed the collision threshold.
‡‡
This case, where both the estimated and real contacts are under threshold, is ignored and not
counted/considered in the performance measure of the proposed method (This case is discussed in
detail in the text).

24
human-robot cooperation. However, this case/situation affects the inference mechanism of the
collided link identification, since as discussed previously, the collision applied to link,
affects the joints 1 to  and the joint  is approximately zero. Therefore, identifying the
collided link in this case will be more complex and need more investigations.
When the collisions occur on link 2, the trained NN succeeds to detect the collisions
effects on the external torques of the joint 2 (high rate, 88 %) and joint 1 (low rate, 56 %)
only and not joint 3. The trained NN succeeds with low rate to detect the collisions effects on
the external torques of the joint 1 since these effects are sometimes small and are not
appeared clearly as discussed above. When there are collisions on link 1, the trained NN
succeeds with high rate (72 %) to detect the collisions affect the external torque of joint 1
only and not joints 2 and 3.
The number of the false positive (FP) collisions is low (8.7 %) which means that the
proposed method has less sensitivity to the external disturbances and unmodelled parameters
and the effect on the continuation of the robot movement is very small. The proposed method
can identify easily the false positive collisions from the correct/real ones, based on the criteria
of the collided link identification previously discussed, if the detected false positive collision
has no effect on the other joints. The case, when the FP collision occurs on link 1, is
considered slightly difficult for the proposed method to differentiate between if it is real
collision or FP collision. In general, the detected false positive collisions are very low (8.7
%). The false negatives (FN) collisions, which are the collisions not detected by the proposed
method, are low (13.3 %). In fact, these FN collisions include two types of the collisions. In
the first type, the estimated contacts by the trained NN are under the threshold whereas the
real ones are above the threshold. This type of FN collisions affects the performance measure
of the trained NN, but in our case this type represents 4.7 % which is very small. In the
second type, both of the estimated contacts and the real ones are under the threshold. This
type of FN collisions happens/occurs based on the configurations of the links of the
manipulator (sometimes the perpendicular distance between the collision and the joint axis is
very small or in another meaning the collision occurs close to the axis of the joint. Therefore,
a very small external joint torque is resulted). It represents 8.6 % and it is also safe since these
collisions are under the threshold. Therefore, the performance measure of the trained NN
cannot be affected by this type of FN collisions, but in the same time identifying the collided
link become more complex and ill-defined problem. For this case, further investigations are
advisable/needed.
From the good presented performance (86.6%, if FN collisions, in which both real and
estimated contacts are under threshold, are ignored) of the proposed method and the safety
considerations stated in section 5, we can summarize that the proposed method has a
significant contribution to the safe cooperation between the human and the robot.
Since the joint motion for collecting data used for training the NN and NN evaluation
limited in the range from -90 deg to 90 deg for each joint, two experiments (experiments 7
from Table B.1 in Appendix B) are done out of this range (the first from 90 deg to 115 deg
and the other from -115 deg to -90 deg range) with random collisions on the three links using
a constant speed (0.5 rad/s) which is normal in robot applications to show the generalization
of the trained NN and its capability to estimate collisions and external torques with data that
are not used in the training process range. The results are shown in Fig. 14 and Fig. 15.

25
(a) Joint 1 (b) Joint 2 (c) Joint 3
Fig. 14. The two external collision torques from KRC and NN during three joints motion out of range
(from 90 to 115 deg).
(a) Joint 1 (b) Joint 2 (c) Joint 3
Fig. 15. The two external collision torques from KRC and NN during three joints motion out of range
(from -115 to -90 deg).
In the figures, it is observed that the trained NN approximates adequately the external
torques provided by KRC but the error is larger than when the motion is in the range of the
workspace where the NN is trained.
The performance measure of the trained NN out of training range joints motion and under
a wide range of operating conditions is determined (experiments 8 from Table B.1 in
Appendix B) using another 150 trials of collisions (TN) which are evaluated with various
magnitudes, points of collision, directions and different velocities of the motion as shown in

26
Table 4. The joints motion ranges are; in the positive range, from 90 deg to 115 deg for joints
2 and 3 and 90 deg to 165 deg for joint 1 whereas in the negative range, from -115 deg to -90
deg for joints 2 and 3 and -165 deg to -90 deg for joint 1. From the specifications of KUKA
LWR robot [55], the range of motion of joint 2 or 3 is  deg and of joint 1 is .
However, in the experiments, the limits are not reached since it is found experimentally that
the robot stops and also for safety reasons; to avoid any unpredictable movement since
variable frequencies are used in the experiments.
Table 4 Summary of the performance of the proposed method for out of training range joints
motion.
Range of
Joints
motion
Collision
Scenario
Joint
Coupled Trained NN
Performance
Measure (%)
TN
TP
FN
FP


*100%
Average
for the
three
joints
Positive
range
(90 deg to
115 deg for
joints 2 and
3 and 90
deg to 165
deg for joint
1)
Collisions
on the end-
effector
Joint 1
25
16
9
2
56 %
61.3 %
Joint 2
25
17
8
1
64 %
Joint 3
25
18
7
2
64 %
Collision
on link 2
Joint 1
25
15
10
2
52 %
54.0 %
Joint 2
25
17
8
1
64 %
Joint 3
------
-----
-----
2
§§
8 %-
Collision
on link 1
Joint 1
25
17
8
1
64 %
56 %
Joint 2
------
-----
-----
1
-4 %
Joint 3
------
-----
-----
1
-4 %
Average Percentage
Case 1
66.7
%
33.3
%
8.7
%
58 %
Case
2***
81.4
%
18.6
%
8.7
%
72.7%
Collisions
on the end-
effector
Joint 1
25
16
9
1
60 %
65.33 %
Joint 2
25
16
9
0
64 %
Joint 3
25
19
6
1
72 %
Joint 1
25
15
10
2
52 %
§§
Negative values such “ -8%” means that the proposed method detects effect of collision by fault after
the th joint such +1, +2 and so on when the collis ion happens on link “”.
***
This case, where both the estimated and real contacts are under threshold, is ignored and is not
counted/considered in the performance measure of the proposed method.

27
Range of
Joints
motion
Collision
Scenario
Joint
Coupled Trained NN
Performance
Measure (%)
TN
TP
FN
FP


*100%
Average
for the
three
joints
Negative
range
(-115 deg to
-90 deg for
joints 2 and
3 and -165
deg to -90
deg for joint
1)
Collision
on link 2
Joint 2
25
18
7
1
68 %
58 %
Joint 3
------
-----
-----
1
-4 %
Collision
on link 1
Joint 1
25
15
10
0
60 %
56 %
Joint 2
------
-----
-----
1
-4 %
Joint 3
------
-----
-----
-----
---------
Average Percentage
Case 1
66
%
34
%
4.67
%
1.33 %6
Case 2
82%
18%
4.67
%
77.33%
Overall performance measure of the
trained NN for the out of range
joints motion
Case 1
 59.7 %
Case 2
 75.0%
From Table 4, the overall effectiveness of the trained NN out of training range joints
motion (positive and negative range) is good (75.0 %, when FN collisions, in which both real
and estimated contacts are under threshold, are ignored) and presents satisfactory
performance and shows its generalization ability. The performance measure of the trained NN
in the positive range is a little bit lower than in the negative range because the configuration
and the motion of the links in the positive range are more complex/difficult compared with
the negative range. The configuration and motion of the links whether in the positive or
negative range are complex/difficult compared with the corresponding ones in the range
where the NN is trained so this is why the effectiveness of the proposed method is lower here
compared with the case in the NN training range. The number of the false positive collisions
is low (8.7 % in the positive range, and 4.67 % in the negative range) which means that the
trained NN has less sensitivity to the external disturbances and unmodelled parameters and
there is no effect on the smoothness of the robot movement. The FN collisions, where the
estimated contacts by the trained NN are under the threshold and the real ones are above the
threshold, are low whether in the positive range (18.6%) or in the negative range (18%). The
FN collisions, where both the estimated contacts and the real ones are under the collision
threshold are also low whether in the positive range (14.7 %) or in the negative range (16%).
Therefore, the cooperation between the human and the robot is in the safe side. It should be
noted here that only 5 deg is not used, as stated before for safety reasons, whether in the
positive or the negative side of the workspace of each joint motion. The performance of the
trained NN is very good in the training range and good/satisfactory outside of this training
range. This effective/followed methodology reduces the NN training time and the human
burden. Therefore, we can summarize that the performance of the proposed method, in the
entire workspace of the three joints motion of the manipulator, is very good and its results are
promising.

28
7. The Uncoupled/Independent NNs
In this section, the investigation of the uncoupled NNs, one NN for each joint, is presented.
The following subsection presents the design and the training process of the uncoupled NNs,
whereas the second subsection provides the performance/effectiveness of the uncoupled
trained NNs.
7.1. Design and Training of the Uncoupled NNs
One independent NN is designed, as shown in Fig. 16, and trained for each joint using the
same data for training the NN coupling the three joints (experiments 1, 2, and 3 from Table
B.1 in Appendix B). The training process is applied using Matlab on an Intel(R) Core(TM) i3-
6100 CPU @ 3.70GHz processor
†††
. The NNs training convergence is shown in Appendix C.
The collision threshold for each joint is defined as it was done before for the one NN coupling
the three joints. It is found that threshold values are
,  and  for joints 1, 2 and 3 respectively.
Fig. 16. The multilayer neural network for each joint, where.
7.2. Performance of the Uncoupled Trained NNs
The experimental scenario for checking the uncoupled trained NNs is the same with the one
followed in the coupled trained NN. Three experiments are executed, using joints motion in
the NNs training range (-90 deg to 90 deg), to check the uncoupled/independent trained NNs
and also using constant joint speed (0.5 rad/s) (experiments 4 from Table B.1 in Appendix B).
†††
It should be noted that the hardware/processor used for the training process of the coupled
NN is changed in the case of the uncoupled ones because it is not available in this time. In
fact, the change of the hardware cannot affect the results and also the comparison between the
coupled NN and the uncoupled ones since it is only affecting the required time for the training
process. It is known that the training process in case of one independent NN for one joint
takes less time than the case of one NN coupling the three joints together. The main point
here is not the required time of the training process, but the best performance of the trained
NN and its effectiveness.











29
The three cases of the collisions (collision on the end-effector, collision on link 2, and
collision on link 1), as discussed before, are shown from Fig. 17 to Fig. 19.
(a) Joint 1 (b) Joint 2 (c) Joint 3
Fig. 17. The three external collision joint torques resulted from KRC, one independent trained NN for
each joint and using the force sensor with collision on the end-effector only.
(a) Joint 1 (b) Joint 2 (c) Joint 3
Fig. 18. The three external collision joint torques resulted from KRC, one independent trained NN for
each joint and using the force sensor with collision on link 2 only.

30
(a) Joint 1 (b) Joint 2 (c) Joint 3
Fig. 19. The three external collision joint torques resulted from KRC, one independent trained NN for
each joint and using the force sensor with collision on link 1 only.
From Fig. 17 to Fig. 19, it is noted that the independent NN for each joint has inferior
performance than the NN coupling the three joints, since the manipulator is a coupled system.
The approximation errors between the external joint torques resulted from the independent
trained NN and KRC are higher as shown in Fig. 17(a), Fig. 17(b) and Fig. 18(b) where very
high error is resulted during the collision. As shown from Fig. 18(a), a false negative collision
occurs where the real collision is detected (above threshold), but the estimated one by the
independent trained NN is not detected (under threshold). In this case, the performance
measure of the trained NN is affected. The collision on link 2 should have small effect on the
joint 3 external torque as illustrated by KRC and force sensor in Fig. 18(c), but as presented
by the independent trained NN this effect is high where the external collision torque is close
to threshold. In Fig. 19(b), a false positive collision occurs since the collision on link 1 should
have small effect on joint 2 external torque as illustrated by KRC and force sensor (under
threshold), but the trained NN does not prove that, since its estimated external torque exceeds
the collision threshold of joint 2.
The actual collision detection time, by the uncoupled NNs and the model-based approach,
for joints 1, 2 and 3 is presented in Table 5. Although the proposed collision thresholds by the
uncoupled NNs are higher than the ones proposed by model-based approach, the collision
detection time by the independent trained NN is lower when the occurring collision on the
end-effector is detected by joint 1 and joint 2 and also in the case, where the collision on link
2 is detected by joint 2. In the other cases, the collision detection time by the model based
approach is the lower.

31
Table 5 The collision detection time obtained from the independent trained NN and model-
based approach for each joint using the constant velocity of 0.5 rad/sec and using the force
sensor as the reference signal for the correct detection.
The performance measure of the uncoupled trained NNs, in the NNs training range (-90
deg to 90 deg) and under a wide range of operating conditions, is presented also using 75
trials of collisions (TN), (experiments 6 from Table B.1 in Appendix B). The collisions are
evaluated with various magnitudes, points of collision, directions and different velocities of
the motion. Table 6 provides the performance of the uncoupled NNs.
Table 6 Summary of the performance of the uncoupled trained NNs obtained for the
different collision scenarios and angular speeds of the joints.
Collision
Scenario
Joint
Uncoupled Trained NNs
Performance
Measure (%)
TN
TP
FN
FP


*100%
Average
for the
three
joints
Collisions
on the end-
effector
Joint 1
25
17
8
3
56 %
72 %
Joint 2
25
22
3
4
72 %
Joint 3
25
24
1
2
88 %
Method
Joint
Threshold
(Nm)
Collision Detection Time (msec)
Collision on
end-effector
Collision on link
2
Collision on link
1
The
Uncoupled
Trained
NNs
Joint 1
2.0075
26.0
Collision is not
detected by fault
(FN)
30.0
Joint 2
1.8551
15.6
23.0
Collision is
detected by fault
Joint 3
1.0636
11.9
----------
----------
Model-
based
approach
Joint 1
1.5237
32.0
33.0
27.4
Joint 2
1.1460
17.0
15.6
----------
Joint 3
0.5452
20.2
----------
----------
External
Force
Sensor
Joint 1
The signal from the external force sensor is taken as the reference
one.
Joint 2
Joint 3

32
Collision
Scenario
Joint
Uncoupled Trained NNs
Performance
Measure (%)
TN
TP
FN
FP


*100%
Average
for the
three
joints
Collision on
link 2
Joint 1
25
18
7
2
64 %
64 %
Joint 2
25
24
1
3
84 %
Joint 3
------
------
------
5
-20 %
Collision on
link 1
Joint 1
25
20
5
1
76 %
48 %
Joint 2
------
------
------
4
-16 %
Joint 3
------
------
------
3
-12 %
Average Percentage
Case 1
83.3 %
16.7 %
18 %
 65.3 %
Case 2‡‡‡
92.7 %
7.3 %
18 %
74.7 %
As presented in Table 6, the overall performance of the uncoupled trained NNs is lower
than the coupled trained one. Therefore, this proves that the dynamic coupling between the
joints should be taken into consideration during the design of the NN for collision detection.
Quantitative comparison between the coupled NN and the uncoupled NNs is presented in the
next subsection.
7.3. Comparison between the Coupled NN and Uncoupled NNs
As shown from Fig. 11 to Fig. 13 and Tables 2, and 3 for the coupled trained NN and from
Fig. 17 to Fig. 19 and Tables 5, and 6 for independent trained NNs, the performance of the
coupled trained NN is better than the uncoupled ones. The minimum MSE after training the
coupled NN (0.034677 Nm, Fig. C.1 in Appendix C) is lower than the ones after training the
uncoupled NNs (0.10501 Nm, 0.23652 Nm, and 0.015453 Nm for joints 1, 2, and 3
respectively, Fig. C.2 in Appendix C). The average of the absolute approximation error values
of contact-free motion in case of the coupled NN (0.1060 Nm, 0.1333 Nm, 0.0864 Nm for
joints 1, 2, and 3 respectively) is also lower than the one in the case of the uncoupled ones
(0.1965 Nm, 0.2416 Nm, 0.0762 Nm). The defined collision thresholds for the coupled NN
(1.6531 Nm, 1.5521 Nm, and 0.8109 Nm for the three joints 1, 2, and 3 respectively) are
lower than the ones defined for the uncoupled NNs (2.0075 Nm, 1.8551 Nm, and 1.0636
Nm). The overall performance measure (86.6 %) of the coupled NN is higher and its false
positive (8.7 %) and negative (4.7 %) collisions are lower than the case of the uncoupled NNs
‡‡‡
This case, where both the estimated and real contacts are under threshold, is ignored and is not
counted/considered in the performance measure of the proposed method.

33
(performance measure is 74.7 %, false positive collisions are 18 %, and false negative
collisions are 7.3 %).
From the previous comparison we can summarize that, the dynamic coupling between the
joints should be taken into considerations during the design of the NN so this is why the
coupled NN which combines the three joints together has better performance and
effectiveness than the uncoupled/independent NNs.
8. Conclusion and Future Work
In this paper, a method is proposed for human-robot collision detection by designing and
training a multilayer neural network coupling three joints of a serial manipulator. The
introduction of the coupled multilayer feedforward NN was taken after thorough investigation
of the joints dynamic coupling and the experimental evaluation of the collision effect to the
joint motion and torques of the manipulator. The inputs to the NN are derived from the joint
position and joint torque sensors and the method is able to detect the collision of the robot
with the human body very quickly (very small time approximately is between 5 to 30
milliseconds) and in the safe region as well as to identify the collided link. The architecture of
the NN is designed experimentally by trial and error and is trained using the collected data
with and without collisions after carrying out experiments with simultaneous motion of the
three joints (A1, A3 and A5) of the 7-DOF KUKA LWR robot which is an example of
collaborative robots. Our method can be used in any robot having joint torque sensors or
where the estimation of the joint torques are available or can be determined without
knowledge of its model and without any external sensors. The estimation of external torque
given by the robot controller is used only for training the NN, but it can be replaced by any
external force sensor, then the trained NN is used to estimate the external torque, detect the
collision, and identify the collided link. The collision thresholds proposed by the NN are
compared with ISO standards and it is found that these thresholds represent very low
percentage (5% to 10.4%) from the maximum external torques that determined by the
maximum permissible force provided by ISO. Therefore, the human-robot collaboration is in
the safe side.
After extensive experimental testing of the proposed method, it can be concluded that it is
efficient in detecting the collisions whether on the end-effector or on the other two links,
since it succeeds with good percentage (86.6%) to detect the collisions in the safe region. The
percentage of the false positive collisions is low, which means that our method has low
sensitivity to the external disturbances and unmodelled parameters and there is no effect on
the continuation of the robot movement to accomplish the cooperative task. The percentage of
the false negatives collisions, where the estimated contacts by the trained NN are under
threshold and the real ones are above threshold, is also low which mean that the cooperation
between the human and the robot is safe. In case of the false negative collisions, where both
the estimated and real contacts are under threshold, the collided link identification becomes
ill-defined problem and needs further investigations. The trained NN is tested using out of
training range motion and presents satisfactory performance and effectiveness, which proves
its generalization ability.
Three uncoupled NNs are designed and trained, one for each joint, and their performance
is compared with the coupled one which presents better performance and higher effectiveness
since the manipulator is a dynamically coupled system.
The promising results require further investigation and experiments to implement a
collision detection system considering motion of all joints of a manipulator. Moreover, the

34
accurate estimation of the collision magnitude can be used to develop an appropriate collision
reaction strategy that mitigates the reaction forces, which can also benefit a lot by having
identified the link of the robot that collided. Implementation of different types of neural
networks (e.g. Radial Basis Function), or deep learning-based approach and comparing their
performance with the proposed one will be considered. A collision detection method based on
only joints positions sensor signals will be investigated.
Acknowledgment The authors would like to thank Mr. Nikitas Xanthopoulos, External
Expert, Department of Mechanical Engineering and Aeronautics, University of Patras, for his
help in programming with C++. Abdel-Nasser Sharkawy is funded by the “Egyptian Cultural
Affairs & Missions Sector” and “Hellenic Ministry of Foreign Affairs Scholarship” for Ph.D.
study in Greece.
Appendix A: Discussion of the Dynamic Coupling
First Part: Dynamic Parameters of the Manipulator
The inertia matrix, Coriolis and centrifugal matrix and gravity vector of the three-links
manipulator from (1) are calculated in [27] as


 




 (A.1)
where  







 
 








  (A.2)
where



 







35
 

 (A.3)
Equations (1) and from (A.1) to (A.3) show the coupling that occurs between the joints.
From (A.1), it is noted that the inertial force of link 3 affects the measured joint torques and
the external torques of the joints 2, and 3. The measured joint torque and the external torque
of joint 1 are dependent on the masses of the link 2 and the link 3 “end-effector” (),
but don not depend on their accelerations. It should be noted that this case is a special
selection of the joints, but in other cases, there might be effect on the measured/external
torques of joint 1 due to the inertial forces of the links 2 and 3. The inertial force of the link 2
affects the measured joint torques and the external torques of joint 2 and not joint 3, whereas
the inertial force of the link 1 affects only the measured joint torque and the external torque of
joint 1 and not joints 2 and 3.
The Coriolis and centrifugal terms presented in (A.2) show the coupling occurs between
the three joints. The Coriolis forces of link 1 or 2 affects the measured joint torques and
external torques of the three joints whereas the Coriolis forces of link 3 affects only the
measured joint torques and external torques of the joints 1 and 2 and not joint 3. From (A.3),
it is clear that the gravitational force of link 3 affects the measured joint torques and external
torques of both joints 2 and 3 which prove the coupling between these two joints. The
gravitational force of link 2 affects only the measured joint torque and external torque of joint
2 whereas there is no gravitational force of link 1.
Second Part: Results of the Experiments that investigate the Coupling
The results of the 18 experiments that done to prove the dynamic coupling between the
joints, as shown in Fig. 2, are presented in Table A.1 where the maximum amplitudes of the
measured joint torques  and the external torques at
the time when the obstacle collides with the manipulator are included to summarize the effect
on the torques of the joints 1, 2, and 3.
Table A.1 The results of the 18 experiments to investigate the joint coupling.
Weight
Speed
Config
uration
Link 2 is collided by the obstacle
Joint 1
Joint 2
Joint 3
§§§
(Nm)

(Nm)
****E
ffect

(Nm)

(Nm)
Effe
ct

(Nm)

(Nm)
Effec
t
0.314
rad/s
(low)
First
-5
-3.6
Yes
1
0.64
Yes
0.23
0.034
Yes
Second
-4.62
-3.7
Yes
1.41
1.1
Yes
0.10
0.0075
No
Third
-6.7
-5.3
Yes
2.25
1.65
Yes
0.11
0.005
No
§§§
The maximum amplitude may be in the positive or the negative direction (+ or – refers to the
direction only), and this is applied to the three measured/externals torques

****
Effect: means if there is high/clear effect (Yes) on the torques of the joint i or (No), when the
collision between the obstacle and the manipulator occurs.

36
Weight
Speed
Config
uration
Link 2 is collided by the obstacle
Joint 1
Joint 2
Joint 3
§§§
(Nm)

(Nm)
****E
ffect

(Nm)

(Nm)
Effe
ct

(Nm)

(Nm)
Effec
t
(10 N)
1 kg
1.0
rad/s
(mediu
m)
First
-11.8
-4.8
Yes
3.4
1.15
Yes
-0.74
0.135
Yes
Second
-9
-5.4
Yes
2.8
1.6
Yes
-0.25
-0.035
No
Third
-8
-4.7
Yes
2.5
1.3
Yes
-0.25
0.033
No
1.885
rad/s
(high)
First
-16
-7
Yes
5.05
1.75
Yes
-1.05
0.15
Yes
Second
-14.5
-7.05
Yes
5.6
2.25
Yes
-0.5
0.07
No
Third
-13
-6.6
Yes
4.8
2.2
Yes
-0.48
0.08
No
(20 N)
2 kg
0.314
rad/s
(low)
First
-7.8
-6.2
Yes
2.15
1.75
Yes
0.335
0.051
Yes
Second
-10.8
-8.5
Yes
3.75
3.1
Yes
0.125
0.009
No
Third
-8.3
-7
Yes
3.7
2.75
Yes
-0.24
0.021
No
1.0
rad/s
(mediu
m)
First
-17
-9
Yes
5.3
2.65
Yes
0.95
0.160
Yes
Second
-18.8
-9.9
Yes
6.35
3.5
Yes
-
0.275
-0.052
No
Third
-17.3
-8.4
Yes
6
3.1
Yes
-0.62
0.08
No
1.885
rad/s
(high)
First
-39
-16.25
Yes
12.5
5.9
Yes
-3.1
0.53
Yes
Second
-45
-14.5
Yes
16.5
5.15
Yes
-1.35
-0.22
No
Third
-31
-12.5
Yes
11.5
4.4
Yes
-0.81
0.135
No
In Table A.1, “Yes” means that the effect on the measured torque and the external torque
of the joint  () is high or clear, when the collision between the obstacle and the
manipulator occurs whereas “No” means that it is very small and negligible. For the effect of
the collision on the measured/external torques of the joints 1 and 2, “Yes” is chosen for the all
cases/configurations, since it is obvious that this effect is high. For the effect of the collision
on the measured/external torques of the joint 3, “Yes” is chosen only for the first
configuration where the effect of the inertial forces is high. This configuration is taken as a
reference configuration to choose “Yes” or “No” for the other, the second and the third,
configurations where there is very small effect from the inertial forces. In the second and the
third configurations, the percentage of the external torque of this joint (joint 3) to the external
torque of joint 1,
 or the percentage of the measured torque of joint 3 to the measured
torque of joint 1, 
 is compared with a threshold and if this percentage is higher than or
equal to this threshold then “Yes” is chosen otherwise “No”. This threshold is defined as 80
% from
, or 80 % from 
 which is calculated from the first/reference configuration.
This threshold is calculated for both weights (1 kg and 2 kg) and the lowest value is chosen.
The threshold also is changing with the increase/change of the speed. It should be noted that
the measured or the external torque of the joint 1 is taken as a reference torque because it has
the highest value or in another meaning it is the most affected one by the collision. The
selected threshold is presented in Table A.2.

37
Table A.2 The threshold value (%) to choose “Yes” or “No” for the effect of the collision,
between the obstacle and the manipulator, on the torques of joint 3 for the second and the
third configurations.
Parameter
Speed
0.314 rad/s
(low)
1.0 rad/s
(medium)
1.885 rad/s
(high)
Threshold
(%)
based on the
ratio, 

0.64
1.36
1.68
based on the
ratio, 

3.432
4.464
5.25
Appendix B: Protocol of the Experiments in the Paper
The protocol for the entire experiments executed in this paper is presented and discussed
in Table B.1. The all experiments, except experiment 6, are used with the proposed coupled
NN. Experiments 1, 2, 3, are used also for training and evaluation the uncoupled NNs and
Experiments 4, and 6 are used for their verification as discussed in section 7. The desired
sinusoidal motion for the three joints simultaneously is chosen as


 (B.1)
where  is the frequency of the sinusoidal motion.
Table B.1 The steps and protocol for all the discussed experimental work in the paper.
Protocol of the Experimental Work
No
Collision
freq1
freq2
freq3

range

range

range
Benefit
Comments
1
No collision
Variable
(0.05 Hz
to
0.0934
Hz)
Variable
(0.05 Hz
to 0.1225
Hz)
Variable
(0.05 Hz
to 0.0789
Hz)
-90 deg to 90 deg
The three
categories
together
are used to
train the
NN,
then every
category is
used to
evaluate
the trained
NN and t he
first one is
used to
determine
the
collision
thresholds
1 – During the
mot ion of the
three joints
simultaneously,
the maximum
frequencies for
these joint s
until the motion
of the robot
stops are as the
following;
max freq1 =
0.12964 Hz,
max freq2 =
0.18302 Hz,
max freq3 =
0.103057 Hz.
2 – By using
the range of the
joint s motion
from “-90 deg
to 90 deg” for
training the NN
and then testing
/checking the
trained NN in
2
Random
collisions on
the three links
Variable
(0.05 Hz
to 0.109
Hz)
Variable
(0.05 Hz
to 0.1487
Hz)
Variable
(0.05 Hz
to 0.089
Hz)
3
Collisions at
known
locations on
the three links
with different
configurations
each time
0.05 Hz (low)
0.09 Hz (medium)
0.1487 Hz (high)
4
Collision only
on one link
from the three
links each
time
0.08 Hz (0.5 rad/s)
-90 deg to 90 deg
Check the
trained NN
wit h
constant
velocity
Variable
(0.05 Hz
to
0.1117
Hz)
Variable
(0.05 Hz
to 0.153
Hz)
Variable
(0.05 Hz
to 0.0911
Hz)

38
Protocol of the Experimental Work
No
Collision
freq1
freq2
freq3

range

range

range
Benefit
Comments
5
25 trials of
collisions with
various
magnitudes,
points of
collision,
directions and
different
velocities of
the motion on
one link from
the three links
(end-effector,
link 2 and link
1) each time.
Variable
(0.05 Hz
to
0.1063
Hz)
Variable
(0.05 Hz
to 0.1441
Hz)
Variable
(0.05 Hz
to 0.088
Hz)
-90 deg to 90 deg
Evaluate
the
efficiency
of the
coupled
trained NN
this range and
out of this
range; in the
negative
direction (from
-115 deg to -90
deg for joints 2
and 3 and -165
deg to -90 deg
for joint 1), and
in the positive
direction (from
90 deg to 115
deg for joints 2
and 3 and from
90 to 165 for
joint 3), this
means that our
trained NN
approximately
work in the
entire
workspace of
the three joints
space motion.
This also
proves the
effect iveness
and the
generalization
ability of the
proposed
method.
Variable
(0.05 Hz
to
0.1129
Hz)
Variable
(0.05 Hz
to 0.1551
Hz)
Variable
(0.05 Hz
to 0.092
Hz)
6
Variable
(0.05 Hz
to 0.115
Hz)
Variable
(0.05 Hz
to 0.160
Hz)
Variable
(0.05 Hz
to 0.0940
Hz)
-90 deg to 90 deg
Evaluate
the
efficiency
of the
uncoupled
trained
NNs
Variable
(0.05 Hz
to
0.1040
Hz)
Variable
(0.05 Hz
to 0.1420
Hz)
Variable
(0.05 Hz
to 0.0859
Hz)
Variable
(0.05 Hz
to
0.1120
Hz)
Variable
(0.05 Hz
to 0.1543
Hz)
Variable
(0.05 Hz
to 0.090
Hz)
7
Random
collisions on
the three links
0.08 Hz (0.5 rad/s)
90 deg to 115 deg
Check the
coupled
trained NN
by out of
range
joint s
mot ion
-115 deg to -90 deg
8
25 trials of
collisions with
various
magnitudes,
points of
collision,
directions and
different
velocities of
the motion on
one link from
the three links
(end-effector,
link 2 and link
1) each time
Variable
(0.05 Hz
to
0.1219
Hz)
Variable
(0.05 Hz
to 0.1701
Hz)
Variable
(0.05 Hz
to
0.09792
Hz)
90
deg to
165
deg
90 deg to 115
deg
Evaluate
the
efficiency
of the
coupled
trained NN
out of
training
range
joint s
mot ion
Variable
(0.05 Hz
to
0.1245
Hz)
Variable
(0.05 Hz
to 0.1810
Hz)
Variable
(0.05 Hz
to 0.100
Hz)
Variable
(0.05 Hz
to
0.12362
Hz)
Variable
(0.05 Hz
to 0.1730
Hz)
Variable
(0.05 Hz
to 0.099
Hz)
Variable
(0.05 Hz
to
0.12013
Hz)
Variable
(0.05 Hz
to 0.1672
Hz)
Variable
(0.05 Hz
to 0.0967
Hz)
-165
deg to
-90
deg
-115 deg to -90
deg
Variable
(0.05 Hz
to
0.1264
Hz)
Variable
(0.05 Hz
to 0.1777
Hz)
Variable
(0.05 Hz
to 0.101
Hz)

39
Protocol of the Experimental Work
No
Collision
freq1
freq2
freq3

range

range

range
Benefit
Comments
Variable
(0.05 Hz
to
0.1150
Hz)
Variable
(0.05 Hz
to 0.1586
Hz)
Variable
(0.05 Hz
to 0.0933
Hz)
Appendix C: The Training Process of the NN
After trying different weights’ initializations with different hidden neurons (40, 50, 70,
90, 120, 150, and 170), the best performance for the coupled NN and uncoupled NNs that
give us the minimum squared error (MSE) and the proper collision thresholds are shown in
Fig. C.1 and Fig. C.2 respectively. For training whether the coupled NN or uncoupled ones,
the best performance occurs after using 150 hidden neurons and 1000 iterations.
Fig. C.1. The lowest mean squared error value to train the NN coupling the three joints together.
(a)

40
(b)
(c)
Fig. C.2. The lowest mean squared error value to train one independent NN for (a) joint 1, (b) joint 2,
and (c) joint 3.
Appendix D: Latencies from the Sensors
As discussed in the paper, Matlab Toolbox is used for the training process of the NN. The
training process of the coupled NN is implemented offline on AMD Ryzen 5 1600 Six-Core
processor since too many data require good hardware with high RAM to make the training
fast. After the training process is finished, the coupled trained NN is implemented on KUKA
LWR robot to check online its efficiency and generalization. The collision detection time,
presented in Table 2, comes/arise from the following time;
 800 µs is required to read from the internal joints torque, position, and velocity
sensors as well as to send motion commands to the robot,
 145 µs is required to read from the external force sensor (its latency 144 µs),
 6 µs is required for the NN calculations.
 1ms (sampling rate) – (800+145+6) µs = 49 µs is required to other calculations and
algorithmic commands.

41
This process is illustrated in Fig. D.1. The required time for reading from the sensors of
KUKA, as presented, is less than 1 msec (the sampling rate), so the collision detection time,
which is presented in Table 2, is enough for reaction. Therefore safe human-robot interaction
is achieved.
Fig. D.1. The time of the occurring loop in KRC.
Compliance with ethical standards
Conflict of interest: The authors declare that they have no conflict of interest.
Ethical approval: All procedures performed in studies involving human participants were in
accordance with the ethical standards of the institutional and/or national research committee
and with the 1964 Helsinki declaration and its later amendments or comparable ethical
standards.
Ethical approval: This article does not contain any studies with animals performed by any of
the authors.
Informed consent: Informed consent was obtained from all individual participants included
in the study.
References
[1] ISO 10218-1, Robots and robotic devices — Safety requirements for industrial robots
— Part 1: Robots, (2011).
[2] ISO 10218-2, Robots and robotic devices — Safety requirements for industrial robots
— Part 2: Robot systems and integration, (2011).
[3] ISO/TS 15066, Robots and robotic devices — Collaborative robots, (2016).
[4] Y. Yamada, Y. Hirasawa, S. Huang, Y. Umetani, K. Suita, Human – Robot Contact in
the Safeguarding Space, IEEE/ASME Trans. MECHATRONICS. 2 (1997) 230–236.
800μs
Send motion
commands to
the robot
Read internal torque
joint sensors
Read the force
sensor
Robot
Neural Network
1 ms
1 ΚHz
Read joint position,
velocity, etc.
Other calculations and
algorithmic commands
6μs
49μs
145µs
144 μs
Latency

42
[5] F. Flacco, T. Kroger, A. De Luca, O. Khatib, A Depth Space Approach to Human-
Robot Collision Avoidance, in: 2012 IEEE Int. Conf. Robot. Autom., RiverCentre,
Saint Paul, Minnesota, USA, 2012: pp. 338–345.
[6] B. Schmidt, L. Wang, Contact-less and Programming-less Human-Robot
Collaboration, in: Forty Sixth CIRP Conf. Manuf. Syst. 2013, Elsevier B.V., 2013: pp.
545–550. doi:10.1016/j.procir.2013.06.030.
[7] F.D. Anton, S. Anton, T. Borangiu, Human-Robot Natural Interaction with Collision
Avoidance in Manufacturing Operations, in: Serv. Orientat. Holonic Multi Agent
Manuf. Robot., © Springer-Verlag Berlin Heidelberg 2013, 2013: pp. 375–388.
doi:10.1007/978-3-642-35852-4.
[8] M. Kitaoka, A. Yamashita, T. Kaneko, Obstacle Avoidance and Path Planning Using
Color Information for a Biped Robot Equipped with a Stereo Camera System, in: Proc.
4th Asia Int. Symp. Mechatronics, 2010: pp. 38–43. doi:10.3850/978-981-08-7723-
1_P134.
[9] S. Lenser, M. Veloso, Visual Sonar : Fast Obstacle Avoidance Using Monocular
Vision, in: Proc. 2003 IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IROS 2003), 2003.
doi:10.1109/IROS.2003.1250741.
[10] B. Peasley, S. Birchfield, Real-Time Obstacle Detection and Avoidance in the
Presence of Specular Surfaces Using an Active 3D Sensor, in: 2013 IEEE Work.
Robot Vis., Clearwater Beach, FL, USA, 2013: pp. 197–202.
doi:10.1109/WORV.2013.6521938.
[11] F. Flacco, T. Kroeger, A. De Luca, O. Khatib, A Depth Space Approach for
Evaluating Distance to Objects, J. Intell. Robot. Syst. 80 (2014) 7–22.
doi:10.1007/s10846-014-0146-2.
[12] T.L. Lam, H.W. Yip, H. Qian, Y. Xu, Collision Avoidance of Industrial Robot Arms
using an Invisible Sensitive Skin, in: 2012 IEEE/RSJ Int. Conf. Intell. Robot. Syst.,
Algarve, Portugal, 2012: pp. 4542–4543.
[13] D. Gandhi, E. Cervera, Sensor Covering of a Robot Arm for Collision Avoidance, in:
SMC’03 Conf. Proceedings. 2003 IEEE Int. Conf. Syst. Man Cybern. Conf. Theme -
Syst. Secur. Assur. (Cat. No.03CH37483), IEEE, Washington, DC, USA, 2003: pp.
4951–4955. doi:10.1109/ICSMC.2003.1245767.
[14] S. Haddadin, A. Albu-sch, A. De Luca, G. Hirzinger, Collision Detection and
Reaction : A Contribution to Safe Physical Human-Robot Interaction, in: 2008
IEEE/RSJ Int. Conf. Intell. Robot. Syst., IEEE, Nice, France, 2008: pp. 3356–3363.
[15] C. Cho, J. Kim, S. Lee, J. Song, Collision detection and reaction on 7 DOF service
robot arm using residual observer, J. Mech. Sci. Technol. 26 (2012) 1197–1203.
doi:10.1007/s12206-012-0230-0.
[16] B. Jung, H.R. Choi, J.C. Koo, H. Moon, Collision Detection Using Band Designed
Disturbance Observer, in: 8th IEEE Int. Conf. Autom. Sci. Eng., Korea, 2012: pp.
1080–1085.
[17] S. Morinaga, K. Kosuge, Collision Detection System for Manipulator Based on
Adaptive Impedance Control Law, in: Proc. 2003 IEEE Int. Conf. Robot.
&Automation, Tsirno, 2003: pp. 1080–1085.
[18] S. Lu, J.H. Chung, S.A. Velinsky, Human-Robot Collision Detection and
Identification Based on Wrist and Base Force / Torque Sensors, in: Proc. 2005 IEEE
Int. Conf. Robot. Autom., Spain, 2005: pp. 796–801.
[19] F. Dimeas, L.D. Avenda, E. Nasiopoulou, N. Aspragathos, Robot Collision Detection
based on Fuzzy Identification and Time Series Modelling, in: Proc. RAAD 2013 ,
22nd Int. Robot. Alpe-Adria-Danube Reg., Slovenia, 2013.
[20] F. Dimeas, L.D. Avendano-valencia, N. Aspragathos, Human - Robot collision
detection and identification based on fuzzy and time series modelling, Robotica.
(2014) 1–13. doi:10.1017/S0263574714001143.
[21] A.-N. Sharkawy, N. Aspragathos, Human-Robot Collision Detection Based on Neural
Networks, Int. J. Mech. Eng. Robot. Res. 7 (2018) 150–157.
doi:10.18178/ijmerr.7.2.150-157.

43
[22] A.-N. Sharkawy, P.N. Koustoumpardis, N. Aspragathos, Manipulator Collision
Detection and Collided Link Identification based on Neural Networks, in: A. Nikos, K.
Panagiotis, M. Vassilis (Eds.), Adv. Serv. Ind. Robot. RAAD 2018. Mech. Mach. Sci.,
Springer, Cham, 2018: pp. 3–12. doi:https://doi.org/10.1007/978-3-030-00232-9_1.
[23] P.A. Devijver, J. Kittler, Pattern Recognition: A Statistical Approach, Englewood
Cliffs, NJ: Prentice-Hall, 1982.
[24] G.P. Zhang, Neural Networks for Classification : A Survey, IEEE Trans. Syst. MAN,
Cybern. C Appl. Rev. 30 (2000) 451–462. doi:10.1109/5326.897072.
[25] V.L. Berardi, G.P. Zhang, The Effect of Misclassification Costs on Neural Network
Classifiers, Decis. Sci. 30 (1999) 659–682. doi:https://doi.org/10.1111/j.1540-
5915.1999.tb00902.x.
[26] Y. LeCun, Y. Bengio, Pattern Recognition and Neural Networks, in: M.A. Arbib
(Ed.), Handb. Brain Theory Neural Networks, MIT Press, 1995: pp. 1–22.
[27] R.M. Murray, Z. Li, S.S. Sastry, A Mathematical Introduction to Robotic
Manipulation, CRC Press, 1994.
[28] A. De Luca, A. Albu-Schaffer, S. Haddadin, G. Hirzinger, Collision Detection and
Safe Reaction with the DLR-III Lightweight Manipulator Arm, in: 2006 IEEE/RSJ Int.
Conf. Intell. Robot. Syst., IEEE, Beijing, China, 2006: pp. 1623–1630.
doi:10.1109/IROS.2006.282053.
[29] K.-L. Du, M.N.S. Swamy, Neural networks in a softcomputing framework, London:
Springer, 2006.
[30] S. Haykin, Neural Networks and Learning Machines, Third Edit, Pearson, 2009.
[31] T. Most, Approximation of complex nonlinear functions by means of neural networks,
in: 2nd Weimar Optim. Stoch. Days 2005, Grand Hotel Russischer Hof, Weimar,
Germany, 2005.
[32] M.A. Nielsen, Neural Networks and Deep Learning, Determination Press, 2015.
[33] S. Ferrari, R.F. Stengel, Smooth Function Approximation Using Neural Networks,
IEEE Trans. NEURAL NETWORKS. 16 (2005) 24–38.
[34] K. HORNIK, M. STINCHCOMBE, H. WHITE, Universal Approximation of an
Unknown Mapping and Its Derivatives Using Multilayer Feedforward Networks,
Neurul Networks. 3 (1990) 551–560.
[35] A.T. Vemuri, M.M. Polycarpou, Neural-Network-Based Robust Fault Diagnosis in
Robotic Systems, IEEE Trans. NEURAL NETWORKS. 8 (1997) 1410–1420.
[36] C. Cho, J. Kim, Y. Kim, J.-B. Song, J.-H. Kyung, Collision Detection Algorithm to
Distinguish Between Intended Contact and Unexpected Collision, Adv. Robot. 26
(2012) 1825–1840. doi:10.1080/01691864.2012.685259.
[37] B. Jung, J.C. Koo, H.R. Choi, H. Moon, Human-robot collision detection under
modeling uncertainty using frequency boundary of manipulator dynamics, J. Mech.
Sci. Technol. 28 (2014) 4389–4395. doi:10.1007/s12206-014-1006-5.
[38] F. Min, G. Wang, N. Liu, Collision Detection and Identification on Robot
Manipulators Based on Vibration Analysis, Sensors. 19 (2019) 1–17.
doi:10.3390/s19051080.
[39] M. Indri, S. Trapani, I. Lazzero, Development of a Virtual Collision Sensor for
Industrial Robots, Sensors. 17 (2017) 1–23. doi:10.3390/s17051148.
[40] A.C. Smith, K. Hashtrudi-Zaad, Application of neural networks in inverse dynamics
based contact force estimation, in: Proc. 2005 IEEE Conf. Control Appl., IEEE,
Toronto, Canada, 2005: pp. 1021–1026. doi:10.1109/CCA.2005.1507264.
[41] H.D. Patiño, R. Carelli, B.R. Kuchen, Neural Networks for Advanced Control of
Robot Manipulators, IEEE Trans. NEURAL NETWORKS. 13 (2002) 343–354.
doi:10.1109/72.991420.
[42] K.Y. Goldberg, B.A. Pearlmutter, Using a neural network to learn the dynamics of the
CMU Direct-Drive Arm II, Pittsburgh, Pennsylvania, 1988.
[43] M. Fumagalli, A. Gijsberts, S. Ivaldi, L. Jamone, G. Metta, L. Natale, F. Nori, G.
Sandini, Learning to exploit proximal force sensing: A comparison approach, in: S. O.,
P. J. (Eds.), From Mot. Learn. to Interact. Learn. Robot. Stud. Comput. Intell.,

44
Springer, Berlin, Heidelberg, 2010: pp. 149–167. doi:10.1007/978-3-642-05181-4_7.
[44] G. Schreiber, A. Stemmer, R. Bischoff, The Fast Research Interface for the KUKA
Lightweight Robot, in: Work. IEEE ICRA 2010 Work. Innov. Robot Control Archit.
Demanding Appl. − How to Modify Enhanc. Commer. Control., Anchorage, 2010: pp.
15–21.
[45] HC10 Collaborative Robot - Yaskawa Motoman, (n.d.).
https://www.motoman.com/collaborative/products.
[46] FRANKA EMIKA, Infanteriestraße 19 80797 Munich, Germany, n.d.
http://donar.messe.de/exhibitor/cebit/2017/G679739/broschuere-ger-499315.pdf.
[47] M. Mihelj, M. Munih, Open Architecture xPC Target Based Robot Controllers for
Industrial and Research Manipulators, in: Work. IEEE ICRA 2010 Work. Innov.
Robot Control Archit. Demanding Appl. − How to Modify Enhanc. Commer. Control.,
Anchorage, 2010: pp. 54–61.
[48] Z. Li, H. Wu, J. Yang, M. Wang, J. Ye, A Position and Torque Switching Control
Method for Robot Collision Safety, Int. J. Autom. Comput. 15 (2018) 156–168.
doi:10.1007/s11633-017-1104-9.
[49] S. Chen, M. Luo, F. He, A universal algorithm for sensorless collision detection of
robot actuator faults, Adv. Mech. Eng. 10 (2018) 1–10.
doi:10.1177/1687814017740710.
[50] J. Schmidhuber, Deep learning in neural networks : An overview, Neural Networks. 61
(2015) 85–117. doi:10.1016/j.neunet.2014.09.003.
[51] K. Du, M.N.S. Swamy, Neural Networks and Statistical Learning, Springer, 2014.
doi:10.1007/978-1-4471-5571-3.
[52] D.W. Marquardt, An Algorithm for Least-Squares Estimation of Nonlinear
Parameters, J. Soc. Ind. Appl. Math. 11 (1963) 431–441.
[53] M.T. Hagan, M.B. Menhaj, Training Feedforward Networks with the Marquardt
Algorithm, IEEE Trans. NEURAL NETWORKS. 5 (1994) 2–6.
[54] KUKA.FastResearchInterface 1.0, KUKA System Technology (KST), D-86165
Augsburg, Germany, 2011.
[55] KUKA Roboter GmbH, Lightweight Robot 4+, Specification, D-86165 Augsburg,
Germany, 2010.
[56] A. Albu-Schaffer, S. Haddadin, C. Ott, A. Stemmer, T. Wimbock, G. Hirzinger, The
DLR lightweight robot : design and control concepts for robots in human
environments, Ind. Robot An Int. J. 34 (2007) 376–385.
doi:10.1108/01439910710774386.