New cognitive info-communication channels for human-machine interaction

Abstract

The main goal of this paper is to discuss a new paradigm for robot teaching and supervising which is based on cognitive info-communication channels for human-machine interaction. According to the studied concept the robot is considered as an unskilled worker “who” (which) is strong and capable for precise manufacturing. “He” (it) has a special kind of intelligence, but “he” is handicapped in some senses, that is why “he” needs special treatment. We must command “him” clearly in a special way and we must supervise “his” work. If we can elaborate a proper way for communicating with this “new worker”, as an additional dimension of robotization, we can get a capable new “colleague”. The ultimate goal is to help the boss to be able to give the daily task to a robot in a similar way as he/she able to give the jobs to the human workers. For example, by providing the CAD documentation with some additional verbal explanation.

Keywords— cognitive info-communication, human-machine interaction

Full text

Recent Innovations in Mechatronics (RIiM) Vol. 4. (2017). No. 1.
DOI: 10.17667/riim.2017.1/9
New cognitive info-communication channels for
human-machine interaction
Csongor Márk Horváth
PPM AS,
Trondheim, Norway
csongor@ppm.no
Szilveszter Kovács
Department of Information Technology
University of Miskolc
Miskolc, Hungary
szkovacs@iit.uni-miskolc.hu
Abstract— The main goal of this paper is to discuss a new
paradigm for robot teaching and supervising which is based on
cognitive info-communication channels for human-machine
interaction. According to the studied concept the robot is
considered as an unskilled worker “who” (which) is strong and
capable for precise manufacturing. “He” (it) has a special kind of
intelligence, but “he” is handicapped in some senses, that is why
“he” needs special treatment. We must command “him” clearly
in a special way and we must supervise “his” work. If we can
elaborate a proper way for communicating with this “new
worker”, as an additional dimension of robotization, we can get a
capable new “colleague”. The ultimate goal is to help the boss to
be able to give the daily task to a robot in a similar way as he/she
able to give the jobs to the human workers. For example, by
providing the CAD documentation with some additional verbal
explanation.
Keywords— cognitive info-communication, human-machine
interaction
I. INTRODUCTION
A. Motivation
The main problem of robot teaching and supervising that it
takes approximately 400 times longer to program a robot than
to execute the actual task. Online programming is difficult,
moreover the robot cannot produce anything during the online
programming process. Hence there is a continuous demand for
new and effective robot teaching methods.
If we use the robot as a manipulator to extend our working
capability we usually need complex and very expensive
sensors or feedback devices. Especially the force or tactile
sensors and feedback devices are problematic and expensive.
If we can develop a way for cheap and effective
communication between the human operator and the robot,
then we can extend the profitable robotization fields.
The European Union (EU) homes to more than 26 million
companies of which 10% are manufacturing based. Out of
these 2.5 million manufacturing companies, more than 95% of
which are small medium enterprises SMEs, employ over 36
million people and contribute approximately 22% of the EU
gross domestic product (GDP). In addition, it is estimated that
in total around 75% of the EU GDP and 70% of employment
in Europe is related to manufacturing as a direct result of
manufacturing related service companies i.e. for every job in
manufacturing there are two linked jobs in manufacturing
related services. Therefore, any reduction in manufacturing
capacity is likely to have a significantly negative socio-
economic impact. “An economy based on service industries
alone will not survive in the longer term” (quote from the EC
report ‘MANUFUTURE – A Vision for 2020’ [1]).
B. ICT Transformation of Manufacturing
Much work has already been done to identify which areas
of info-communication technologies (ICT) will have the most
impact on the future of manufacturing. The conclusions of the
ManVis study [1] clearly states that ICT is a key cross cutting
technology and that it is a fundamental foundation for
improved competitiveness in the manufacturing industry. The
ManVis final report suggests that competition from low wage
economies will prevail in the foreseeable future. ICT can,
however, minimize the inevitable decline of European
manufacturing by cost reduction technologies such as
automation and enhanced labour productivity. High
technology manufacturing will be based on the efficient use of
sophisticated manufacturing technologies, such as highly
automated operations. This will be dependent on close links
with the science base.
A long-range vision (15 years) has been formulated
towards the development and use of industrial robotics in
manufacturing scenarios of the future: The future family of
industrial robots will support intuitive instruction, worker
cooperation and rapid deployment.
C. Needs of emerging robotics in end-user industries
There are numerous new fields of applications and
industrial branches in which robot technology is not
widespread. The reason is the lack of flexibility, the
challenging technical requirements and the high costs
involved. New robotic applications will increasingly emerge
from new industries and from SMEs, which cannot use
today’s inflexible robot technology or which still depend on
manual operations under strenuous, unhealthy and hazardous
conditions.
Relieving people from poor working conditions (e.g.,
operation of hazardous machines, handling poisonous or
heavy material, working in dangerous or unpleasant
environments) leads to new opportunities for applying robotics
technology. Examples of poor working conditions can be

Recent Innovations in Mechatronics (RIiM) Vol. 4. (2017). No. 1.
DOI: 10.17667/riim.2017.1/9
found in foundries, the metal working industry. Besides the
need of handling objects at very high temperatures, work
under unhealthy conditions takes place in manual fettling
operations, which contribute to about 40% of the total
production cost in a foundry. Manual fettling means heavy
lifts, strong vibrations, metal dust and high noise levels,
resulting in annual hospitalization costs of more than €150m
in Europe. Poor working conditions can also be found in
slaughterhouses, fisheries and cold stores where low
temperatures, hazardous tools and repetitive tasks contribute
to unattractive working conditions. Other examples where
robots can improve the working environment are painter
workshops, glazier workshops and recycling plants.
If sensor information can be reliably used for robot control
and if robot instruction schemes may be intuitive (e.g., by
using more intuitive interaction mechanisms, built-in process
knowledge and automatic motion generation), and the existing
digital information can be easily accessed, then many
applications areas, where the present robot technology has
failed can be envisioned.
D. ROBOTIC market
Till now applications of robot automation technologies
have been developed specifically for capital-intensive large-
volume manufacturing resulting, in relatively costly and
complex systems, which account for some 70% of all
industrial applications; however, it is estimated that only 15%
of possible applications are currently automated, with small
and medium enterprises being particularly underrepresented.
The world market peaked in 2005, reaching about 126,700
newly installed industrial robots, 30% more than in 2004 [2].
This is the highest number ever recorded for one year.
Nevertheless, developments were quite dissimilar in the three
large industrial regions of Europe, America and Asia. While
robotics investment boomed in Asia and America, order
intakes in Europe were far more moderated. The automotive
industry affected the results in all three regions. In Asia, in
addition to the automotive sector, strong demand from the
electronic components industry, the communication
equipment industry and the computer industry reinforced the
gain in market share already seen in 2004.
Europe is one of the world leaders in robotics equipment
production and use; about 33% of worldwide yearly
shipments, with a sale volume of about € 3.1 billion, concerns
European products; when considering industrial automation
robotics’ supply chain, annual sales volume growths up to
about € 13 billion. Robotics has been a major R&D topic that
was funded both at national and EU level. Industrial robotics
research in Europe find a great support from the necessity
expressed by more than 228,000 European small and medium
sized industries of a more flexible and less bound to mass
production automation technology; these necessities require
future industrial robots to combine highest productivity and
flexibility with minimal manufacturing equipment life-cycle-
cost.
II. COGNITIVE INFO-COMMUNICATION CHANNELS FOR HUMAN
MACHINE INTERACTION
The concept discussed in this paper is rather different from
the traditional human-machine interface approach, which try
to provide the illusion of direct contact with the remote
environment or the working pieces. Our approach is based on
the fact that our brain extends and converts the information
received by our sensory organs to a wide range of sensory
representations. This cross-modal plasticity in cortical areas
allows us to send different types of information to the brain
via various different cognitive ways, so as we can interchange
the sensory information between the human sensors. It is well
known that when we recognize an image, only 10% of the
total visual information comes from our eye, the remaining
part is from our brain. Of course, it assumes a previous
learning process. Our main challenge is that after a certain
learning period we can “see” by the fingertips or by the ears
(as the blinds can do it), we can “feel” tactile force or we can
“listen” to a story by the eyes (as deaf people can do).
Actually, this fact can be observed in many, everyday life
situations as well. When someone tune the violin, he tunes,
practically controls his hand, according to the information he
experiences in his ears. A robot would measure the tenseness
of the string and do the tuning based on that information. That
we call as non-conventional communication (see in Fig. 1).
The suggested concept opens new cognitive communication
channels. For instance, force feedback can well be transferred
to human by voices, which can be as helpful as the force
feedback, but it is much cheaper sometimes it can be more
efficient. In some cases, we do not need precise quantitative
information, qualitative feedback is satisfactory.
Fig. 1. Differences between conventional and non-conventional
information channels
III. INTELLIGENT SPACE AS A NEW PARADIGM IN HUMAN-
MACHINE COMMUNICATION
The Intelligent Space (“iSpace”) is an intelligent
environment which provides both information and physical
supports to humans and robots to enhance their performances
[3]. In order to observe the dynamic environment, many
intelligent devices, which are called Distributed Intelligent
Network Devices, are installed into the room as shown
in Fig. 2.

Recent Innovations in Mechatronics (RIiM) Vol. 4. (2017). No. 1.
DOI: 10.17667/riim.2017.1/9
The DIND is a fundamental element of the iSpace. It
consists of three basic components including sensors,
processors and network devices. By communicating with each
DIND [4], iSpace can perceive and understand events in the
whole space. In addition to observation, the iSpace actuates
intelligent agents such as mobile robots [5], computer devices
and digital equipment. In order to enable humans to operate
the agents in the iSpace, a suitable human interface is needed.
For that reason, it has proposed the so-called “spatial
memory” as an interface between human and the iSpace [6].
The spatial memory system enables human users to store
computerized information into the real world by assigning a
three-dimensional position to the information, and to retrieve
the information by directly indicating the point using their
own hands. Therefore, by using the spatial memory, human
users are able to smoothly implement and utilize information
and services in the iSpace by themselves.
Fig. 2. Concept of the Intelligent Space
On the other hand, when we consider a situation in which
the iSpace try to provide suitable services for humans
autonomously, observation of human activities is more
important. Human activities have been observed and analyzed
through usage history of the spatial memory because
information spatially arranged can be regarded as description
of human activities in a certain environment. The observation
approach provides us to distinguish human activities even in a
same area without any knowledge about human activities.
However, the approach cannot take into account interaction
between humans and physical objects except for iSpace agents
in the observation, because the spatial memory hasn’t had a
function to associate the stored information to physical objects
but to three-dimensional positions. Meanwhile, we can assume
that physical objects, which are used for users’ daily life and
used to accomplish their activities, are important to describe or
estimate human activities. Therefore, we focus on physical
interaction between humans and physical objects in order to
more precisely and widely describe human activities.
There are several researches focusing on determination of
physical objects. Nakauchi, et. al [7] estimate sequence of
humans’ cooking activities based on sequence of physical
objects used. Nishida, et. al [8] have developed a language
learning system which provides suitable information for users
according to the position of physical objects. From these
works, we can say that determination of physical objects and
their position are useful to estimate human activities in the real
world. Intelligent Space (iSpace) consists of three functions:
“Observing”, “Recognizing and Searching”, and “Acting”.
Concretely, the iSpace offer appropriate service by
understanding human intention and state in the space based on
observation of space and human. In the Intelligent Space, the
various sensor DINDs will be arranged such as cameras, laser
range finders, microphones as well as these two sensing
systems, and the information provided from them is vast. In
order to utilize their information efficiently and usefully, a
database to store information is needed. The system structure
of the iSpace is shown in Fig. 3. The iSpace can guide the
mobile robot moving in it [9].
Fig. 3. Structure of Intelligent Space
A. Spatial memory
The spatial memory enables humans to store computerized
information such as digital files and commands into the real
world by assigning three-dimensional position as the memory
address. Humans can retrieve and store such information by
directly indicating the point using their own body, e.g. user's
hand and user's head. That's why a point on user's body is
called a human indicator. Fig. 4 shows a schematic concept of
the spatial memory. The spatial memory system has the
following advantages to achieve intuitively and
instantaneously access and store computerized information.
First, the users can arrange computerized information at a
suitable location using their own eyes and own body action.
Fig. 4. Schematic concept of the spatial memory

Recent Innovations in Mechatronics (RIiM) Vol. 4. (2017). No. 1.
DOI: 10.17667/riim.2017.1/9
B. System integration (Middleware technology)
To manage a rapidly growing need for sensor
communication in robotic applications several suitable
architectures, named middleware’s, is being developed for
easy system integration. Unfortunately, most of these
middleware technologies are developed independently of each
other and are often dedicated for specific user applications
[10]. In Table 1. the most important participants of the robot
middleware competition are briefly compared.
TABLE 1. COMPARISON OF MIDDLEWARE TECHNOLOGIES [11]
Name
Middleware
technology
Open source?
Relevant
contributors
ASEBA
CAN
Yes
EPFL
CLARAty
Multi-level
mobility
abstraction
Yes, but
some
restrictions
NASA
Microsoft RS
Web-
Services
No
Microsoft
Miro
CORBA
Yes
University of
California
Orca
ICE
Yes
KTH
Stockholm
OrIN
DCON,
SOAP, XML
Yes, but
some
restrictions
JARA
Open-R
Aperios OS
No
Sony
Orocos
RealTime
Toolkit
Yes
Katholieke
Universiteit
Leuven
Player
Client /
Server
architecture
Yes
Multiple
RT-
Middleware
CORBA
Yes
AIST
UPnP
HTTP,
SOAP, XML
Yes
University of
Coimbra
Urbi
Client /
Server
architecture
Yes
Gostai
Each of the above listed middleware solutions are
composed from modularized components and have a
hierarchical setup. The differences occur in the portability
between different vendor’s robots (how many robots are
supported), the capability of adaptation to system changes
(some are capable of plug-and-play discovery other need
system restart), the way of communication between system
components and the programming environment (contains
integrated development environment or not).
Almost every competitor is capable of hiding low level
robot programming (motor drives, sensor value read out,
camera image access, etc.) and provides standard interfaces
for high level object oriented robot programming.
The Japanese originated RT-Middleware is a middleware
solution that is under standardization and also proved to be
industry ready and adopted by many industrial partners
(Toshiba (different system components), Honda (ASIMO
humanoid robot), AIST (OpenHRP humanoid robot), etc.) and
also many research institutes.
C. Supervisory systems
Manufacturing equipment undergoes continuous
development as newer and better versions/solutions are pushed
out into the market almost at a daily basis. To invest and keep
pace with the newest technologies is a possibility of only a
very few of the manufacturing companies. Small and medium
sized enterprises (SME) cannot keep up with the investments,
compared with their larger counterparts, and are facing special
challenges.
During the last few decades much attention has been made
to offline programming (generation and transfer of the
numerical control code (NC-code)) of CNC machines. There
exist three older basic standards for the generation of the
numerical control (NC) code namely: ISO 6983, DIN66025
and RS274D [12]. The NC data, from these standards, are
often referred to as M and G codes. Reported weaknesses from
the usage of M and G codes are, among others [13]:
1) Low level language merely presenting the cutter
location (motion) without any reference to the work-
piece geometry. Creates very long code sets where
editing is almost impossible.
2) Vendor specific supplements to the M and G codes are
often out of the limited scope of the standard and in
such cases the NC code will not be inter-machine
exchangeable.
3) One way data flow from design to manufacturing.
Experiences from the shop-floor are difficult to push
back to the design stage.
As to overcome these shortcomings the ISO 14649
standard, often referred to as the STEP-NC standard, was
derived from the initial work in the European Project
OPTIMAL (ESPRIT III 8643) by the WZL of Aachen
University in the middle of the 1990s [14]. In general, the
STEP-NC is a new object-oriented model for data transfer
between computer aided design (CAD) and computer aided
manufacturing (CAM) systems. STEP-NC data transfer
specifies the steps of the machining process rather than the
cutter location (as for ISO 6983 systems) [15]. These higher-
level commands, describing the actual machining process, are
by far more comprehendible, informative and interchangeable
than the G and M- codes in use.
In order to transfer and/or distribute the generated machine
code to the NC-machines a software solution named Direct
Numerical Control or Distributed Numerical Control (DNC-
software) has been developed by various manufactures. DNC
programs are typically used to store larger NC program at
separate computer, then the DNC will feed (upon request) the
NC controller (which typically has a more limited memory
capacity). Newer versions of DNC software features, among

Recent Innovations in Mechatronics (RIiM) Vol. 4. (2017). No. 1.
DOI: 10.17667/riim.2017.1/9
others, possibility to edit the NC code and a simplified
visualization of the manufacturing process [16].
Recently modern NC-machines have the possibility for
inter machine (I/O communication), typically targeted for
communication with a shop-floor controller. However, this is
a weaker feature in the history of development of NC-
machines, compared to the development of offline
programming tools and its appurtenant DNC software.
Historically, in older machines there existed no prepared
solution for inter-machine communication, so all the usual I/O
signals (e.g. start spindle, stop spindle, open/close door/chuck)
had to be captured individually from the controller by doing
some-kind-of intervention on the NC controller itself. Also,
some NC codes (M-codes) were dedicated for I/O
communication (e.g. wait until confirmation etc). In general
hardware intervention is a challenging task due to (very often)
poor and limited documentation of the controllers. Re-
engineering of a controller is time consuming and all hardware
intervention tasks carry the possibility for damages. Delivery
times/cost for damaged components leads also to the
minimization of any hardware intervention tasks.
The industrial robot is another key component in a
FMC/FMS setup. Typically, industrial robots are used for
material handling, replacing the human operator from
repetitive/hard work tasks. Normally, these robots have
excellent capability for I/O communication and are especially
adapted for integration with other cell members. Thereby, in
many smaller FMC setups we will find that the industrial
robot acts as a cell controller. However, robots do not share
the same language, even the same vendor often has different
languages between different versions/types in their portfolio.
In larger FMS system with several robots doing different work
tasks the possibility that all robots are similar in kind and
language will be very limited. Anyway, robots and NC-
machines do not share the same language and software
communication between these two groups of machines is often
non-existent. Again, this leads to more specialized solutions to
be derived from case to case and all communication is
normally done on a hardware level with dedicated I/O ports
where the typical NC-machine has limited functionality.
IV. COGNITIVE INFO-COMMUNICATION CHANNELS FOR HUMAN
MACHINE INTERACTION
One of the main challenges in supervisory control is to
transfer the sensory environment of the robot to the
supervisor. Namely, the supervisor would like “to be present”
in the process as much as possible. There are basically two
problems here. Firstly, the process may need the optimization,
or subcontrol of physical signals for which a human operator
does not possess sensors (for instance the measurement of
current during welding). Secondly, the operator may require
other, supplementary signals to understand the process. In
other words, he/she can sense reality much more precisely
using his/her learned 6th or 7th sense, a mechanism that is not
really applicable in automated control (for instance, when the
pilot hears the engine of the racing car and feels the
acceleration in his body and sees the moving environment
around him at the same time, then after some practice he can
easily optimize the engine’s power; while a computer would
probably measure the inertia and rpm of the engine instead to
control the acceleration). We should emphasize here that in
many cases humans can learn and connect their understanding
to the physical measurement system, however, we have to find
the best channel for this communication. Such channels were
already recognized, for instance, when choosing between
digital and analogue representations of measured details. It is
well known that the pilot in an airplane, who is forced to take
into consideration many details at the same time, prefers to use
an analogue representation in most cases even though it is not
as precise as a digital number on the screen. We also should
underline here that each supervisor may have different
professional channels for process control (a professional cellist
can tune his/her cello to perfection very easily, while other
individuals may have more trouble in such cases and may
refer to the support of electronic tuning devices which provide
a visual evaluation output). We may therefore conclude that
humans use cognitive sensing and, therefore, communication
with the process to be controlled can hardly or cannot be
broken down to typical robot sensor systems, not even by
measuring (or simply conjecturing) all the necessary details.
We have to develop a cognitive transfer mechanism to
improve and specialise communication channels.
A. Possible research and development goals
Therefore, various cognitive communication channels are
needed to be developed that can be easily combined under the
middleware platform and fit to the robot sensor system
according to the special capabilities of the operator (as a
matter of fact the operator should need practice to use the
communication channel to perfection). For instance, an
operator who has a good sense of hearing can use his/her ears
rather than eyes to understand distances (note the difference
between tuning a cello using the ears or using an electronic
tuning device).
The cost of establishing the developed communication
channels are also needed to be considered. For instance, force
feedback can classically be implemented mechanically, as
shown in Fig. 5. This system has a limited degree of freedom
in comparison to a human being, and it is very expensive.
From a theoretical point of view, it is also complicated since
we need to compensate the weight of the system in each
position in case of zero force-feedback and it should be
modified according to geometrical specifics dictated by the
user. It is also important to note that this system is not easily
portable and its deployment is also complicated. However, the
force feedback can be easily transformed via cognitive
channels such as sounds and vibrations [17] (or even more
effectively using a combination of these) and then transferred
to humans depending on the task to be accomplished. These
tools are much cheaper and simpler, see Fig. 6, furthermore
would be much more comfortable. In the same way, the
temperature detected on the arms and body of the worker can
easily be transferred via vibration tools put on the arms of the
user.

Recent Innovations in Mechatronics (RIiM) Vol. 4. (2017). No. 1.
DOI: 10.17667/riim.2017.1/9
B. System integration (Middleware technology)
The robot middleware framework can be easily be adapted
and extended to include the typical components of an FMC(S)
system like industrial robots, CNC machines, automated
guided vehicles (AGVs) etc. Preferably these machines should
be “broken down” to the lowest system level (in general to the
motor controller) and middleware components (device drivers)
needed to be developed, followed by a task dependant
assembly and programming. This will create a truly flexible
system where it is possible to control all the actions within
machines and their interactions with the other members. It
would be possible to merge two or more machines and achieve
a full coordinated control. As an example a 6- axis robot could
be merged with a 3- axis NC-machine into a 9 axis machine
with the possibility of a synchronous machining/ handling of
the work piece.
Érzékelő-kar
(1997~) 7DOF
Érzékelő-kar
(1997~) 7DOF
Fig. 5. Classical force feedback system
Fig. 6: “simple” tools for cognitive channels
However, a complete system “break down” will, in many
cases, be very challenging to achieve because of the week
documentation in older machines. In such cases the system
“break down” and the development of the device drivers
should be stopped at a level where the necessary data can be
retrieved. Fig. 7. shows the conceptual layout of a FMC
system were RT-Middleware components exist in different
levels [18]. RTCs can be merged to become a higher level
RTC or it can be used “as is” in the cell controller. As the
work task requirements changes the structure can be
remodelled and if necessary a deeper system intervention and
development of lower level RTCs could be carried out.
By using a CORBA based software controller (also
introduced by J. Shin, S. Park, C. Ju, H. Cho [19]), and device
drivers (also introduced in the comprehensive work of H. Van
Brussel and P. Valckenaers [20]) this architecture will be
highly flexible and suitable for low series production in the
SME sector.
Fig. 7. Component based system setup [18]
C. The supervisory system
An industrial robot usually works in a production line. The
robot is programmed once and the program is repeated over a
thousand times. In case of grinding and deburring
applications, usually the robot has to be reprogrammed based
on individual observation of every workpiece.
Reprogramming the robot with Offline programming each
time is not acceptable, as it is time consuming and needs
highly qualified operators. Usually grinding and deburring is
carried out by not so qualified people, who are capable of
doing tedious and monotonous work in hard and unhealthy
work-environment.
Using an industrial robot for grinding and deburring,
which is at least as punctual as the human, would be a good
alternative in the above mentioned application. A fully
automated grinding or deburring system would be expensive
and the overall result could be worse than in case of the
manual grinding or deburring. Utilizing the experience from
manual grinding, an operator helped with a 90% automated
system will result in the best efficiency, healthier working
environment, higher production number and less cost. The
methodology of Supervised and adaptive programming of
Industrial Robots [21] is shown in Fig. 8.

Recent Innovations in Mechatronics (RIiM) Vol. 4. (2017). No. 1.
DOI: 10.17667/riim.2017.1/9
The image from the camera is passed through many filters
to help the operator in error identification. The operator’s task
is to identify the errors (burrs, irregularities) of workpiece in
the image provided by the camera. Error identification is done
by the operator by drawing lines, curves or regions on the
image. The Operator is involved only in this task; all other
system components are fully automated (Robot program
generation, robot program compile, robot program upload).
Fig. 8. Industrial use of SAPIR [21]
D. The teaching methods
In manufacturing engineering, man-machine interaction
has gone from typical online programming techniques into
virtual reality based offline programming methodologies.
Today, a wide range of offline software tools is used to
imitate, simulate and control real manufacturing systems.
However, also these new methodologies lack capability when
it comes to human-machine communication. Typically, today
communication with the virtual environment is done via a
keyboard/mouse interface while feedback to the operator is
given from the computer screen. These desktop reality systems
uses only a limited spectre of human senses and there is quite
a low sensation of “being inside” the system. Programming is
still done on the premises of the machines.
By introducing modern technologies, like motion capturing
and augmented reality principles, with the goal of adding
human representations into the programming environment it
will be possible to create a new programming concept where
the human operator can interfere with machines in a cognitive
manner. Thus, the knowledge of a skilled operator is needed
to be captured and an automatic knowledge transfer can
program the robot system [22].
This case the human operator can freely move around in an
industrial environment (e.g. shop-floor) and identify the work-
pieces which are necessary to modify via some robot action.
As before mentioned, such a typical situation could be a robot
grinding process following a molding process where the work-
pieces suffer from some irregularities (burrs). By sight, the
human operator can so very easily identify the area of problem
but cannot exactly quantify the error in term of necessary
material removal to reach the ideal final geometry.
After work-piece identification, the next step in interactive
robot programming is to let the operator make a decision on
what kind of machining action is necessary to undertake.
Naturally, larger modifications with several machining passes
requires a more detailed analysis and often leads to selecting
other machinery than a robot for the machining process (e.g.
milling- or turning- machine). On the other side, if the cutting
depth and material removal is moderate the robot can serve as
an excellent alternative.
Assuming the selection of robot machining, the next step is
to capture the machining path. Here the human operator e.g.
can just move his finger, hand, or arm over the region of
interest while a motion capture system will store all its
movements.
The path from the motion tracking system must be
transferred onto to the ideal work piece (e.g. CAD-model) in
order to verify that the path actually is located as expected.
The path is automatically projected (as a line) onto the surface
of the CAD model. By transferring the robot path directly onto
the CAD model actually defines what the final end-geometry
should look like after the manufacturing process. By this the
risk of cutting into the final geometry is reduced. At a later
stage, before starting up the actual machining process the
mean cutting depth can be calculated (indicated) by a few
“Teach” point along the path which again, based on the
material characteristics in the work-piece and cutting tool,
gives an answer if more than one machining pass is necessary.
Further, the line from the CAD model is fed-back to the
operator either through a (handheld) computer screen or as
suggested via an augmented virtual reality system in the sense
of a Head Mounted Display (HMD). This ability of instant
feed- back makes it possible for the operator to relocate his
path according to what he sees at the real work-piece. By
placing the operator “into the loop” of adjusting/relocating the
generated path, the accuracy requirements of the motion
capturing module are minimized.
Before machining can start, the work- piece base
coordinate system must be established with respect to the
robot base coordinate system. The accuracy of the machining
process will greatly depend on the quality of this work. Also,
simulations of the complete robot motions can be undertaken
in order to check for singularities, out of range limitations, and
possible collisions [23].
Finally, the machining process can be executed. As
mentioned above cutting depth calculations may be
undertaken in order to verify that the operation is done within
certain limits of the machinery.
V. CONCLUSION
Typically, industrial robots are used as transporting
devices (material handling of work pieces between machines)
or in some kind of additive- (e.g. welding, painting etc.) or
subtractive- manufacturing process (e.g. grinding, deburring
etc.). Also, the industrial robot controller has good capability
of I/O communication and often acts as cell controller in a
typical set-up of a flexible manufacturing cell or system. Thus,
in advanced manufacturing systems the industrial robot serves
as a key component for coordinated control and effective
utilisation of the complete production unit.
However, in many regions there are a very low number of
industrial robots in use. The reason is the fact that modern

Recent Innovations in Mechatronics (RIiM) Vol. 4. (2017). No. 1.
DOI: 10.17667/riim.2017.1/9
manufacturing equipment are mainly developed for larger
production series than it is typical in SME environment. In
smaller series production standard equipment (robots) are just
too time-consuming when it comes to reprogramming and
path adjustments.
Also from a technical point of view, robot technology
needs to be adapted to new advanced application areas (e.g.
machining) in small series production units. As an example,
the programming time of a robot grinding path is 400 times
the execution time. This clearly indicates the need for a certain
lot size. This leads to the conclusion that methodologies for
initial programming and process variable adjustment of the
robot trajectory must be focused, as to create more rapid,
interactive and human friendly systems.
In recent years, significant amounts in dollars, euros and
yen have been invested, both nationally and internationally, in
the development of increasingly advanced robots that can,
independently or in interaction with humans, perform
increasingly complex tasks and functions.
However, intelligent robots are not just entertainment to
captivate, or perhaps frighten, us. Robot technology has an
extremely wide range of applications, and users are
formulating promising perspectives for innovation. For
instance, intelligent technological toys, advanced welding
robots, robots for tending to animals in the agricultural sector,
robots for complicated surgery at hospitals, robots that can
navigate and collect samples on Mars or robots that can
understand and respond to human speech.
There is probably no doubt that increasingly advanced
robots and intelligent products will be developed in the future.
But technical intelligence still leaves a lot to be desired. To
date, we have only been able to produce robots that are
programmed to perform repetitive tasks with a series of
specific movements and functions. Advanced robots are not
only a question of technology and intelligence. Increased
utilisation of robot technology can also lead to a need for
debate on the ethical and societal aspects resulting from the
way robot technology will affect our daily lives – both at
home and at work.
The manufacturing industry is often faced to the problem
of rapid product renewal, production processes with large
variations and small series, which place demands on flexible
production lines with short change-over times. So there is
need for advanced cognitive robots, that are flexible and that
can quickly be recalibrated for other production series.
An important key concept for robots will be production
cells that consist of machinery, equipment, sensors, robots and
operators. And there will be the need and opportunity to make
integration and communication between the various elements
of a cell more efficient and targeted towards varied one-off
production and a high degree of process variation.
Furthermore, there will be a need to develop intelligent user
interfaces that can promote interaction between robots and
operators and that can help the user to quickly and flexibly
understand a new task.
VI. ACKNOWLEDGEMENT
The project acknowledges the financial support of
Hungarian Research Fund (OTKA K120501).
References
[1] European Commission. Directorate General for Research,
MANUFUTURE High Level Group, “MANUFUTURE: A Vision for
2020 : Assuring the Future of Manufacturing in Europe”, 2004.
[2] International Federation of Robotics (IFR), [Online],“World Robotics
2007”,
http://www.worldrobotics.org/downloads/2007_Executive_Summary.pd
f, 2007
[3] Korondi P, Hashimoto H,”Intelligent space, as an integrated intelligent
system” Proceedings of the 15th International Conference on Electrical
Drives and Power Electronics EDPE-2003. pp. 24-31,2003
[4] Yoshihisa Toshima, Noriaki Ando and Hideki Hashimoto, "Information
Display System using Active Projector in Intelligent Space -Integration
of distributed devices based on RT-Middleware", The 12th International
Sysmposium on Artificial Life and Robotics (AROB'07), pp.183-186,
2007
[5] Baranyi Peter, Nagy Istvan, Korondi Peter, Hashimoto Hideki, “General
guiding model for mobile robots and its complexity reduced neuro-fuzzy
approximation” in Proceedings of 9th IEEE International Conference on
Fuzzy Systems, pp. 1029-1032, 2000
[6] Mihoko Niitsuma, Hiroshi Hashimoto, Hideki Hashimoto, "Spatial
Memory as an Aid System for Human Activity in Intelligent Space",
IEEE Transactions on Industrial Electronics, Vol.54, No.2, pp.1122-
1131, 2007
[7] Y.Nakauchi, T.Fukuda, K.Noguchi, T.Matsubara, “Intelligent Kitchen:
Cooking Support by LCD and Mobile Robot with IC-Labeled Objects”,
Proc. of the 2005 IEEE/RSJ Int. Conf. on Intelligent Robotics and
Systems, pp.2464-2469, 2005
[8] Y. Nishida, M. Hiramoto, F. Kusunoki, H. Mizoguchi, “Learning by
Doing: Space-Associate Language Learning Using a Sensorized
Environment”, in Proc. of IEEE Int. Conf. on Intelligent Robots and
Systems, pp. 1583-1588, 2005
[9] Mizik S, Baranyi P, Korondi P, Sugiyama M, „Virtual Training of
Vector Function based Guiding Styles” Proc. of Scientific Bulletin of
Politehnica University of Timisoara, Transactions on Automatic Control
and Computer Science 46(60):(1), pp. 81-86, 2001
[10] Tetsuo Kotoku, "Robot Middleware and its Standardization in OMG" in
International Conference on Intelligent Robots and Systems (IROS'06)
Workshop on Robotic Standardization, Beijing, China, 11-13 Oct. 2006
[11] B. Solvang, G. Sziebig and P. Korondi, "Multilevel Control of Flexible
Manufacturing Systems", in Proc. of Conference on Human System
Interaction, 2008
[12] Y. Liu, X. Guo, W. Li, K. Yamazaki, K. Kashihara, M.Fujishima, “An
intelligent NC program processor for CNC system of machine tool,” in
Robotics and Computer-Integrated Manufacturing, vol. 23, Elsevier, pp.
160–169, 2007
[13] X. Zhu, Y. Wang, H. Fu, “A 3-D Simulation System for Milling
Machining Based STEP-NC” in Proceedings of the 2006 IEEE
International Conference on Mechatronics and Automation, Dalian,
China, 2006
[14] S.H. Suh, B.E. Lee, D.H. Chung, S.U. Cheon, “Architecture and
implementation of a shop –floor programming system for STEP-
compliant CNC” in Computer–Aided Design, vol. 35, Elsevier, pp.
1069–1083, 2002
[15] Sibul Lazar, Sziebig Gabor, Korondi Peter, „Review of Man-Machine
Interaction in Five-axis Machining with STEP-NC”, IFAC PAPERS
ONLINE 48:(19) pp. 166-171, 2015
[16] Solvang Bjørn, Sziebig Gábor, Korondi Péter, „Vision Based Robot
Programming”, IEEE International Conference on Networking, Sensing
and Control (ICNSC 2008). pp. 949-954, 2008

Recent Innovations in Mechatronics (RIiM) Vol. 4. (2017). No. 1.
DOI: 10.17667/riim.2017.1/9
[17] Sziebig Gábor, Solvang Bjørn, Kiss Csaba, Korondi Péter, “Vibro-tactile
feedback for VR systems” in 2nd International Conference on Human
System Interaction (HSI), pp. 406-410, 2009
[18] Bjorn Solvang, Gábor Sziebig, Péter Korondi, “Shop_Floor Architecture
for Effective Human-Machine and Inter-Machine Interaction”, ACTA
POLYTECHNICA HUNGARICA 9:(1) pp. 183-201, 2012
[19] J.Shin, S.Park, C.Ju, H.Cho, “CORBA-based integration framework for
distributed shop floor control” in Computers & Industrial Engineering,
vol. 45, Elsevier, pp. 457–474, 2003
[20] H. Van Brussel, P. Valckenaers, ” Robotics, FMS and CIM” in
Handbook of Industrial Robotics, edited by S.Y. Nof, 2nd edition, John
Wiley & Sons INC, New York, ISBN-0-471-17783-0, pp. 811-824,
1999
[21] Solvang Bjørn, Korondi Péter, Sziebig Gábor, Ando Noriaki SAPIR:
Supervised and Adaptive Programming of Industrial Robots, 11th IEEE
International Conference on Intelligent Engineering Systems (INES
2007), pp. 281-286, 2007
[22] Husi G; Szász C; Chindris V: Artificial Immune System Implementation
upon Embryonic Machine for Hardware Fault-tolerant Industrial Control
Applications Journal Of Computer Science And Technology (ISSN:
1000-9000) (eISSN: 1860-4749) 10: (4) pp. 60-66. (2010)
[23] Husi G: Position Singularities and Ambiguities of the KUKA KR5
Robot International Journal Of Engineering And Technology Research
1: (1) pp. 44-50. (2015)