ArticlePDF Available

Approach to Assistive Robotics Based on an EEG Sensor and a 6-DoF Robotic Arm

Authors:

Abstract and Figures

This work presents an approach to Assistive Robotics based in a Brain Computer Interface (BCI) on the Steady State Visual Evoked Potentials (SSVEP), using a 6-DOF Robotic Arm, an Emotiv EPOC sensor and a Kinect camera. The proposed architecture has the objective of integrating different systems, with the aim of creating a robotic assistant for the disabled or movement restricted user, improving the life quality of those people. The robot assistance assignment is related with actions like pick and place, or feed and hold food for the user, etc. The proposed methods are based in image and signal processing, BCI decision making, localization of a target point, trajectory planning for the robot arm, and execution of the assistive task. The most remarkable results validate the proposed approach as a good integration method of different technologies for assistive robotics tasks.
Content may be subject to copyright.
First Submitted version without corrections
For the final vesion check
http://www.praiseworthyprize.org/jsm/index.php?journal=ireme&page=article&op=view&path%5B%5D=19047
International Review of Mechanical Engineering (I.RE.M.E.), Vol. xx, n. x
Manuscript received January 2007, revised January 2007 Copyright © 2007 Praise Worthy Prize S.r.l. - All rights reserved
Approach to Assistive Robotics based on an EEG sensor and a 6-DoF
robotic arm
Camilo Andrés Cáceres1, João Maurício Rosário2, Dario Amaya 3
Abstract This work presents an approach to Assistive Robotics based in a Brain Computer
Interface (BCI) on the Steady State Visual Evoked Potentials (SSVEP), using a 6-DOF Robotic Arm,
an Emotiv EPOC sensor and a Kinect camera. The proposed architecture has the objective of
integrating different systems, with the goal of creating a robotic assistant for the disabled or
movement restricted user, improving the life quality of those people. The robot assistance
assignment is related with actions like pick and place, or feed and hold food for the user, etc. The
proposed methods are based in image and signal processing, BCI decision making, localization of
a target point, trajectory planning for the robot arm, and execution of the assistive task. The most
remarkable results validate the proposed approach as a good integration method of different
technologies for assistive robotics tasks. Copyright © 2016 Praise Worthy Prize S.r.l. - All rights
reserved.
Keywords: Assistive Robotics, EEG, Brain Computer Interface (BCI), SSVEP, Mechatronics
I. Introduction
Nowadays, with the medicine and public health
improvements, life expectancy is longer for people [1], [2]
and because of that, the increasing need of the assistive
technologies has grown up through the last decades. By
another hand, the elderly are not the only population that
requires this kind of assistive technology, others include
motor disabled patients, as locked-in or spinal cord injured
patients who requires this assistive technology for
improvement of their life quality.
Solutions that assisted disabled or non-disabled people
with a BCI, the main integration includes the use of a hand
orthosis [3], [4] displacement of a wheelchair [5]–[11],
rehabilitation devices [12]–[14] or the use of a robotic
device [15]–[17]. A common factor in the cited works is
the interest of improving the life quality of impaired
individuals using the different fields of engineering with a
clear objective.
In [18] they presented the results of a work using EEG
signals, and visual stimulus for an user login and
password. Based on the generated stimulus, it was
possible for the user authentication by the wavelet analysis
in sub-bands. Finally, it was possible to obtain the
matchless characteristic of a user through the application
of the discrete wavelet transform (DTW). Similarly, in
[19] a work was developed based on the movement
intention using EEG and superficial electromyography
(sEMG) signals, with the purpose of developing a
rehabilitation method using hybrid robotics. In the
developing of the work, it was possible to restore the
control of some neuromotor functions in patients with
neural injuries and a Smart Walker equipment. This
experiment achieved the detection of some brain signals,
which are related with the human body motion, as a
contribution for future works.
Following the methodology of the assistive robotics
technology commanded by a BCI, the present work
integrates a robotic arm with a BCI system in a prototype
of an assistive robot.
The main objective is to assist a disabled user with
basic daily life tasks. In this case a pick and place task was
chosen, what could be replaced in the future for a task of
feed and hold food for the user or another related assistive
task. The proposed task was exemplified with some color
cubes, located in an objective area. The task is performed
by a six degrees of freedom robotic arm, which follows
the taken decisions by the user using a BCI system, based
on SSVEP. The objective environment is supervised by a
Kinect camera. Each step of the procedure is integrated
with the objective of obtaining an approach to assistive
robotics.
The paper is organized as follows: First, the materials
and methods used for the development of the work, in
order to propose a base for the integration of each
procedure and method. Finally, the results of the work, the
full-integrated system, and the most important results of
each analyzed subsystem and the conclusions of the
proposed method.
II. Materials and Methods
The proposed robotic assistance architecture is based
on different subsystems, which integrated can help the
disabled user, with motion restrictions, with pick and
place tasks, or even grasp and serve, all by using a robot
arm. The system is shown in Fig 1, where it is possible to
appreciate the different subsystems.
Cáceres, C.A., Rosário, J.M., Amaya, D.
Copyright © 2016 Praise Worthy Prize S.r.l. - All rights reserved International Review of Mechanical Engineering, Vol. xx, n.
x
Fig 1. General view of the proposed architecture
The first subsystem is the Kinect Camera, which is used
for detecting the objects in the interest area and localize
them. The next element is the EEG sensor, Emotiv EPOC,
the one thru Steady State Visual Evoked Potentials
(SSVEP) allows to choose the interest object. The last
component is a robotic arm with 6 degrees of freedom
(DOF), and has the task of interacting with the
environment and assisting the user.
The order of the logic algorithms and processes follow
Fig 2, where all the processes start with the detection of
some objects using the Kinect Camera. Afterwards, with
the object selection by the user thru the BCI system, the
object location is calculated based on the Kinect Camera.
Subsequently, with the target points and using a trajectory
generator for a nonspecific and simulated robotic arm of
6-DoF, the robot can complete its task, assisting the
human. Each process has a deeper description on the next
sections, grouped by sensors.
Fig 2. Proposed Architecture Flowchart
II.1. Kinect Camera
The use of the Kinect camera sensor has two objectives.
The first one is based on use of the RGB camera, for
detecting the available objects in the interest area, and the
second is with the RGBD image, for localizing the object
position in the coordinates X, Y and Z, in respect to the
known Kinect position.
The images are taken and processed with Microsoft C#
Express 2015, the library for image processing
AForge.NET for C# and Matlab. The object detection is
based on the general analysis proposed by [20] and
supported by [21], where the image is processed using the
HSL color space and morphological transformations.
The objective detection zone is a black area with color
cubes, which will simulate the interest objects, in reality it
could be food or a bottle of water, a useful object to the
user.
The used method follows the steps and specifications
shown in the Table 1. The values were calibrated manually
for the given conditions of the environment. A sample
image transformation between each step, starting with a
RGB image of the object area and finishing with the
centroid positioning and labeling.
TABLE I
IMAGE PROCESSING METHOD AND SPECIFICATIONS
The distance calculated to the interest object is based
on the [22], [23] with the mathematical equations (1),
where the position (X, Y, Z) measured from the Kinect
sensor of the object is calculated based on the RGB and
RGBD images.
󰇛 󰇜󰇛󰇜

󰇛 󰇜󰇛󰇜
 (1)
󰇛󰇜
Step
Process
Specifications
Software
1
Image
Capture
-
C#-
Kinect
2
HSL
Filtering
Hue: 60 15
Saturation: 0.5 - 1
Luminance: 0-1
C#-
AForge
3
Gray Scale
-
C#-
AForge
4
Threshold
Threshold value 1
C#-
AForge
5
Erosion
x 1 (one time)
C#-
AForge
6
Dilatation
x 5 (five times)
C#-
AForge
7
Centroid
Detection
and labeling
-
Matlab
Cáceres, C.A., Rosário, J.M., Amaya, D.
Copyright © 2016 Praise Worthy Prize S.r.l. - All rights reserved International Review of Mechanical Engineering, Vol. xx, n.
x
II.2. Brain computer Interface System
The Brain Computer Interface (BCI) system allows the
user to interact with the system. The used method requires
an EEG sensor and a visual stimulus source. The used
method is the SSVEP based on the method proposed by
[24] and adjusted to the Emotiv EPOC sensor by [25]. The
method was adapted to the architecture, using the PC LCD
screen (60 Hz), for the stimulus, and the Emotiv EPOC, as
the EEG sensor.
The generated visual stimuli is created by a developed
software, coded in Processing 2.2.1, a based Java language
compiler. The user has on the screen 3 blinking buttons, 3
different colored squares with a number inside, which
blink in different frequencies.
The buttons are distributed in the screen as is shown in
Fig 3; each button represents the object or objects detected
by the Kinect Camera in the interest area.
Fig 3. Stimulus Software Developed
The buttons have different flashing frequencies and
colors, chosen according to [24][28] and the physical
limitations of the available hardware, like the screen
frequency update (60 Hz) and the sample frequency of the
Emotiv EPOC EEG (128 Hz). The chosen colors and
frequencies are shown in Table 2.
TABLE II
SSVEP STIMULUS SOFTWARE SPECIFICATIONS
Button
Flickering
Frequency
Color (HEX)
1
6 Hz
Red (#00BDFF)
2
4.3 Hz
Blue (#00BDFF)
3
5 Hz
Purple (#E500FF)
According to the procedure, the next step is to capture
and analyze the EEG stimulated data. In this case, the data
capture and processing was made using Matlab, the Signal
Processing Toolbox and the Emotiv System libraries for
capturing the EEG data row.
According to [24] the EEG sensor signal of interest is
Oz, but the adaptation made by [25] showed the possibility
of using O1 and O2 in the case of using the Emotiv EPOC
sensor. The location of the sensors O1 and O2 are shown
Fig 4.
The chosen signal sources EEG locations, O1 and O2,
which are associated with the brain visual cortex and
because of that; they are directly associated with the
chosen BCI method, the SSVEP. The signals processing
procedure and specifications are showed in Table 3, after
some pre-processing.
The goal of the preprocessing is to remove the signal
noise and calculate the Fast Fourier Transform (FFT) and
after the Power Spectral Density (PSD), then acquire the
frequency with the higher power for the PSD in each taken
sample, according to the procedure proposed by [24] and
tested and adjusted by [25] to the Emotiv EPOC.
Fig 4. Location of the positions O1 and O2 for the EEG sensor in the
10-20 System (Image Adapted of [25])
TABLE III
SSVEP SIGNAL PROCESSING
Process
Details
Average of the signals O1 and O2
(O1+O2)/2
Signal Autocorrelation
-
Hanning Window
2 seconds
Bandpass Filtering
4.0 - 15 Hz
Fast Fourier Transform (FFT)
-
Power Spectral Density (PSD)
-
By another side, the chosen signal classification system
is based on the highest power of the frequency added with
the power of the first harmonic in the PSD. The
dominating frequency class will be the one with the
highest power in the PSD, the same rules are proposed by
[24] and [25], with processing shown in Table 3 and the
variable definitions according to [24] and [25]:
Variables:
SData: Most recent two seconds of EEG data.
CData: A concatenation of up to the three most recent
sets of SData.
Cáceres, C.A., Rosário, J.M., Amaya, D.
Copyright © 2016 Praise Worthy Prize S.r.l. - All rights reserved International Review of Mechanical Engineering, Vol. xx, n.
x
And classification rules according to [24] and [25],
where the analysis is made after the preprocessing and
detection of the highest PSD frequency power according
to the procedure:
The second greatest value in SData < 0.35 (Empirical
value according to [5]).
The second greatest value in CData < 0.45 (Empirical
value according to [5]).
The same class is domination in four consecutive
iterations.
The pseudocode of the last rules is presented in Fig 5,
where the function PSD_Highf returns the associated
group to the highest frequency power in the PSD.
Fig 5. Pseudocode of the Classifier
After classifying the signal and recognizing the interest
object, it is possible to get the object position measured
from the Kinect camera.
II.3. Robotic Arm of 6-DOF
In the present architecture, the chosen robot is a non-
specific 6-DoF robot with spherical wrist, showed in Fig
6. Modeled using the DenavitHartenberg (DH)
parameters it is possible to construct the robot Direct
Kinematics. The table 4 shows the DH parameters of the
chosen robot, where 
 and . The software used
for the simulation of the robot arm is Matlab and the
Robotics Toolbox [29].
Fig 6. 6-DoF Robotic Arm. Image made using the RoboAnalyzer
Academic Free Software (Available on: http://roboanalyzer.com/ ).
TABLE IV
DH PARAMETERS
Link
1
2
3
4
5
6
For calculating the inverse kinematic, the kinematic
decoupling method was used according to [30], and for the
trajectory planning the polynomial interpolation method
in the joints space proposed in [31] was used. The
corresponding equations with the 5th order polynomial
interpolation for the trajectory planning are presented in
(2) and (3), with zero initial conditions for two steps.
󰇛󰇜
󰇗󰇛󰇜 (2)
󰇘󰇛󰇜
󰇛󰇜
󰇛󰇜
󰇗󰇛󰇜
󰇗󰇛󰇜
󰇘󰇛󰇜
󰇘󰇛󰇜
 

(3)
Cáceres, C.A., Rosário, J.M., Amaya, D.
Copyright © 2016 Praise Worthy Prize S.r.l. - All rights reserved International Review of Mechanical Engineering, Vol. xx, n.
x
II.4. Integration
Retaking the manipulator initial point could be a default
point. By another way, the interest point position was
chosen by the user by using the BCI then measured with
the Kinect. Finally, the final point is chosen depending on
the robot task, which could be place a picked object, hold
a bottle of water for the user, or even feed the patient.
It is important to transform the position of the interest
object measured into coordinates of the robot frame. In
this case, the available information is the Kinect location
and the object position measured from the Kinect frame,
then using homogeneous transformations matrices, it is
possible to find the object position from the robot frame
as shown in (4).

 󰆓
󰆓 
󰆓
󰆒 (4)
The used system is based on Fig 7, where Pi is the
interest point, So the robot frame and Kinect frame will be
named as k, and, with this configuration and frames
position it is possible to modify the equation (4) and using
the principle of homogeneous transformation matrix to
join a rotation and translation and obtain (5).
Fig 7. Used Simulation



󰇛󰇜



󰇛󰇜 󰇛󰇜
󰇛󰇜 󰇛󰇜



(5)
Now with the position of the initial default point, the
interest point found using (5) and the final point, it is
possible to create a trajectory using the inverse kinematics
based on table 4 and the equation (4). The Fig 8 shows the
trajectories between the desired points and the full
environment simulated in Matlab.
Fig 8. Simulated Environment
III. Results
Through the proposed approach to assistive robotics, it
is possible to understand the integration process and the
involved processing of each system. The mathematics and
engineering processes involved in the proposed approach
are presented in blocks in Fig 9.
Fig 10. Blocks diagram of the Proposed Architecture
Aiming to the integration results as a full-integrated
system is important to check the most results of each
subsystem to determine the global result.
The first subsystem to analyze is the image processing
integrated with the BCI software. By one side, the objects
detection through image processing is accurate for the
given laboratory conditions with controlled light and
external noise. The object selection depends entirely on
the brain signal processing and a good screen flickering.
The next system to analyze is the BCI SSVEP system
performance, the time response, classifier accuracy and
the ITR (information transfer rate) [32], the results are
presented in table 5. According to the table, it is possible
Cáceres, C.A., Rosário, J.M., Amaya, D.
Copyright © 2016 Praise Worthy Prize S.r.l. - All rights reserved International Review of Mechanical Engineering, Vol. xx, n.
x
to compare and check the similar results with the tests
made by [24] and [25], and recognize the accuracy and
ITR, and acceptable selection time for each choice.
TABLE V
SSVEP BCI PERFORMANCE RESULTS
Test
Total
Selections
Average
Selection
time (s)
Accuracy
(%)
ITR
(bits/min)
1
57
9.50
83.33
33.14
2
49
8.17
83.37
36.95
3
25
8.33
100
33.44
4
22
11.00
100
24.32
5
30
7.50
100
39.26
6
46
7.66
83.26
38.66
Mean ±
Std
38.17 ±
14.39
8.69 ±
1.33
91.65 ±
9.13
34.30 ±
5.52
It is important to remember, the object selection for the
proposed architecture needs two bits, then according to the
table 5, the needed time for choosing an object is 8,69 ±
1.33 seconds, without counting the relaxing time or any
another time; with an accuracy of 91.65 ± 9.13 %. The Fig
11 shows the selection of an object in each step.
Fig 11. Object Selection
The last system to analyze is the robotic arm and its
trajectories between the initial (0.595, 0, 0.83) m, interest
(0.3, 0.24, 0.35) m and final point (0.4, -0.6, 0.65) m with
the chosen 5th order polynomial, with the Kinect in the
coordinates (0.3, 0.3, 1.35) m. The behavior of the joints
(q) position (Fig 12); speed (Fig 13) and acceleration (Fig
14). The full 3D trajectory is presented in Fig 15, and the
full simulation system was already presented in Fig 8.All
the 3D simulations present an offset in the Z-axis of -0.35.
Fig 12. Joints Position
Fig 13. Joints Speed
Fig 14. Joints Acceleration
Cáceres, C.A., Rosário, J.M., Amaya, D.
Copyright © 2016 Praise Worthy Prize S.r.l. - All rights reserved International Review of Mechanical Engineering, Vol. xx, n.
x
Fig 15. Robot Arm Trajectory
IV. Conclusions
It can be concluded that the proposed Architecture for
Assistive Robotics is a reality with the current technology,
whereas the correct integration and synergy of the
different branches of the engineering and applied
mathematics is very important. The Assistive Robotics is
a need to fulfill in the modern society and provides an
opportunity for improving the quality of life, especially in
assisting elder and disabled people in daily tasks, by tasks
such as objects pick and place, eat or drink assistance,
displacement, etc.
In addition, the proposed architecture is a valid
methodology for a robotics assistance architecture and
could be comparable to the human way of carrying out a
basic daily task, where the human decides and acts. The
architecture starts with sensing the interest area, by what
could be the human vision system. The next step is the
human choice of choosing the desired object using a BCI
system, which could be comparable to a brain decision.
The last task is planning and executing the assistive task
with a robotic system, which could be comparable with the
movement and action of the human hands, arms or legs to
achieve a goal.
Finally, the integration of the different fields of
engineering aim to explore new and interesting areas for
improving human quality of life like the studied area in
this article, the assistive robotics.
Acknowledgements
This work was supported by the National Council for
the Improvement of Higher Education (CAPES) with a
scholarship for Master Studies in Mechanical Engineering
at the State University of Campinas (UNICAMP).
References
[1] Centers for Disease Control and Prevention (CDC), Ten great
public health achievements--worldwide, 2001-2010., vol. 60,
no. 24. American Medical Association, 2011.
[2] H. Lancaster, “Mortality and the Evolution of Society,”
Expect. Life, pp. 111, 1990.
[3] R. Ortner, B. Z. Allison, G. Korisek, H. Gaggl, and G.
Pfurtscheller, “An SSVEP BCI to control a hand orthosis for
persons with tetraplegia,” IEEE Trans. Neural Syst. Rehabil.
Eng., vol. 19, no. 1, pp. 15, Feb. 2011.
[4] G. Pfurtscheller, T. Solis-Escalante, R. Ortner, P. Linortner,
and G. R. Muller-Putz, “Self-paced operation of an SSVEP-
based orthosis with and without an imagery-based ‘brain
switch’: A feasibility study towards a hybrid BCI,” IEEE
Trans. Neural Syst. Rehabil. Eng., vol. 18, no. 4, pp. 409
414, 2010.
[5] L. Montesano, M. Díaz, S. Bhaskar, and J. Minguez,
“Towards an intelligent wheelchair system for users with
cerebral palsy.,” IEEE Trans. Neural Syst. Rehabil. Eng., vol.
18, no. 2, pp. 193202, Apr. 2010.
[6] Z. Bahri, S. Abdulaal, and M. Buallay, “Sub-band-power-
based efficient Brain Computer Interface for wheelchair
control,” in 2014 World Symposium on Computer
Applications and Research, WSCAR 2014, 2014, vol. 4, no. 1,
pp. 3440.
[7] R. Leeb, D. Friedman, G. R. Müller-Putz, R. Scherer, M.
Slater, and G. Pfurtscheller, “Self-paced (asynchronous) BCI
control of a wheelchair in virtual environments: A case study
with a tetraplegic,” Comput. Intell. Neurosci., vol. 2007,
2007.
[8] B. Rebsamen, E. Burdet, C. Guan, H. Zhang, C. L. Teo, Q.
Zeng, C. Laugier, and M. H. Ang Jr., “Controlling a
Wheelchair Indoors Using Thought,” IEEE Intell. Syst., vol.
22, no. 2, pp. 1824, Mar. 2007.
[9] E. J. Rechy-Ramirez, H. Hu, and K. McDonald-Maier, “Head
movements based control of an intelligent wheelchair in an
indoor environment,” 2012 IEEE Int. Conf. Robot.
Biomimetics, pp. 14641469, 2012.
[10] S. M. T. M??ller, T. F. Bastos, and M. S. Filho, “Proposal of
a SSVEP-BCI to command a robotic wheelchair,” J. Control.
Autom. Electr. Syst., vol. 24, no. 12, pp. 97105, Jun. 2013.
[11] M. Palankar, K. J. De Laurentis, R. Alqasemi, E. Veras, R.
Dubey, Y. Arbel, and E. Donchin, “Control of a 9-DoF
wheelchair-mounted robotic arm system using a P300 brain
computer interface: Initial experiments,” in 2008 IEEE
International Conference on Robotics and Biomimetics,
ROBIO 2008, 2008, pp. 348353.
[12] S. Fok, R. Schwartz, M. Wronkiewicz, C. Holmes, J. Zhang,
T. Somers, D. Bundy, and E. Leuthardt, “An EEG-based
brain computer interface for rehabilitation and restoration of
hand control following stroke using ipsilateral cortical
physiology,” Proc. Annu. Int. Conf. IEEE Eng. Med. Biol.
Soc. EMBS, pp. 62776280, 2011.
[13] J. Webb, Z. G. Xiao, K. P. Aschenbrenner, G. Herrnstadt, and
C. Menon, “Towards a portable assistive arm exoskeleton for
stroke patient rehabilitation controlled through a brain
computer interface,” Proc. IEEE RAS EMBS Int. Conf.
Biomed. Robot. Biomechatronics, pp. 12991304, 2012.
[14] T. Lüth, D. Ojdanić, O. Friman, O. Prenzel, and A. Gräser,
“Low level control in a semi-autonomous rehabilitation
robotic system via a brain-computer interface,” in 2007 IEEE
10th International Conference on Rehabilitation Robotics,
ICORR’07, 2007, pp. 721728.
[15] A. Vourvopoulos and F. Liarokapis, “Brain-controlled NXT
Robot: Tele-operating a robot through brain electrical
activity,” Proc. - 2011 3rd Int. Conf. Games Virtual Worlds
Serious Appl. VS-Games 2011, pp. 140143, 2011.
[16] A. Güneysu and H. L. Akin, “An SSVEP based BCI to
control a humanoid robot by using portable EEG device,”
Conf. Proc. ... Annu. Int. Conf. IEEE Eng. Med. Biol. Soc.
IEEE Eng. Med. Biol. Soc. Annu. Conf., vol. 2013, pp. 6905
6908, Jul. 2013.
[17] C. Escolano, J. M. Antelis, and J. Minguez, “A telepresence
Cáceres, C.A., Rosário, J.M., Amaya, D.
Copyright © 2016 Praise Worthy Prize S.r.l. - All rights reserved International Review of Mechanical Engineering, Vol. xx, n.
x
mobile robot controlled with a noninvasive brain-computer
interface.,” IEEE Trans. Syst. Man. Cybern. B. Cybern., vol.
42, no. 3, pp. 793804, Jun. 2012.
[18] P. Kumari and A. Vaish, “Brainwave based user
identification system: A pilot study in robotics environment,”
Rob. Auton. Syst., vol. 65, pp. 1523, 2015.
[19] A. C. Villa-Parra, D. Delisle-Rodríguez, A. López-Delis, T.
Bastos-Filho, R. Sagaró, and A. Frizera-Neto, “Towards a
robotic knee exoskeleton control based on human motion
intention through EEG and sEMGsignals,” Teodiano Bastos-
Filho/ Procedia Manuf., vol. 00, no. Ahfe, pp. 00, 2015.
[20] C. A. Cáceres Flórez, O. L. Ramos Sandoval, and D. Amaya
Hurtado, “Procesamiento de imágenes para reconocimiento
de daños causados por plagas en el cultivo de Begonia
semperflorens (flor de azúcar),” Acta Agronómica, vol. 64,
no. 3, pp. 273279, 2015.
[21] G. Louverdis, M. I. Vardavoulia, I. Andreadis, and P.
Tsalides, “A new approach to morphological color image
processing,” Pattern Recognit., vol. 35, no. 8, pp. 17331741,
2002.
[22] J. Smisek, M. Jancosek, and T. Pajdla, “3D with Kinect,” in
Consumer Depth Cameras for Computer Vision, London:
Springer London, 2011, pp. 325.
[23] J. Figueroa and L. Contreras, “Development of an Object
Recognition and Location System Using the Microsoft Kinect
TM Sensor,” Rob. 2011 Robot …, pp. 440449, 2012.
[24] A. Vilic, T. W. Kjaer, C. E. Thomsen, S. Puthusserypady, and
H. B. D. Sorensen, “DTU BCI speller: An SSVEP-based
spelling system with dictionary support,” in Proceedings of
the Annual International Conference of the IEEE Engineering
in Medicine and Biology Society, EMBS, 2013, pp. 2212
2215.
[25] F. T. Hvaring and A. H. Ulltveit-moe, “A Comparison of
Visual Evoked Potential ( VEP ) -Based Methods for the
Low-Cost Emotiv EPOC Neuroheadset,” Norwegian
University of Science and Technology, 2014.
[26] R. J. M. Godinez Tello, S. M. T. M??ller, A. Ferreira, and T.
F. Bastos, “Comparison of the influence of stimuli color on
steady-state visual evoked potentials,” Rev. Bras. Eng.
Biomed., vol. 31, no. 3, pp. 218231, Sep. 2015.
[27] T. Cao, F. Wan, P. U. Mak, P. I. Mak, M. I. Vai, and Y. Hu,
“Flashing color on the performance of SSVEP-based brain-
computer interfaces,” in Proceedings of the Annual
International Conference of the IEEE Engineering in
Medicine and Biology Society, EMBS, 2012, pp. 18191822.
[28] M. Aljshamee, M. Q. Mohammed, R.-U.-A. Choudhury, A.
Malekpour, and P. Luksch, “Beyond Pure Frequency and
Phases Exploiting: Color Influence in SSVEP Based on BCI,”
Comput. Technol. Appl., vol. 5, pp. 111118, 2014.
[29] P. Corke, Robotics, vision and control: fundamental
algorithms in MATLAB. Springer Science & Business Media,
2011.
[30] R. N. Jazar, “Inverse Kinematics,” Theory Appl. Robot., pp.
263296, 2007.
[31] Y. Guan, K. Yokoi, O. Stasse, and A. Kheddar, “On robotic
trajectory planning using polynomial interpolations,” 2005
IEEE Int. Conf. Robot. Biomimetics - ROBIO, pp. 111116,
2005.
[32] W. Speier, C. Arnold, and N. Pouratian, “Evaluating true bci
communication rate through mutual information and
language models,” PLoS One, vol. 8, no. 10, 2013.
Authors’ information
Camilo Andrés Cáceres Studied at UMNG
(Nueva Granada Military University), Bogotá,
Colombia reciving the B Sc. degree in
Mecatronics Engineering in 2014. Right now
is in the last stage of his M. Sc. In Mechanical
Engineering at Campinas State University,
São Paulo, Brazil. His interests are the
technology integration, brain computer
interfaces, signal and image processing,
software development, control and systems modeling, robotics and
optimization based in Meta-heuristics and bio-inspired methods.
João Maurício Rosário graduated in
Mechanical Engineer at the Universidade
Estadual de Campinas (1981), he hold a
Master in Industrial Automatic Production
from the Universite de Nancy I (1986), master
in Mechanical Engineering from Universidade
Estadual de Campinas (1984) and a PhD in
Automation, Specialist in Robotics from the
École Centrale des Arts et Manufactures Paris
(1990). He has experience in Mechatronics engineering, acting on the
following subjects: Robotics, Modelling and Control, and Industrial
Automation.
Dario Amaya was educated at UAN, Bogotá,
Colombia receiving the B Sc. degree in
Electronics Engineering in 1995 and the M.Sc.
degree in Teleinformatic in 2007 by the
Faculty of Engineering at the Francisco José
de Caldas District University, UFJC in
Bogotá, Colombia. He was awarded the Ph.D.
degree in 2011 in Mechanical Engineering at
Campinas State University, São Paulo, Brazil,
working on hybrid control. He had worked as a professor and researcher
at the Military University, Colombia since 2007 been involved in
Robotics, Mechatronics and Automation areas.
... Stimulus generation for the elicitation of the SSVEP can be based on light-emitting diodes LEDs [114] or monitors [46,51,101,103,108,115]. In this study, intermittent visual stimuli are presented on the computer screen. ...
... Different studies in robotic application obtained acquisition signals from the occipital/parietal brain areas. There are many interfaces used for acquisition signal such as Enobio (Neuroelectrics) with 32 electrodes [40], Epoc Headset (Emotiv) [51,101,137], Ultracortex (OpenBCI) [45,47], Biosemi [103], Neuracle [46,48], BioRadio (Great Lakes NeuroTechnologies) [39], DSI (Wearable sensing) [42], Mindo 4S (National Chiao Tung University Brain Research Center) [43,138], EEG amplifier Nuamps Express (Neuroscan) [44] and NeuSen W8 (Neuracle) [41]. See Table 8 for a summary. ...
... See Table 8 for a summary. The references reviewed used a combination of frequencies in the low and mediumrange [46][47][48]50,103,137] or in the low range [51,101], while the present study used frequencies in the medium range. This study used a black/white stimulus color as in [51], but with different frequencies, while other studies used stimulus color such as red, blue, and purple, or they used other types of stimuli such as images or names. ...
Article
Full-text available
Robotics has been successfully applied in the design of collaborative robots for assistance to people with motor disabilities. However, man-machine interaction is difficult for those who suffer severe motor disabilities. The aim of this study was to test the feasibility of a low-cost robotic arm control system with an EEG-based brain-computer interface (BCI). The BCI system relays on the Steady State Visually Evoked Potentials (SSVEP) paradigm. A cross-platform application was obtained in C++. This C++ platform, together with the open-source software Openvibe was used to control a Stäubli robot arm model TX60. Communication between Openvibe and the robot was carried out through the Virtual Reality Peripheral Network (VRPN) protocol. EEG signals were acquired with the 8-channel Enobio amplifier from Neuroelectrics. For the processing of the EEG signals, Common Spatial Pattern (CSP) filters and a Linear Discriminant Analysis classifier (LDA) were used. Five healthy subjects tried the BCI. This work allowed the communication and integration of a well-known BCI development platform such as Openvibe with the specific control software of a robot arm such as Stäubli TX60 using the VRPN protocol. It can be concluded from this study that it is possible to control the robotic arm with an SSVEP-based BCI with a reduced number of dry electrodes to facilitate the use of the system.
... Another case is presented in [4]: a Neuroevolutive controller has been designed, based on the kinematic model of a nonholonomic car-like robot, to avoid obstacles and to reach a goal without crashing. On the other hand, the usage of the mathematical models has been necessary to test and to control a robotic arm, adapted for disabled people, using brain signals, as presented by [5]. ...
Article
Full-text available
Nowadays, the design phase of any mechatronic system is mandatory before the implementation and the construction of a prototype. In the design phase, it is crucial to minimize failures, without under-dimensioning, any drive system in the desired prototype, using different simulation tools to test the proposed design under varied conditions.In this paper, the design, the simulation, and the control of a holonomic or omnidirectional mobile robot are presented. The design approach consists of a mathematically based method. The obtained model is an approach of the studied mobile robot, that allows an approach to the system’s behaviour, to a testing platform, and to a simulator for the desired robot, including most of the innerrobot’s interactions. The main contributions of this paper are the presentation and the study case of a proposed structured methodology for the study of any mechatronic-robotic system.
Article
The robotic arm is one of the most prominent manipulators used in the field of robotics and therefore serves as a benchmark problem in control systems. Recent developments in the field of artificial intelligence demonstrate the use of intelligent control techniques for various systems. This paper presents a neuro-fuzzy based control scheme for a robotic arm with 6 degrees of freedom used for the sorting and placing of coloured balls. A webcam based image acquisition scheme is used to locate and identify various colored balls and subsequently the robotic arm is moved according to required trajectory. The performance of the proposed neuro-fuzzy controller is compared with a traditional PID control to illustrate the superiority of the intelligent control scheme. Experiments performed on a hardware test bench validate the improvement in performance achieved using the neuro-fuzzy control scheme.
Chapter
Full-text available
We analyze Kinect as a 3D measuring device, experimentally investigate depth measurement resolution and error properties and make a quantitative comparison of Kinect accuracy with stereo reconstruction from SLR cameras and a 3D-TOF camera. We propose a Kinect geometrical model and its calibration procedure providing an accurate calibration of Kinect 3D measurement and Kinect cameras. We compare our Kinect calibration procedure with its alternatives available at the Internet and integrate it into an SfM pipeline where 3D measurements from a moving Kinect are transformed into a common coordinate system by computing relative poses from matches in its color camera.
Article
Full-text available
A BCI (brain computer interface) established a new direct communication channel using the brain activity between the human brain and machine. The visual stimulus with a certain frequency is present to the BCI users; it exists in a particular condition to observe a continuous brain response respect to frequent of visual stimuli. A significant problem when engaged the SSVEP (steady-state visual evoked potential) based on BCI, it will be exhausted and may suffer for the users when staring at flashing stimuli.This experimental study investigates how the differences in LED’s-colors influence of SSVEP with respect to (i.e., frequencies and phases). The result shows that the visualization of phase delays in lower frequencies greater than in higher frequencies.
Article
Full-text available
En este trabajo se presentan los resultados obtenidos en el reconocimiento de plagas utilizando la visión de máquina por computador como elemento de diagnóstico. La captura de las imágenes se realizó por medio de un agente robótico aéreo (drone) equipado con una cámara, lo que permitió capturar las imáge-nes del estado de las hojas de un cultivo de la planta conocida como ‘flor de azúcar’ (Begonia semperflo-rens). Estas imágenes fueron procesadas utilizando técnicas de visión de máquina con el fin de identificar el posible ataque de plagas en el cultivo. Las técnicas utilizadas corresponden a filtros morfo-lógicos, difuminado gaussiano y filtrado HSL. Como resultado principal de este trabajo se detectaron perforaciones de hojas ocasionadas por el ataque de plagas, específicamente babosas, caracoles, arañas rojas y minadoras. </div
Article
Full-text available
Introduction The main idea of a traditional Steady State Visually Evoked Potentials (SSVEP)-BCI is the activation of commands through gaze control. For this purpose, the retina of the eye is excited by a stimulus at a certain frequency. Several studies have shown effects related to different kind of stimuli, frequencies, window lengths, techniques of feature extraction and even classification. So far, none of the previous studies has performed a comparison of performance of stimuli colors through LED technology. This study addresses precisely this important aspect and would be a great contribution to the topic of SSVEP-BCIs. Additionally, the performance of different colors at different frequencies and the visual comfort were evaluated in each case. Methods LEDs of four different colors (red, green, blue and yellow) flickering at four distinct frequencies (8, 11, 13 and 15 Hz) were used. Twenty subjects were distributed in two groups performing different protocols. Multivariate Synchronization Index (MSI) was the technique adopted as feature extractor. Results The accuracy was gradually enhanced with the increase of the time window. From our observations, the red color provides, in most frequencies, both highest rates of accuracy and Information Transfer Rate (ITR) for detection of SSVEP. Conclusion Although the red color has presented higher ITR, this color was turned in the less comfortable one and can even elicit epileptic responses according to the literature. For this reason, the green color is suggested as the best choice according to the proposed rules. In addition, this color has shown to be safe and accurate for an SSVEP-BCI.
Article
Full-text available
Brain-computer interface (BCI) systems are a promising means for restoring communication to patients suffering from "locked-in" syndrome. Research to improve system performance primarily focuses on means to overcome the low signal to noise ratio of electroencephalogric (EEG) recordings. However, the literature and methods are difficult to compare due to the array of evaluation metrics and assumptions underlying them, including that: 1) all characters are equally probable, 2) character selection is memoryless, and 3) errors occur completely at random. The standardization of evaluation metrics that more accurately reflect the amount of information contained in BCI language output is critical to make progress. We present a mutual information-based metric that incorporates prior information and a model of systematic errors. The parameters of a system used in one study were re-optimized, showing that the metric used in optimization significantly affects the parameter values chosen and the resulting system performance. The results of 11 BCI communication studies were then evaluated using different metrics, including those previously used in BCI literature and the newly advocated metric. Six studies' results varied based on the metric used for evaluation and the proposed metric produced results that differed from those originally published in two of the studies. Standardizing metrics to accurately reflect the rate of information transmission is critical to properly evaluate and compare BCI communication systems and advance the field in an unbiased manner.
Article
Full-text available
Brain Computer Interfaces (BCIs) are systems that allow human subjects to interact with the environment by interpreting brain signals into machine commands. This work provides a design for a BCI to control a humanoid robot by using signals obtained from the Emotiv EPOC [11], a portable electroencephalogram (EEG) device with 14 electrodes and sampling rate of 128 Hz. The main objective is to process the neuroelectric responses to an externally driven stimulus and generate control signals for the humanoid robot Nao accordingly. We analyze steady-state visually evoked potential (SSVEP) induced by one of four groups of light emitting diodes (LED) by using two distinct signals obtained from the two channels of the EEG device which reside on top of the occipital lobe. An embedded system is designed for generating pulse width modulated square wave signals in order to flicker each group of LEDs with different frequencies. The subject chooses the direction by looking at one of these groups of LEDs that represent four directions. Fast Fourier Transform and a Gaussian model are used to detect the dominant frequency component by utilizing harmonics and neighbor frequencies. Then, a control signal is sent to the robot in order to draw a fixed sized line in that selected direction by BCI. Experimental results display satisfactory performance where the correct target is detected 75% of the time on the average across all test subjects without any training.
Article
The world is seeing the emergence of Brainwaves computing and its application in our daily communication and security is on horizon. One of the interesting area interaction modality is robotic environment where involvement of illegitimate users may cause severe threat to environment thereby, a continuous monitoring is mandatory. In this paper, we have presented a study of potential of brainwaves and it’s computing for providing higher security. This work utilizes EEG signal generated by visual stimuli for keying in the password. In order to construct a model for user authentication, we have selected the time-frequency analysis method called wavelet analysis to decompose original signal corresponding to EEG sub-band frequency, thereby extracted statistical measures and energy calculation of each decomposed waves. Furthermore, we have selected and employed the Neural Network based on learning vector quantization for correct classification. Classification rate has been calculated over the different combination of channels.KeywordsEEG signal processingWavelet transforms and authenticationAutonomous system security
Conference Paper
This paper presents the development of an object recognition and location system using the Microsoft KinectTM, an off-the-shelf sensor for videogames console Microsoft Xbox 360TM which is formed by a color camera and depth sensor. This sensor is capable of capturing color images and depth information from a scene. This vision system uses a) data fusion of both color camera and depth sensor to segment objects by distance; b) scale-invariant features to characterize and recognize objects; and c) camera’s internal parameters combined with depth information to locate objects relative to the camera point of view. The system will be used along with a robotic arm to grab objects. ‘
Conference Paper
This paper presents a user-friendly human machine interface (HMI) for hands-free control of an electric powered wheelchair (EPW). Its two operation modes are based on head movements: Mode 1 uses only one head movement to give the commands, and Mode 2 employs four head movements. An EEG device, namely Emotiv EPOC, has been deployed in this HMI to obtain the head movement information of users. The proposed HMI is compared with the joystick control of an EPW in an indoor environment. The experimental results show that Control Mode 2 can be implemented at a fast speed reliably, achieving a mean time of 67.90 seconds for the two subjects. However, Control Mode 1 has inferior performance, achieving a mean time of 153.20 seconds for the two subjects although it needs only one head movement. It is clear that the proposed HMI can be effectively used to replace the traditional joystick control for disabled and elderly people.
Conference Paper
Recent research has shown the benefits of using robotic devices to aid in stroke patient rehabilitation. In addition, the use of brain computer interfaces is showing promising applications in both controlling robotic devices and aiding in neural rehabilitation. Unfortunately, traditional rehabilitative devices are cumbersome and are not being used outside the laboratory environment, therefore potentially limiting patient access to these new rehabilitative technologies. This project seeks to address this issue by working toward creating a portable rehabilitative system consisting of an arm exoskeleton controlled by a brain computer interface (BCI). A wireless commercially available BCI system was used (Emotiv EPOC) to actuate the device in a single direction. A pilot study was conducted to evaluate the performance of the system on four healthy adult volunteers. Results show all users were able to control the device with success rates above chance.