Content uploaded by Jorge Ferraz Abreu
Author content
All content in this area was uploaded by Jorge Ferraz Abreu on May 04, 2020
Content may be subject to copyright.
Evaluation Methodologies of Assistive Technology
Interaction Devices: A participatory mapping in
Portugal based on community-based research
Andreia Sias Rodrigues∗
A.S. Rodrigues∗
andreiarodrigues@ifsul.edu.br
Instituto Federal Sul-riograndense
Pelotas, RS
Marcelo Bender Machado
Instituto Federal Sul-riograndense
Pelotas, Brasil
marcelo@ifsul.edu.br
Ana Margarida Pisco Almeida
Universidade de Aveiro
Aveiro, Portugal
Jorge Ferraz de Abreu
Universidade de Aveiro
Aveiro, Portugal
Tatiana Aires Tavares
Universidade Federal de Pelotas
Pelotas, RS, Brasil
ABSTRACT
A great challenge on Assistive Technology (AT) is the eval-
uation phase of the interaction devices. Generally the gap
between users and devices is discovered during the use. This
fact highlights the relevance to discuss this barriers early in
the design process, specially, in the evaluation phase. This
paper explores current practices about evaluation method-
ologies for Assistive Technology (AT) interaction devices
for people with motor disabilities, both considering the use
of computers and TV-sets. To achieve this objective the re-
sults from a systematic mapping study about AT evaluation
methodology and a eld research was done in Portugal with
experts in Assistive Technology Evaluation and end users.
This eld research was done, with a semi-structured inter-
views and for the data analysis, a content analysis of these
interviews was carried out. The ndings from this research
suggest that there are key dimensions that should be ad-
dressed when evaluating AT: qualication issues remain
most important, especially the developers’ awareness of end
user needs as barrier on computer access. Other important is-
sues are economic barriers involved in the acquisition of AT
interaction devices as well as the emotional issues involved
in using these devices. Finally, the paper concludes with a
∗Both authors contributed equally to this research.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for prot or commercial advantage and that copies bear
this notice and the full citation on the rst page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior specic permission and/or a fee. Request
permissions from permissions@acm.org.
IHC ’19, October 22–25, 2019, Vitoria - ES, Brazil
©2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-6971-8/19/10.. .$15.00
https://doi.org/10.1145/3357155.3358458
discussion on directions for evaluation methodologies of AT
interaction devices.
CCS CONCEPTS
•Human-centered computing →Field studies.
KEYWORDS
assistive technology, evaluation methodologies, interaction
device, people with desabilities
ACM Reference Format:
Andreia Sias Rodrigues, A.S. Rodrigues, Marcelo Bender Machado,
Ana Margarida Pisco Almeida, Jorge Ferraz de Abreu, and Tatiana
Aires Tavares. 2019. Evaluation Methodologies of Assistive Technol-
ogy Interaction Devices: A participatory mapping in Portugal based
on community-based research. In Proceedings of the 18th Brazilian
Symposium on Human Factors in Computing Systems (IHC ’19), Oc-
tober 22–25, 2019, Vitoria - ES, Brazil. ACM, New York, NY, USA,
9 pages. https://doi.org/10.1145/3357155.3358458
1 INTRODUCTION
The use of information and communication technologies
(ICT) is part of our daily activities. ICTs have undoubtedly
become important instruments for inclusion and interaction
in our society [
24
]. In other words, ICT have transformed
the way people live, the way that people study, stay in touch
with family and friends and, in general, how people spend
their free time. In the specic case of people with disabil-
ities, ICT have an important role in increasing their level
of safety, independence, and fostering social connectedness,
and assisting their daily-life routines [39].
Assistive Technology (AT) refers to the use of technologi-
cal devices or software to assist end users’ daily lives regard-
less their level of disability. In fact, ICT assumes the form of
AT when the goal is the computer use itself, in which certain
technical aids are necessary to enable or facilitate a task (for
example, adaptations in software, keyboard and mouse, etc.).
IHC ’19, October 22–25, 2019, Vitoria - ES, Brazil Rodrigues A. S., et al.
Commercially available o-the-shelf input devices (i.e. key-
board and mouse) can compromise the end users’ computer
access (especially of those with motor disability). Evaluating
how are these interaction devices targeting end users’ needs
is of major importance.
Usability and Accessibility [
19
] are particularly relevant to
investigate evaluation processes and to support the design of
evaluation methodologies [
5
,
6
,
29
,
38
] For the International
Organization for Standardization (ISO), Usability refers to
the degree to which a product can be used by specic users
to achieve determined goals with eectiveness, eciency
and satisfaction, in a context of specic use [18, 30]
More recently, the Usability concept has been continu-
ally rebuilt and it has become increasingly rich and complex.
Usability now includes aspects such as fun, well being, collec-
tive eectiveness, aesthetics, creativity, human development
support, among other dimensions. The current understand-
ing of Usability is therefore dierent from the concept given
in the early steps of HCI in the 1980s. At the turn of the
century, the rise of digital services (e.g., the web and smart-
phones) added new concerns to HCI, giving rise to yet an-
other signicant concept than usability: the User Experience
(UX) [
5
,
10
] According to [
15
] the UX goes beyond eciency,
quality of tasks and satisfaction, as it considers the hedonic,
cognitive, aective, emotional, social and physical aspects
of the interaction.
Considering the evaluation methods context applied to AT,
the main challenge is how to ensure that a device meets users’
needs and adapts to users rather than requiring users to adapt
to it. Although this is generally neglected, it is particularly
important when it comes to users with special needs and
abilities. According to [
11
], one of the factors associated
with improper use and consequent abandonment by users
is the lack of involvement of users and of their relatives in
the designing process and evaluation of the AT devices [
11
]
has shown that around 35% of the purchased AT devices
are unused and abandoned, causing frustrations or sense of
defeatism led by product disappointment. This is partially
due to the little care given to the end user evaluation of the
developed product or even in the prototype phase or even
to the lack of an evaluation method applied to people with
disabilities.
It is under this scenario that the study presented in this
paper aims at understanding the evaluation of AT interac-
tion devices in the view of nine AT experts and nine AT
end users in Portugal, its functionalities, use contexts, main
problems and its rationale in order to nd evaluation prac-
tices/procedures in AT interaction. An overview of a system-
atic mapping study focusing on evaluation methodologies of
interaction AT devices is also described to better frame the
problem being addressed.
2 SYSTEMATIC LITERATURE MAPPING (SLM)
A systematic literature mapping [
34
] was adopted in or-
der to carry out a contextualising of the area of Assistive
Technology Evaluation Methodologies in Human-Computer
Interaction with a focus on motor disability. Table 1 shows
an overview of Systematic Mapping results, including the
evaluation methodologies used in works, the results presen-
tation (quantitative or qualitative), the duration (if they were
punctual or temporal), and if evaluation experts and end
users were involved. The entire protocol can be found in
[37].
As reported in literature, this review revealed that the Us-
ability aspect (testing methods, interviews and task analysis)
is the most frequently used, which seems to underline the
role that end users recognition has as a knowledge source.
However, the methods are still very focused on assessing
the device itself in terms of performance and functionality
rather than on the disability issue.
It is important to note that studies that report using Us-
ability or user experience evaluation methods do not de-
scribe how they developed it, despite the recognition of its
importance. In the design, development and evaluation of
any interaction product, the human aspects are considered
from the beginning, and when it comes to AT products it is
expected that these human factors will be exploited to the
maximum, precisely because these end users present speci-
city’s that are not known by developers and they need to
be identied.
The evaluations data analysis in 94% of the works is pre-
sented in quantitative form, by means of numerical metrics,
comparisons between devices and tasks execution time. A
small number of articles reporting evaluation experiments
including experts and people with motor disabilities have
been identied in this Systematic Mapping.
In [
17
] the evaluation of the computer interaction device
also focused on performance analysis of the device, but in-
volved 3 users with severe motor disabilities. The evaluation
was a performance metrics comparison between two pre-
viously used interaction devices and the proposed device
prototype, and the results expressed at task execution times.
This work did not involve any expert in the evaluation phase.
Eleven of the seventeen papers [
2
,
7
,
13
,
16
,
20
,
22
,
23
,
25
,
27
,
28
,
41
] developed prototypes of computer interaction
devices for people with disabilities, conducted task-based Us-
ability evaluations, all involving people without disabilities,
to validate the proposal. None of these works involved ex-
perts or end users in the AT prototype evaluation processes
developed. The duration of the evaluations was punctual, in
a single session. The results presented were in quantitative
form.
Evaluation Methodologies of AT Interaction Devices: A participatory mapping in Portugal IHC ’19, October 22–25, 2019, Vitoria - ES, Brazil
Table 1: Overview of systematic mapping results
Evaluation Methods Reference Results Duration
Experts
End users
Usability Metrics analysis - Task based
[
2
,
7
,
13
,
16
,
20
,
22
,
23
,
25, 27, 28, 41]
Quantitative
Single Session
- -
Usability Metrics Analysis and User Expe-
rience
[1, 21, 32, 40] Quantitative Single session - 1
Usability Metrics Analysis and satisfaction
query
[17] Quantitative Single session - 3
Observation, Therapeutics session [26] Qualitative
Multiple ses-
sions
3 7
In the work proposed by [
1
,
21
,
32
,
40
] Usability aspects
were evaluated in their interaction devices proposals, with
metrics of performance, precision, sensitivity and task ex-
ecution speed of these prototypes, all results expressed in
quantitative form. They involved one person with disability
in these processes and no experts in evaluation. Regarding
the tests duration, these were performed in a single session.
The work proposed by [
17
], a prototype dTDS (dual Tongue
Drive System) designed to enable people with disabilities to
use computers using the tongue, in addition to the analy-
sis of task execution metrics also collected the satisfaction
questionnaire, involving in its evaluation process three end
users. But the main purpose of this study was to compare
the performance of TDS, SR software (Dragon) and dTDS in
completing comprehensive computer access tasks and dis-
cover the strengths and weaknesses of each modality. They
suggests that the results could be used to develop advanced
AT that will take advantage of the best modalities that are
available to potential end users for achieving maximum per-
formance. However, the evaluation was completely focused
on the prototype analysis itself and no expert was involved.
The experiment happened in one session.
In only one of the papers [Manresa Yee et al. (2013)]
there was the deciency prescription, functionalities, user
needs and a qualitative analysis of the results, reporting
the end user experience, the diculties, the gains involved.
SINASense is a motionbased interface that oers engaging
ways of interaction to users with severe to profound disabil-
ities, who have never accessed a computer. In the design
phase they counted the end users needs and goals compiled
by therapists and parents. The authors were concerned to
carry out a preliminary evaluation of three months with the
prototype system with disabled children between 4 and 12
years of age. All sessions were accompanied by a therapist
who supported and guided the user during the interaction.
The results showed that the proposal promoted their active
participation and engagement with what was happening
around them and increased their response and they discuss
these results with qualitative metrics, such as the intentional
frequency of a movement, being classied in a very low in-
crease, low increase and high increase. Evaluation experts in
AT analyzed the evaluations along with the therapeutic sta
of the users. Due to these multiple knowledge the results
have made reference to improved quality of the movements
allied to the ability to use the interaction device.
In order to know how the TA evaluation processes are
done in the eld, an investigative study was carried out in
Portugal with specialized professionals and interviews with
end users were also made to know the main challenges faced,
what are the methodologies used and to discuss with the
results of the mapping.
3 COMMUNITY-BASED RESEARCH
An inquiry was conducted with Portuguese experts in AT
evaluation and Portuguese AT end users aiming to under-
stand how they both perceive the AT interaction devices, and
what evaluation methods they use. Eighteen semi-structured
interviews were applied (9 to experts and 9 to end users with
motor disability) aiming also to understand the possibility of
proposing new evaluation processes and/or the adaptation
of existing methodologies.
Regarding the prole, seven experts were connected to
the academy (with experience in occupational therapy) and
two of them, besides belonging to the academic area, were
also linked to AT product industry, details in the Table 2.
The attempt was to achieve at least the same number of
experts interviewees. Inclusion criteria were to invite end
users of AT interaction devices, as indicated by the experts
interviewed. There were 9 subjects (2 females, aged between
19 and 49 years), as Table 3 shows. All the end users signed
a consent form approved by our institutional review board
and authorised the audio interview recording as well as au-
thorised to use it for academic purposes.
The interviews’ protocols were created aiming to select
questions in order to highlight these issues and also to ex-
plore other directions that could help to better understand
IHC ’19, October 22–25, 2019, Vitoria - ES, Brazil Rodrigues A. S., et al.
Table 2: Professional qualications of the experts inter-
viewed
Experts Assignments
Exp1
Engineering Professor. Coordinator of the Center for
Rehabilitation and Accessibility Engineering and Pres-
ident of the Portuguese Society of Rehabilitation Engi-
neering, Support and Accessibility Technologies. The
expert has 20 years of professional experience in the
eld of Rehabilitation Engineering.
Exp2
Professor at the Polytechnic School
Exp3
Coordinator of the Resource Center for Digital Inclu-
sion of the Polytechnic Institute.
Exp4
Superior School of Education Professor and Social
Sciences of the Polytechnic Institute. Eective mem-
ber of the scientic committee and the coordinating
commission of the Center for Research Inclusion and
Accessibility in Action of Portugal
Exp5 Occupational Therapist. Professor and Coordinator of
the Department of Health Sciences and Technologies
at the Health School of the Polytechnic Institute
Exp6 Special Education Teacher (Ministry of Education and
Science) and Coordinator of Resource Centers for Spe-
cial Education
Exp7
Businessman of assistive technology products and
Management and Technology Professor
Exp8 Deputy Director of the Gerontology Degree and Mem-
ber of the Oce for Education and Quality
Exp9
Engineering Department Professor
Table 3: End users prole
Disability Age Sex Employment
U1
Cerebral palsy
46 M professor
U2
Cerebral palsy
37 M Graphic editor
U3
Cerebral palsy
45 M Advocate
U4 Paraplegic 28 M Freelance Editor
U5 Paraplegic 32 M Photographer
U6 Paraplegic 35 M Unemployed
U7
Cerebral palsy
19 M Student
U8
Cerebral palsy
49 F Unemployed/Retired
U9
Cerebral palsy
21 F Unemployed
existing gaps in the AT interaction devices evaluation pro-
cesses. The interviews’ main focus was regarding interac-
tion devices both for the Television and for the Computer
scenarios. The rst part of the interview, about the use of
interaction devices, was applied both to experts and to end
users, whereas the second part, more directly to experts, was
about evaluation methods of these devices.
The interviews were recorded, verbatim transcribed and
then coded [
3
], following ve phases for its conduction: anal-
ysis organisation; coding; categorisation; treatment of re-
sults; inference and interpretation of results.
Obtained Results
Regarding the evaluation methods used by the experts, con-
trary to the scenario found in the systematic mapping, the
most widespread techniques are: the case-by-case analysis,
since the experts pointed out that they rst identify the
functional parts of the end user’s body, study medical pre-
scription, and only after they that reect on the evaluation
method to be used, that is, according to the precepts of As-
sistive Technology the human aspects are the priority in AT
devices evaluations [31].
The multi-methods technique, a methods combination
that could prioritise the well-being of those involved, was
the answer that all interviewees gave to the was the answer
that all respondents gave about the evaluation method they
use.
The most cited method was the empirical one, followed by
trial and error (see Figure 1). Experts have quoted that each
evaluation is based in learning from the experience, once
they learn facts through lived and witnessed experiences to
then draw conclusions. This seems to be especially important
when dealing with the issue of disability. One Expert cited:
”We conduct evaluations based on empirical data. We collect
reports from caregivers, parents, relatives and therapists. We
studied the clinical report - to understand what led the person
to the condition of the disability and to know or try to study
the evolution of the disability (future perspectives). In order to
consider the learning time of the solution and the degenerative
severity that it may have, it does not cause further hopes and
frustrations in a product that, at a given time, may not meet
the end user needs”. Empirical knowledge is sensitive and
subjective and in experts opinion there are no assessment
scales available that can be used to support this practice.
Figure 1: Evaluation methods used by AT experts
Evaluation Methodologies of AT Interaction Devices: A participatory mapping in Portugal IHC ’19, October 22–25, 2019, Vitoria - ES, Brazil
Only two experts cited Usability methods, such as device
functionality, performance metrics and user satisfaction ques-
tionnaires. But only in situations where the user’s disability
is very light and requires few adaptations.
However, no expert cited any UX methodology and this is
somehow aligned with the results found in the systematic
mapping: the experts sample interviewed size confers with
the systematic mapping, in the sense that only 1 of the 17
works found has performed an evaluation based on the UX
methodology.
As for the dimensions that are important to consider dur-
ing evaluation, the experts and end users stressed the impor-
tance of knowing the end user historic: as shown in Figure 2,
seven experts and seven end users highlighted the record
unit related to Frustrations (several attempts to use an AT
for the same purpose without success).
Figure 2: Dimensions that end users and experts consider
most important to be evaluated
In order to facilitate the overview and to obtain knowledge
about the main problems and objectives in the computer and
television access by the people with disabilities, the interview
codication, experts and end users, sought to identify general
aspects that could help in the improvement of evaluation
processes more inclusive.
This coding resulted in a priory category, called barriers,
presents the results concerning the barriers to the domain
of to computer and television access, and has the following
subcategories: economic barriers, AT knowledge barriers,
technical barriers and emotional barriers.
Barriers
The objective of this section was to identify barriers to com-
puter and TV access by people with disabilities from the
perspective of experts and end users and to discuss whether
existing evaluation methods could transpose or help to re-
duce such barriers.
The results obtained are already known, but not yet solved,
it is important to highlight the barriers that still aect people
with disabilities. The barriers identied were Economic Bar-
riers, Knowledge Barriers, Technical Barriers and Emotional
Barriers.
The rst record units group represent the economic bar-
riers for computer access and are presented in Figure 3. AT
interaction devices are usually produced in small quantities
for specialised markets and therefore tend to be expensive
[11].
Figure 3: Economic barriers on computer access
A range of barriers to computers and tv access were iden-
tied including the Portugal AT workers professional quali-
cation and the dissemination about what already exists in
terms of computer and tv accessibility. Record units collected
from the interview regarding this group related to computer
access are presented on Figure 4.
According to the experts, there is little evidence that med-
ical professionals receive adequate training in assistive tech-
nology to prepare them to prescribe assistive technology to
their patients, as quoted Exp3: ”The knowledge in the medical
profession with people with disabilities are concerned with ther-
apy and physical therapy, but they are not prepared to prescribe
and suggest AT equipment - they care about health, but leave
them out of the digital life. Doctors, and health professionals
do not have this training.”
The end users speech shows that the professionals with
whom they contact, should know to guide them better in
the choices of AT, as cited by U3: ”Professionals often indicate
IHC ’19, October 22–25, 2019, Vitoria - ES, Brazil Rodrigues A. S., et al.
equipment that is too expensive or will take too long to be
procured with the help of the government, and often occurs
that hey themselves have never seen the equipment being used,
we are almost always disoriented”.
Figure 4: Record units obtained from interview text related
to the AT Qualication barriers on computer access.
The barriers most cited by end users refer to the technical
issues, shown in Figure 5. These barriers will have conse-
quences on emotional barriers, and these barriers can be
mitigated by appropriate Usability evaluation methods, for
example, with the adaptations reported in this paper.
Figure 5: Record units obtained from interview text related
to the technical barriers on computer access.
Other important barriers identied in the interviews were
of emotional aspects, as cited by end user, named U7: ”I like to
communicate with people, but the keyboards are not ecient, I
nd them ugly and very tiring, even with the virtual keyboards,
so I prefer to listen to music during laser hours instead of failure
in the activity”.
As [
9
] underline, psychological and social-emotional de-
sign requirements are of most importance when creating new
assistive technology; they suggest that sensitivity to fam-
ily values and cultural dierences must be included. Record
units collected from the interview regarding emotional bar-
riers related to computer access are presented on Figure 6
.
Figure 6: Record units obtained from interview text related
to the Emotional barriers on computer access.
It’s worth mentioning that as these products highlight
users disabilities they may, in the end users opinion, con-
tribute to social stigma associated with disability, hindering
adoption of these devices: as end user, named U2, quoted ”...
usually these AT devices are very ugly and tiresome, I always
try to use what everyone else uses, even though I need a lot of
time to adapt.”. Following this, ve (5) end users reported
that even with large and diverse diculties they prefer to
adapt to traditional devices, which users without disabilities
use.
4 DISCUSSION
Economic and knowledge barriers have been identied and
highlighted, but evaluation methods fail to address these
problems.
About Technical Barriers
For the technical barriers identied in this research, the HCI
eld is able to assist in the design phase as well as in the
evaluation phase. However, according to experts, and also
denoted by the literature, considering AT devices evaluated
using existing methods, it has failed with regard to people
with disabilities.
Evaluation Methods HCI. According to experts’ answers, the
importance of human aspects in evaluations is evident. Al-
though HCI evaluation methodologies known as Usability
and User Experience are extremely relevant and recognised,
in considering aspects of AT this methodologies still focus on
the issue of the device to the detriment of the users’ needs.
Some Usability metrics, such as speed, time, precision in
the execution of tasks, that were the quantitative metrics
found in the literature used to validate AT devices are not so
relevant when considering AT devices.
Evaluation Methodologies of AT Interaction Devices: A participatory mapping in Portugal IHC ’19, October 22–25, 2019, Vitoria - ES, Brazil
Regarding the evaluation methods used by the experts,
contrary to the scenario found in the systematic mapping,
the most widespread techniques are: the case-by-case anal-
ysis, since the experts pointed out that they rst identify
the functional parts of the end user’s body, study medical
prescription, and only after they that reect on the evalua-
tion method to be used, that is, according to the precepts of
Assistive Technology the human aspects are the priority in
AT devices evaluations [4].
To exemplify, the proposal in [
31
], was the use of an auto-
mated sphere promotes playing with children with cerebral
palsy. The entire design process was executed with prescrip-
tion from occupational therapists, physicians, parental and
caregiver assistance, and observation of children. The eval-
uation occurred in several sessions, therapists and family
members observed the children interaction with the sphere
and as a conclusion the gain was in being able to promote
play. The project was designed for a child, considering their
specic needs and was evaluated with the same child.
Results Analysis. Usability aspects were evaluated in their
interaction devices proposals, with metrics of performance,
precision, sensitivity and task execution speed of these pro-
totypes, all results expressed in quantitative form and these
analyses focused on the metrics collected are not enough to
validate them as Assistive Technology devices.
It is important to note that studies that report using Usabil-
ity or user experience evaluation methods do not describe
how they developed it, despite the recognition of its impor-
tance. In the design, development and evaluation of any in-
teraction product the human aspects are considered from the
beginning, and when it comes to AT products it is expected
that these human factors will be exploited to the maximum.
These end users present specicity that are not known by
developers and they need to be identied, either with the
involvement of the family, the caregivers and always with
end user support.
According to [
31
], multidisciplinary aspects related to the
prescription, skills and needs of the user must be considered
before developing a prototype and then used to validate the
proposal.
People involved in evaluations. This case study [
31
] contra-
dicts the results found in this literature review, where most
of the works develop interaction AT prototypes for people
with disabilities and after evaluating with multiple people
without any disability. And considering AT, a device that can
be successful with one person, can have completely dierent
results with another. As Assistive Technology, by denition,
deals with accessibility features that are specically aimed
at people with disability, impairments or reduced mobility
[12].
A small number of articles reporting evaluation experi-
ments including experts and people with motor disabilities
have been identied in this Systematic Mapping. This reports
the diculty of involving the experts and the end users both
in the project and in the evaluation phases.
When the involvement of people with disabilities occurs,
the result is presented in a quantitative way, with analysis
of task execution metrics and these metrics are not enough
to validate them as ecient Assistive Technology devices.
The end user responses presented in this paper highlight this
statement.
Another important aspect is the presence of a expert in AT
product evaluation. In only one of the articles, the evaluation
process was attended by a professional with experience in
evaluating assistive technology products with people with
disabilities, in [
26
] all evaluate sessions were accompanied
by a therapist who supported and guided the end user during
the interaction.
This connection of the technological area with the clinical
aspects, according to experts interviewed is essential, due to
the experience obtained, knowledge about the deciency, the
person’s history and AT prescription and the own analysis
of the device [14].
Duration of Experiments. Most of the works in the system-
atic mapping perform a single evaluation session to validate
the developed AT interaction device prototype, but a point
evaluation may not respond properly.
First of all, because in many cases people, especially people
with disabilities, feel that moment, a moment of excitement,
they have the emotional aspect changed, inevitably and tak-
ing into account all history, they usually feel evaluated.
So even if they did not like it, they would not feel comfort-
able rejecting the proposal, and therefore if the evaluation
had any proposal for self-report, it would probably bring
positive results, but with much interference in that sense.
In the case of the person performing tasks, experimenting
with new prototypes, can bring inexpressive numeric results,
because those metrics can be completely insignicant and
opposed to the desire and goals of the end users.
For the experts, a temporal assessment, which takes into
account learning in the use of a new device, requires a pe-
riod of adaptation and acceptance. In the work of [
26
] the
evaluation period was three months with a frequency of
weekly evaluate sessions, where they could observe the im-
provement or not in the device use and if the learning was
eective.
About Emotional Barriers
It is clear that it is the emotional barriers that are the most
dicult to detect and the most complex to be solved, because
IHC ’19, October 22–25, 2019, Vitoria - ES, Brazil Rodrigues A. S., et al.
in many cases people with disabilities can not express them-
selves, they simply abandon their devices and thus generate
greater frustrations.
The issue of frustration has been much cited by the experts
and this also contrasts with the opinions of end users that
they try to adapt to what already exists, since they do not
want to use other devices that are ugly or that have a negative
eect on aesthetics, responsibility is assigned to the end user.
In the literature, there are dierent approaches have been
used to assess the emotional response of users, [
33
,
35
,
43
].
One of the paths that is presented is to make use of self-report
instruments such as Questionnaires, Think Aloud [
42
] , The
Self-Assessment Manikin (SAM)[
8
] and Emocards [
36
]. In
addition, another alternative for emotional evaluation is the
use of physical and logical sensors (for example, checking
heart rate and user interaction time) as a way of collect-
ing data that can be modelled and analysed to evaluate the
emotional response of user.
In this work, the technical barriers were rst addressed,
since we believe that if these barriers are reduced they will
aect the emotional barriers. Taking into account the hu-
man aspects from the beginning, including health profession-
als and the therapeutic area in the evaluations, providing a
temporal evaluation, are some of the actions that will have
consequences on the emotional barriers.
Technical barriers were rst addressed, since we believe
that if these barriers are reduced, they will aect emotional
barriers. Taking into account human aspects from the begin-
ning, including social, family, user history, including health
professionals, or the therapeutic area, also considering the
opinion of technical experts in evaluation, providing a tem-
poral assessment, more heuristic, are some of actions that
will have consequences on emotional barriers.
5 CONCLUSION
This research presented the literature results on the evalua-
tion methods used in AT interaction devices in the computer
and tv access. An inquiry was also carried out on the AT eval-
uation methods interaction devices, with experts in assistive
technology evaluation and also with end users.
The main research objective was to deepen knowledge
on the AT interaction devices evaluation process, trying to
better understand if there are possible gaps in this domain,
aiming to propose new strategies that would lead to the
creation of new or adapted evaluation methodologies that
could ultimately increase the eectiveness of these devices
regarding the disability issue.
According to the end users and experts, there are many
barriers to access to information technology and TV, with au-
tonomy, for people with disabilities, after coding grouped in
economic barriers, barriers to knowledge, technical barriers
and emotional barriers.
All of them are already known, but we have identied
them to highlight and also to nd where evaluation methods
could be useful in reducing the barrier.
Economic and knowledge barriers have been identied
and highlighted, but evaluation methods fail to address these
problems.
These dierent challenges, technical and emotional barri-
ers discussed here are presented in order not to bring answers
and nal solutions to the problems, but rather to highlight
the need for continuity in the process of deepening and
seeking greater attention and relative care to people with
disabilities, in order to avoid or overcome the distortions en-
countered, which make it dicult to achieve the objectives
to be achieved through TA.
There are a number of methods, both design and evalu-
ation, but there are specic points that need to be covered
in the development and evaluation processes of interaction
Assistive Technology devices. Particularly in the human as-
pects considered from the beginning, it is fundamental to
aggregate the clinical area in the evaluation processes of
HCI AT devices, considering the AT concept as a multidis-
ciplinary area. In this way we believe that the emotional
barriers will also be mitigated.
ACKNOWLEDGMENTS
The author would like to thank all the experts and end users
who kindly dedicated themselves to participating in the in-
terviews.
REFERENCES
[1]
Adriano O Andrade, Adriano A Pereira, Carlos G Pinheiro Jr, and
Peter J Kyberd. 2013. Mouse emulation based on facial electromyogram.
Biomedical Signal Processing and Control 8, 2 (2013), 142–152.
[2]
Rui Azevedo Antunes, Luís Brito Palma, Fernando V Coito, Hermínio
Duarteramos, and Paulo Gil. 2016. Intelligent human-computer inter-
face for improving pointing device usability and performance. Control
and Automation (ICCA), 2016 12th IEEE International Conference on
(2016), 714–719.
[3]
Laurence Bardin. 2010. Análise de conteúdo.(1977). Lisboa (Portugal):
Edições 70 (2010), 225.
[4]
Rita BERSCH et al
.
[n.d.]. Fatores humanos em tecnologia assistiva:
uma análise de fatores críticos nos sistemas de prestação de serviços.
Revista Plurais, Salvador: UNEB 1, 1 ([n. d.]), 34–41.
[5]
Nigel Bevan. 2008. Classifying and selecting UX and usability measures.
In International Workshop on Meaningful Measures: Valid Useful User
Experience Measurement, Vol. 11. 13–18.
[6]
Nigel Bevan, Nigel Claridge, and Helen Petrie. 2005. Tenuta: sim-
plied guidance for usability and accessibility. In Proceedings of HCI
International.
[7]
Zhen-Peng Bian, Junhui Hou, Lap-Pui Chau, and Nadia Magnenat-
Thalmann. 2016. Facial Position and Expression-Based Human–
Computer Interface for Persons With Tetraplegia. IEEE journal of
biomedical and health informatics 20, 3 (2016), 915–924.
[8]
Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the
self-assessment manikin and the semantic dierential. Journal of
behavior therapy and experimental psychiatry 25, 1 (1994), 49–59.
Evaluation Methodologies of AT Interaction Devices: A participatory mapping in Portugal IHC ’19, October 22–25, 2019, Vitoria - ES, Brazil
[9]
Aimée K Bright and Lynne Coventry. 2013. Assistive technology for
older adults: psychological and socio-emotional design requirements.
In Proceedings of the 6th International Conference on Pervasive Tech-
nologies Related to Assistive Environments. ACM, 9.
[10] Gilbert Cockton. 2012. Usability evaluation. (2012).
[11]
Albert M Cook and Janice Miller Polgar. 2014. Assistive Technologies-
E-Book:Principles and Practice. Elsevier Health Sciences.
[12]
CAT-Comitê de Ajudas Técnicas. 2007. Ata da Reunião VII, de dezem-
bro de 2007 do Comitê de Ajudas Técnicas. Secretaria Especial dos Di-
reitos Humanos da Presidência da República (CORDE/SEDH/PR) (2007).
[13]
Ovidiu Draghici, Izmail Batkin, Miodrag Bolic, and Ian Chapman. 2013.
The MouthPad: A tongue-computer interface. Medical Measurements
and Applications Proceedings (MeMeA), 2013 IEEE International Sympo-
sium on (2013), 315–319.
[14]
Teólo Alves Galvão Filho. 2013. A construção do conceito de Tecnolo-
gia Assistiva: alguns novos interrogantes e desaos. Revista entreideias:
educação, cultura e sociedade 2, 1 (2013).
[15]
Marc Hassenzahl and Noam Tractinsky. 2006. User experience-a re-
search agenda. Behaviour & information technology 25, 2 (2006), 91–97.
[16]
Baihan Huang, Anthony HP Lo, and Bertram E Shi. 2013. Integrating
EEG information improves performance of gaze based cursor control.
Neural Engineering (NER), 2013 6th International IEEE/EMBS Conference
on (2013), 415–418.
[17]
Xueliang Huo, Hangue Park, Jeonghee Kim, and Maysam Ghovanloo.
2013. A dual-mode human computer interface combining speech and
tongue motion for people with severe disabilities. IEEE Transactions on
Neural Systems and Rehabilitation Engineering 21, 6 (2013), 979–991.
[18]
SFSEN ISO 9241-11. 1998. Ergonomic Requirements for Oce Work
with Visual Display Terminals (VDTs)–Part II Guidance on Usability.
International Organization for Standardization 9241-11 (1998).
[19]
ISO9241-11. 1998. Ergonomic requirements for oce work with vi-
sualdisplay terminals (VDTs): Part 11: Guidance on usability [Internet].
International publisher for Standardization . https://www.iso.org/
home.html
[20]
Marcelo Archajo Jose and Roseli de Deus Lopes. 2015. Human–
computer interface controlled by the lip. IEEE journal of biomedical
and health informatics 19, 1 (2015), 302–308.
[21]
Emmanuel K Kalunga, Sylvain Chevallier, Olivier Rabreau, and Eric
Monacelli. 2014. Hybrid interface: Integrating BCI in multimodal
human-machine interfaces. Advanced Intelligent Mechatronics (AIM),
2014 IEEE/ASME International Conference 322, 10 (2014), 530–535.
[22]
Andrew Kurauchi, Wenxin Feng, Carlos Morimoto, and Margrit Betke.
2015. HMAGIC: head movement and gaze input cascaded pointing.
Proceedings of the 8th ACM International Conference on PErvasive Tech-
nologies Related to Assistive Environments (2015), 47.
[23]
Kwang-Ryeol Lee, Won-Du Chang, Sungkean Kim, and Chang-Hwan
Im. 2017. Real-Time âĂIJEye-WritingâĂİ Recognition Using Elec-
trooculogram. IEEE Transactions on Neural Systems and Rehabilitation
Engineering 25, 1 (2017), 40–51.
[24]
Pierre Lévy. 1999. Cibercultura. 1
ª
edição. São Paulo: Editora 34 (1999).
[25]
Pamela C Levy, Nirvana S Antonio, Thales RB Souza, Rogério Caetano,
and Priscila G Souza. 2013. ActiveIris: uma solução para comunicação
alternativa e autonomia de pessoas com deciência motora severa.
Proceedings of the 12th Brazilian Symposium on Human Factors in Com-
puting Systems (2013), 42–51.
[26]
Cristina Manresa Yee, Joan Jordi Muntaner, and Diana Arellano. 2013.
A motion-based interface to control environmental stimulation for
children with severe to profound disabilities. CHI’13 Extended Abstracts
on Human Factors in Computing Systems (2013), 7–12.
[27]
DTG Mariano, AM Freitas, LMD Luiz, AN Silva, P Pierre, and ELM
Naves. 2014. An accelerometer-based human computer interface driv-
ing an alternative communication system. Biosignals and Biorobotics
Conference (2014): Biosignals and Robotics for Better and Safer Living
(BRC), 5th ISSNIP-IEEE (2014), 1–5.
[28]
Joao MS Martins, Joao MF Rodrigues, and Jaime AC Martins. 2015. Low-
cost Natural Interface Based on Head Movements. Procedia Computer
Science 67 (2015), 312–321.
[29]
César Mauri, Toni Granollers, Jesús Lorés, and Mabel García. 2006.
Computer vision interaction for people with severe movement restric-
tions. Human Technology: An Interdisciplinary Journal on Humans in
ICT Environments (2006).
[30] Jakob Nielsen. 2003. Usability 101: Introduction to usability. https://
www.nngroup.com/articles/usability-101-introduction- to-usability/
[31]
Eliza Oliveira. 2015. O uso de dispositivo multi-sensoriais para promoção
do brincar em crianças com paralisia cerebral. Master’s thesis.
[32]
Diogo Pedrosa and Maria da Graça C. Pimentel. 2014. Text Entry Using
a Foot for Severely Motor-impaired Individuals. Proceedings of the
29th Annual ACM Symposium on Applied Computing (2014), 957–963.
https://doi.org/10.1145/2554850.2554948
[33]
Christian Peter and Bodo Urban. 2012. Emotion in human-computer
interaction. In Expanding the frontiers of visual analytics and visual-
ization. Springer, 239–262.
[34]
Kai Petersen, Robert Feldt, Shahid Mujtaba, and Michael Mattsson.
2008. Systematic Mapping Studies in Software Engineering. EASE 8
(2008), 68–77.
[35]
Srinivasan Ramakrishnan and Ibrahiem MM El Emary. 2013. Speech
emotion recognition approaches in human computer interaction.
Telecommunication Systems 52, 3 (2013), 1467–1478.
[36]
Karen Reijneveld, Michiel De Looze, Frank Krause, and Pieter Desmet.
2003. Measuring the emotions elicited by oce chairs. In Proceedings
of the 2003 international conference on Designing pleasurable products
and interfaces. ACM, 6–10.
[37]
Andreia Sias Rodrigues, Vinicius Kruger Costa, Rafael Cunha Cardoso,
Marcelo Bender Machado, and Tatiana Aires Tavares. 2019. Um Es-
tudo de Mapeamento Sistemático sobre Metodologias de Avaliação em
Interação Humano-Computador voltadas à Tecnologia Assistiva com
foco em Pessoas com Deciência Motora. iSys-Revista Brasileira de
Sistemas de Informação 11, 3 (2019), 90–126.
[38]
Yvonne Rogers, Helen Sharp, and Jenny Preece. 2011. Interaction design:
beyond human-computer interaction. John Wiley & Sons.
[39]
Angela L Sauer, Andra Parks, and Patricia C Heyn. 2010. Assistive
technology eects on the employment outcomes for people with cog-
nitive disabilities: a systematic review. Disability and Rehabilitation:
Assistive Technology 5, 6 (2010), 377–391.
[40]
Sima Soltani and Amin Mahnam. 2013. Design of a novel wearable
human computer interface based on electrooculograghy. Electrical
Engineering (ICEE), 2013 21st Iranian Conference on (2013), 1–5.
[41]
Cihan Topal, Serkan Gunal, Onur Koçdeviren, Atakan Dogan, and
Omer Nezih Gerek. 2014. A low-computational approach on gaze
estimation with eye touch system. IEEE transactions on cybernetics 44,
2 (2014), 228–239.
[42]
MW Van Someren, YF Barnard, and JAC Sandberg. 1994. The think
aloud method: a practical approach to modelling cognitive. Citeseer.
[43]
Rogério Aparecido Campanari Xavier and Vânia Paula de
Almeida Neris. 2014. Measuring the emotional experience of
users through a hybrid semantic approach. In Proceedings of the
13th Brazilian symposium on human factors in computing systems.
Sociedade Brasileira de Computação, 226–235.
