Content uploaded by Johanna J M Geytenbeek
Author content
All content in this area was uploaded by Johanna J M Geytenbeek on Jul 04, 2023
Content may be subject to copyright.
Full Terms & Conditions of access and use can be found at
https://www.tandfonline.com/action/journalInformation?journalCode=iaac20
Augmentative and Alternative Communication
ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/iaac20
Implementation of the C-BiLLT, an accessible
instrument to assess language comprehension in
children with limited motor and speech function:
an international clinician survey
Jael N. Bootsma, Kristine Stadskleiv, Michelle Phoenix, Johanna J. M.
Geytenbeek, Jan Willem Gorter, Dayle McCauley, Sara Fiske, Fiona Campbell,
Natasha Crews & Barbara Jane Cunningham
To cite this article: Jael N. Bootsma, Kristine Stadskleiv, Michelle Phoenix, Johanna J. M.
Geytenbeek, Jan Willem Gorter, Dayle McCauley, Sara Fiske, Fiona Campbell, Natasha
Crews & Barbara Jane Cunningham (2023): Implementation of the C-BiLLT, an accessible
instrument to assess language comprehension in children with limited motor and speech
function: an international clinician survey, Augmentative and Alternative Communication, DOI:
10.1080/07434618.2023.2197060
To link to this article: https://doi.org/10.1080/07434618.2023.2197060
Published online: 13 Jun 2023.
Submit your article to this journal
Article views: 46
View related articles
View Crossmark data
RESEARCH ARTICLE
Implementation of the C-BiLLT, an accessible instrument to assess language
comprehension in children with limited motor and speech function: an
international clinician survey
Jael N. Bootsma
a,b
, Kristine Stadskleiv
c,d
, Michelle Phoenix
a,b
, Johanna J. M. Geytenbeek
e
,
Jan Willem Gorter
e,f
, Dayle McCauley
b
, Sara Fiske
g
, Fiona Campbell
a,h
, Natasha Crews
i
and
Barbara Jane Cunningham
b,i
a
School of Rehabilitation Science, McMaster University, Hamilton, Canada;
b
CanChild Centre for Childhood Disability Research, McMaster
University, Hamilton, Canada;
c
Department of Special Education, University of Oslo, Oslo, Norway;
d
Department of Child Neurology, Oslo
University Hospital, Oslo, Norway;
e
Department of Rehabilitation Medicine, Amsterdam University Medical Centres, Noord-Holland,
Netherlands;
f
Department of Pediatrics, McMaster University, Hamilton, Canada;
g
Pedagogical Psychological Services, Municipality of Oslo,
Oslo, Norway;
h
Technology Access Clinic Developmental Pediatrics and Rehabilitation RJCHC, McMaster Children’s Hospital, Hamilton,
Canada;
i
School of Communication Sciences and Disorders, Western University, London, Canada
ABSTRACT
This study assessed implementation of the Computer-based Instrument for Low-motor Language
Testing (C-BiLLT). The C-BiLLT is an accessible language comprehension assessment tool originally
developed for children with cerebral palsy and complex communication needs. The purpose of the
current study was to understand the clinical contexts in which the C-BiLLT is used in the Netherlands,
Belgium, and Norway and assess barriers and facilitators to implementation. An online survey was dis-
tributed to rehabilitation clinicians working in the Netherlands, Dutch-speaking parts of Belgium, and
Norway. A total of 90 clinicians reported their training in and use of the C-BiLLT; assessed its accept-
ability, appropriateness, and feasibility; and commented on perceived barriers as well as advantages of
the tool. Acceptability, appropriateness, and feasibility were all rated highly. The C-BiLLT was used
with various populations and age groups but most often with children who were younger than
12 years of age, and those with cerebral palsy. The main implementation facilitator was clinicians’
motivation; the main barriers were related to resources and complexity of cases. Findings suggest
implementation of new assessment tools is an ongoing process that should be monitored following
initial training, in order to understand clinical contexts in which the tools are being used.
ARTICLE HISTORY
Received 22 January 2022
Revised 6 September 2022
Accepted 6 September 2022
KEYWORDS
Cerebral palsy; computer-
based instrument for low
motor language testing;
evidence-based assessment;
implementation; speech-
language pathology
An international clinician survey
Cerebral palsy is the most common cause of childhood-onset
physical disability worldwide (Oskoui et al., 2013). Children
with cerebral palsy are at high risk for communication disor-
ders as a result of motor, sensory, language, and cognitive
impairments associated with the disability (Mei et al., 2016;
Pennington et al., 2004; Rosenbaum et al., 2007). Language
impairments affecting comprehension of vocabulary and
grammar are found across the entire spectrum of motor
functioning (Pirila et al., 2007; Stadskleiv et al., 2018; Vaillant
et al., 2020; Voorman et al., 2010; Vos et al., 2014). Motor-
speech disorders are also common and lead to reduced intel-
ligibility or even the absence of functional speech (Andersen
et al., 2010; Hustad et al., 2014; Sigurdardottir & Vik, 2011).
Such communication impairments often have a pervasive
and detrimental impact on children’s social, educational, and
emotional well-being and development (Clarke et al., 2011,
2012; Raghavendra et al., 2012). In many cases where both
motor and speech skills are impacted, children have complex
communication needs and require augmentative and alterna-
tive communication (AAC).
AAC interventions offer children with cerebral palsy who
have complex communication needs access to more expres-
sive language, which, in turn, supports more effective and
efficient communication (Branson & Demchak, 2009, Romski
et al., 2015). Depending on the child, AAC may be as
straightforward as choosing an activity by selecting from one
or two symbols on a tablet, to using eye-gaze to operate a
speech-generating device with thousands of graphic sym-
bols. There is substantial evidence that AAC interventions
tailored to children’s individual needs enable critical commu-
nication skills such as turn-taking, requesting, commenting,
and narrating (Branson & Demchak, 2009; Machalicek et al.,
2010). Furthermore, AAC can expand children’s expressive
vocabularies and enable them to learn to read and write
(Barker et al., 2013; Sennott et al., 2016; Solomon-Rice &
Soto, 2014). In addition to supporting expressive communica-
tion, augmented and often visual input using AAC methods
can support children’s understanding of spoken language
(Drager et al., 2010).
CONTACT Jael Bootsma bootsj1@mcmaster.ca School of Rehabilitation Sciences, McMaster University, 1280 Main St W, Hamilton, ON L8S 4L8, Canada
ß2023 International Society for Augmentative and Alternative Communication
AUGMENTATIVE AND ALTERNATIVE COMMUNICATION
https://doi.org/10.1080/07434618.2023.2197060
In order to prescribe appropriate AAC systems, children’s
language comprehension must be accurately assessed so
that interventions can be tailored to incorporate individual
strengths and address specific communication challenges.
Ideally, assessments should be carried out using a multidis-
ciplinary approach that addresses both current abilities and
future needs (Andersen et al., 2010). Underestimation of lan-
guage comprehension may result in prescribing an AAC sys-
tem that is too limited, which will not only prevent the child
from expressing themselves at the appropriate level but also
may hinder the developmental process. At the same time,
overestimating a child’s abilities may result in prescribing a
device that is too difficult to use. In either case, the child will
be unable to achieve their communicative potential and
opportunities for participation and social engagement will be
restricted.
Assessment with traditional language comprehension
measures is challenging for many children with cerebral
palsy and complex communication needs because these
tools require the child to speak, point to pictures, or manipu-
late small objects (Geytenbeek et al., 2010; Stadskleiv, 2020).
If an assessment cannot be completed, clinicians will often
base AAC decisions on informal observation alone (von
Tetzchner & Stadskleiv, 2016); thus, children with more
severe functional limitations are less likely than those with
typical motor functioning to receive a formal assessment of
their language comprehension (Smits et al., 2011; Stadskleiv,
2020; Stadskleiv et al., 2018) and may receive services that
do not support their full communicative potential.
The Computer-Based Instrument for Low-motor Language
Testing (C-BiLLT; Geytenbeek et al., 2014) has been devel-
oped to ensure language comprehension can be accurately
and reliably assessed in children with cerebral palsy and
complex communication needs. It measures children’s com-
prehension of spoken words and sentences, and children can
indicate their response to the test items using different
modalities, such as a touch screen or eye-tracking, enabling
test participation for those with speech or other fine motor
limitations that would prevent them from engaging in a
traditional language comprehension assessment. The C-BiLLT
utilizes web-based software in combination with several dif-
ferent access methods, both direct and indirect. Current dir-
ect selection methods include touch screen and eye-gaze
computer control; indirect selection methods include the use
of input switches and partner-assisted scanning.
The C-BiLLT is unique in that that it allows even children
with the most complex needs to reliably and autonomously
respond to test items by allowing them to utilize the access
method that best suits their needs. Administration of the C-
BiLLT involves completing three test sections: The first is a
pretest, during which the child is asked to correctly identify
at least five out of eight concrete familiar objects that are
presented in sets of two, and then identify the same objects
presented as photographs. The pretest is done to assess
whether and how children can communicate a choice. Next
is the Computer-based Test Part 1, during which pairs of
items are presented visually on a monitor. The child is asked
to select the picture that matches the vocabulary item
verbally requested by the examiner (e.g., “Where is the …”).
Items are presented in sets of two, in three 10-item sections.
To address the risk of a chance effect, parallel versions of
each 10-item section can be presented if a child answers one
item incorrectly. Finally, there is the Computer-Based Test
Part 2, during which children are presented with 56 items
starting with items at the one-word level and progressing to
complex compound sentences. This section tests comprehen-
sion of vocabulary, morphology, and syntax. Items are pre-
sented on the screen in sets of four. The test is terminated
once the child completes five tasks incorrectly.
The C-BiLLT has been available for use in the Netherlands
since 2015 and is part of the Dutch best practice guidelines
for pediatric rehabilitation for spastic cerebral palsy
(Nederlandse Vereniging van Revalidatieartsen, 2015). A
translated and validated version was introduced in Norway
in 2019 (Fiske et al., 2020) and validation studies are cur-
rently being conducted in Canada, Sweden, Germany,
Romania, and the UK/Ireland. Both the Dutch and Norwegian
versions of C-BiLLT show good construct validity, excellent
internal consistency, and excellent reliability in samples of
typically developing children, and in Dutch children with
cerebral palsy (Fiske et al., 2020; Geytenbeek et al., 2015). At
present, clinicians who want to use the C-BiLLT must partici-
pate in a 1 or 2- day instructional course through the
Amsterdam University Medical Center or the University of
Oslo. The course presents information on the test’s psycho-
metric properties and clinical background, and describes
how to set up the hardware required to administer the C-
BiLLT and how to use the C-BiLLT web module. The course is
run by C-BiLLT investigators. Clinicians who complete the
course are provided with credentials to access the software
via www.c-billt.com. The accompanying hardware required to
administer the C-BiLLT is purchased separately. Initial efforts
to implement the instrument into practice have included dis-
semination in scientific and clinical journals, conference pre-
sentations, and professional trainings, but to date, no formal
evaluation of these implementation efforts has been
conducted.
It is critical to understand how the C-BiLLT has been used
in clinical practice as new interventions, procedures, and
assessment instruments cannot be effective unless they are
implemented well (Proctor et al., 2011; Straus et al., 2013).
Understanding factors that influence and support implemen-
tation of the C-BiLLT depends on understanding current
users and their practice contexts (de Veer et al., 2011),
including their perceived barriers and facilitators to C-
BiLLT use.
The current study was informed by the knowledge-to-
action framework (Graham et al., 2006), which can be used
to conceptualize the process of moving new tools into prac-
tice. It comprises two distinct but related components: (a)
knowledge creation, and (b) the action cycle, which can
interact. The publication and validation of the C-BiLLT com-
pleted the knowledge creation component of the framework,
with training materials being adapted to fit the local
(European) context.
2 J. N. BOOTSMA ET AL.
Recent reviews suggest that a collaborative approach
between researchers and clinicians could increase acceptance
and implementation of new tools like the C-BiLLT (Campbell
& Douglas, 2017; Olswang & Prelock, 2015). Therefore, the
purpose of the current study was to understand current clin-
ical contexts in which the C-BiLLT was used in the
Netherlands, Belgium, and Norway and assess barriers and
facilitators to implementation. Specifically, the aims of this
study were to understand clinicians’(a) professional back-
ground, caseloads, and C-BiLLT training and use; (b) percep-
tions about the C-BiLLT’s appropriateness, acceptability, and
feasibility; and (c) perceived barriers and facilitators to C-
BiLLT use in clinical practice.
Method
Participants
In the Netherlands and Belgium, 177 clinicians who had par-
ticipated in a C-BiLLT training (mainly speech-language path-
ologists) and had provided emails with consent for future
contact were invited via email to participate in an on-line
survey to share their perceptions of and experiences with
the instrument. Invitations were also sent by email to the
directors of 56 speech-language pathology divisions of
Dutch pediatric rehabilitation centers, special education day-
care centers, and centers for individuals with developmental
disabilities, and recipients were asked to circulate the survey
among their clinicians. In Norway, email invitations were sent
to 76 clinicians who had participated in a C-BiLLT training
and provided their email and consent for future contact.
Participation was voluntary and anonymous, and clinicians
were asked to independently review a consent form and
indicate their agreement to participate prior to completing
the survey.
A total of 309 invitations were issued; of these, 32 were
undeliverable. Of the remaining 277 invitations sent, 114 sur-
veys were returned 94 from the Netherlands and Belgium,
and 20 from Norway. Surveys that were less than 50% com-
plete (n¼24, 21%) were discarded, as a comparison of
responses for completed and non-completed surveys yielded
no difference in terms of sex (Fisher’s exact test, p¼1) or
years of work experience (Kruskal–Wallis H test, v
2
(3) ¼
0.064, p¼.8). The final sample included 90 clinicians (79
from The Netherlands, 11 from Norway). The exact response
rate cannot be reported because it was not known how
many individual clinicians at the speech-language pathology
divisions of the rehabilitation sites received or completed the
survey. Respondents from the Netherlands and Belgium all
received the survey in Dutch. We did not ask respondents
where they were from, so we there was no way of differenti-
ating between respondents from the Netherlands and
Belgium. Because most invitations were sent to clinicians in
the Netherlands, responses from both of those countries
were grouped and reported as “Netherlands”(Tables 1–4).
Most survey respondents were female (n¼87, 97%); the
male respondents (n¼3, 3%) came from Norway; 38
respondents indicated a different practice setting than the
ones listed on the survey.
See Table 1 for complete demographic details.
Research design
An exploratory survey study was used to collect multiple
data points at the same time from a relatively large sample
of clinicians. Approval for this study was granted by Western
University’s Health Science Research Ethics Board (Project
number: 116859).
Materials and measures
Survey
The online survey (Supplemental material) was developed
specifically for this study to assess clinicians’use of C-BiLLT.
Survey items were developed within the knowledge-to-action
framework (Graham et al., 2006) in order to understand the
local contexts in which the C-BiLLT was being used (i.e., pro-
vide a snapshot of current practices) and assess barriers and
facilitators to its use in practice. The survey was designed in
English and then translated into Dutch and Norwegian. An
initial draft was pilot-tested with clinical colleagues who had
C-BiLLT experience. Following their review, minor adjust-
ments to the wording of questions were made.
The survey contained six question pertaining to respond-
ent demographics, C-BiLLT training, and C-BiLLT use. Those
who had not used the instrument were asked to explain why
and what would need to change in order for them to do so.
Table 1. Professional background of respondents who used the C-BiLLT in
their practice.
Characteristics
Netherlands Norway
nn
Year of training
2014 6
2015 11
2016 4
2017 10
2018 11
2019 15 6
2020 9 5
Do not remember 8 0
Did not take training 5 0
Profession
SLP 78 1
Psychologist 0 4
Educator 0 5
Other 1 1
Practice setting
Pediatric rehabilitation center 33 3
Hospital 13 7
School 16 1
Private practice 6 1
Other: Facility for individuals with IDD 20 0
Other: Early intervention 3 0
Other: National competence center
a
13
Other: Not defined 0 0
Years of work experience
<10 13 4
>10 66 7
Note. SLP: speech-language pathologist; IDD: intellectual and developmental
disability. Some numbers sum to greater than 90, as clinicians may have
chosen more than one answer.
a
National competence centers are Norwegian
centers through which care is provided for certain conditions.
AUGMENTATIVE AND ALTERNATIVE COMMUNICATION 3
Those who had used C-BiLLT answered closed-ended ques-
tions about their experiences, such as how often they used
the instrument, types of access method(s), how long assess-
ments typically took, and whether they were ever unable to
complete testing. They were also asked open-ended ques-
tions about how easy or difficult the C-BiLLT was to use and
what they would like to see changed about the instrument.
Measures
Three tools were used to measure acceptability, appropriate-
ness, and feasibility of the C-BiLLT: (a) the Acceptability of
Intervention Measure, (b) the Intervention Appropriateness
Measure, and (c) the Feasibility of Intervention Measure
(Weiner et al., 2017). These measures were selected because
they were commonly used in research as indicators of imple-
mentation success (Weiner et al., 2017). Each measure has
four items, is quick and easy to administer, and freely avail-
able. In the present study, the Norwegian and Dutch adapta-
tions of the three measures were used (Engell et al., 2018;
Simon et al., n.d.). Respondents were asked to rate all items
for each measure using a 5-point Likert scale that ranged
from 1 (completely disagree)to5(completely agree).
Acceptability was defined as the perception that the C-
BiLLT was seen as agreeable, palatable, or satisfactory
(Proctor et al., 2011; Weiner et al., 2017). Appropriateness was
defined as the C-BiLLT’s perceived fit, relevance, or compati-
bility for a given practice setting, provider, or consumer;
and/or its perceived fit to address a particular problem
(Proctor et al., 2011; Weiner et al., 2017). Lastly, feasibility was
defined as the extent to which the C-BiLLT could be easily
used within a given clinical context (Proctor et al., 2011;
Weiner et al., 2017). Although these implementation out-
comes are closely related, there are some important distinc-
tions (Proctor et al., 2011). For instance, a technique or
process deemed appropriate for addressing a specific prob-
lem may not be acceptable to the clinician who is using it,
or even feasible due to training or resource requirements.
Procedures
Data collection and analysis
The survey was open from February 15 to March 15, 2021,
with responses collected and managed using REDCap, an
electronic data capture tool hosted at the tenth author’s uni-
versity (Harris et al., 2019). Survey responses were analyzed
Table 2. Characteristics of clinicians’caseloads (n¼90).
Characteristics
Netherlands Norway
nn
Number of clients tested
<3 clients 4 6
3–10 clients 22 4
10–20 clients 25 1
>20 clients 28 0
Age
a
(n¼89)
<5 years 56 8
6–12 years 58 8
13–17 years 26 2
>18 years 25 2
Diagnosis
a
Cerebral palsy 74 6
Down syndrome 31 1
Rett syndrome 27 0
Angelmann syndrome 19 0
Language disorder 43 5
Syndrome other than listed 22 0
IDD 17 0
ASD 16 3
Other (e.g., ABI, metabolic, muscle
or mitochondrial diseases)
13 4
Unknown 31 4
GMFCS
a,b
I285
II 29 2
III 39 2
IV 57 3
V644
I don’t know 2 2
CFCS
a,b
I91
II 21 2
III 48 2
IV 70 3
V503
I don’t know 2 2
Note. ASD: autism spectrum disorder; ABI: acquired brain injury; IDD: intellec-
tual or developmental disability; GMFCS: Gross Motor Function Classification
System (Palisano et al., 1997b); CFCS: Communication Function Classification
System (Hidecker et al., 2011).
a
Numbers may sum to greater than 90, as clinicians had the option to choose
more than one answer;
b
While these classification systems are validated for use
in the population of people with cerebral palsy, we did not specify that respond-
ents should limit their responses only to their clients with cerebral palsy.
Table 3. C-BiLLT users’assessment practices.
Netherlands Norway
nn
Frequency of use
Yearly or less 7 2
Half-yearly 15 3
Once every 3 months 25 5
Monthly 27 1
Weekly 5 0
Access method(s) used
Direct selection with pointing
Touchscreen monitor 64 3
Tablet 29 7
Pointing without touching (finger or other) 53 3
Direct selection with eye gaze
Eye gaze (no tech) 25 1
Eye tracker 31 2
Indirect selection
Input switches 19 1
Head mouse 2 0
Partner assisted scanning 6 2
Child’s own access device 24 0
Multiple access methods in one session
Never 27 7
Sometimes 47 4
Usually 5 0
Unable to complete on at least one occasion
Yes 55 4
No 22 7
Reasons/causes for non-completion (n¼59)
a
Issues re: access methods 8 1
Vision 13 2
Hearing 0 0
Attention 43 3
Behavior/compliance 12 2
Task orientation 45 3
Doubts re: reliability 30 1
Technical issues 16 2
Other 4 0
Note. Numbers may sum to greater than 90 because clinicians had the
option to choose more than one answer.
4 J. N. BOOTSMA ET AL.
descriptively, including reporting of counts and percentages.
Demographic data were analyzed separately for the respond-
ents from the Netherlands and Dutch-speaking parts of
Belgium and Norway in order to better understand the back-
grounds of respondents from the different countries.
Caseload characteristics and user experiences were analyzed
for all regions combined. Results for the Acceptability of
Intervention Measure, Intervention Appropriateness Measure,
and Feasibility of Intervention Measure were reported using
means and standard deviations for Dutch and Norwegian
respondents separately. Free-text survey responses were
translated from Dutch and Norwegian into English by the
first and seventh authors, who were fluent in both lan-
guages. Qualitative data were managed and analyzed using
Dedoose, a web-based application for qualitative and mixed
methods research (Dedoose Version 8.3.45, 2018). Responses
to open-ended survey items about barriers and facilitators to
C-BiLLT use were analyzed using thematic analysis (Braun &
Clarke, 2006), which involved the following steps: (a) familiar-
ization with the data, (b) generation of initial codes, (c)
development of themes, (d) review of themes and gener-
ation of a thematic map of the analysis, (e) generation of
names and definitions of identified themes, and (f) producing
a report of the analysis.
Steps 1–3 were carried out by the first author; Steps 4–6
were completed collaboratively with the entire research
team. To ensure reliable coding, a trained graduate research
assistant reviewed 10% of the free-text responses during
Steps 2 and 3. Cohen’sjwas calculated to determine the
agreement between two raters on these 52 excerpts, which
was almost perfect: j¼0.92 (95% CI, 0.83 to 1), p<.0005
(McHugh, 2012).
Results
The 24 incomplete surveys included responses from eight
clinicians who said they had never used the C-BiLLT in their
practice. Of these respondents, six (four from Norway and
two from the Netherlands) indicated that they had partici-
pated in training but had not used the C-BiLLT for the fol-
lowing reasons: parental leave (n¼1); the COVID 19
pandemic (n¼3); no eligible clients (n¼1), lack of hardware
(n¼1) or no reason stated (n¼2). Three of these clinicians
answered the question about what would need to change in
order for them to use the C-BiLLT. One indicated acquiring
the right hardware and one cited COVID-19 restrictions (e.g.,
removing the forced stoppage of in-person appointments).
The two others had registered for an upcoming training and
were not yet using the C-BiLLT.
Understand clinicians’caseloads and C-BiLLT use
C-BiLLT user caseloads
Respondents worked in a variety of practice settings and
most had over 10 years of work experience. They reported
using the C-BiLLT most often with children younger than
12 years of age, and the majority (58%) had used it with
between three and 20 children at the time of survey comple-
tion. Most children with whom the C-BiLLT was being used
had a diagnosis of cerebral palsy; however, the C-BiLLT was
also being used with children who had other neurodevelop-
mental disabilities (e.g., Down syndrome, Rett syndrome, aut-
ism spectrum disorder, language disorder, intellectual
disability). The majority of children with whom the test was
being used had significant motor and communication limita-
tions as classified by the Gross Motor Function Classification
System (GMFCS; Palisano et al., 1997) and Communication
Function Classification System (CFCS; Hidecker et al., 2011).
However, the test was also being used with clients who
spanned the entire spectrum of motor and communicative
functioning (Table 2), including children without physical lim-
itations who could have participated in traditional language
testing (e.g., Language disorder, intellectual disability).
C-BiLLT users’assessment practices
Most clinicians indicated that they used the C-BiLLT at least
once per quarter and employed a variety of access methods
with clients (Table 3). Duration and number of sessions
needed to complete an assessment using C-BiLLT varied
widely. Clinicians (n¼84) indicated a mean minimum C-
BiLLT administration time of 24 min (range: 1–90 min), and a
mean maximum time of 56 min (range: 15–240 min).
Table 4. Users’ratings of the C-BiLLT’s acceptability, appropriateness, and feasibility.
Baseline characteristic
Mean (SD)
Netherlands Norway
Acceptability of intervention measure (Weiner et al., 2017) 4.35 (0.62) 4.39 (0.24)
C-BiLLT meets my approval 4.34 (0.55) 4.36 (0.67)
C-BiLLT is appealing to me 4.35 (0.66) 4.27 (0.65)
I like C-BiLLT 4.34 (0.66) 4.18 (0.75)
I welcome C-BiLLT 4.37 (0.61) 4.74 (0.48)
Intervention appropriateness measure (Weiner et al., 2017) 4.08 (0.71) 4.16 (0.09)
C-BiLLT seems fitting for my clients. 4.11 (0.72) 4.09 (0.70)
C-BiLLT seems suitable for my clients 4.08 (0.73) 4.18 (0.60)
C-BiLLT seems applicable for my clients 4.06 (070) 4.27 (0.65)
C-BiLLT seems like a good match with my clients 4.08 (0.69) 4.09 (0.70)
Feasibility of intervention measure (Weiner et al., 2017) 4.07 (0.64) 4.43 (0.16)
C-BiLLT seems implementable 4.06 (0.67) 4.27 (0.65)
C-BiLLT seems possible 4.12 (0.53) 4.36 (0.50)
C-BiLLT seems doable 4.11 (0.62) 4.27 (0.65)
C-BiLLT seems easy to use 3.92 (0.76) 4.00 (0.77)
Note. Scale from Weiner et al. (2017)1¼lowest, 5 ¼highest; SD: standard deviation.
AUGMENTATIVE AND ALTERNATIVE COMMUNICATION 5
Respondents (n¼42) indicated needing between one and
five sessions to complete an assessment, with 20% (n¼8)
stating that they always required at least two sessions; how-
ever, they did not feel that the time needed to complete an
assessment was a barrier to using the test.
Over 60% of clinicians (n¼56) indicated they had used
multiple access methods within one test session, with a pref-
erence for direct selection methods. Sixty-six percent (n¼59)
indicated they had on occasion been unable to complete a
C-BiLLT assessment, with most citing child’s lack of attention
(n¼46) or difficulty orienting to task (n¼48) as the main
reasons. Others said they had doubts about the reliability of
the child’s responses (n¼31) or had difficulty finding a good
access method (n¼9).
Assessing acceptability, appropriateness, and feasibility
of the C-BiLLT
Clinicians rated all three implementation outcomes highly.
Acceptability was rated the highest (M¼4.35, SD ¼0.62 and
M¼4.39, SD ¼0.24) among respondents from the
Netherlands and Norway, respectively. Feasibility and appro-
priateness also received mean ratings above 4 in both prac-
tice contexts (Table 4).
Identifying barriers and facilitators to using the C-BiLLT
A range of barriers and facilitators to C-BiLLT use were iden-
tified. Barriers were defined as factors that would limit use of
the C-BiLLT and facilitators were defined as factors that
would help clinicians to use the C-BiLLT. The reported bar-
riers and facilitators were grouped into four broad themes:
(a) Factors Inherent to the C-BiLLT (e.g., hardware, software,
and test characteristics), (b) Factors Related to Children the
C-BiLLT is used with, (c) Factors Related to Clinician using
the C-BiLLT, and (d) Factors Related to Environment in which
the C-BiLLT is used. These themes are illustrated in Figure 1,
which depicts a C-BiLLT being accessed via eye-tracking. The
screen is tilted and on a rolling mount so that it is parallel to
the child being tested. The identified barriers and facilitators
associated with each theme are presented next.
Factors inherent to the C-BiLLT
Barriers inherent to the C-BiLLT included those associated
with its hardware, software, and images. Hardware barriers
included the time needed to set up the testing environment,
including connecting different devices. One respondent
noted, for example, how they had to contend with the
“fussiness”of connecting many cables. In some cases, clini-
cians reported that the barriers related to hardware were too
significant to overcome, especially in relation to using the
eye-tracking module (e.g., “I still can’t link the computers
with the eye-tracking system”). Challenges related to soft-
ware included format and navigation. One respondent noted
that both processes were “Very rigid and not user friendly,”
while another stated that “Navigating within the test is
difficult.”Respondents also commented that optimal per-
formance depended on strong and stable internet connec-
tions across multiple sites, which were not always available;
and that some of the photos used for the test were outdated
and likely not recognizable to children (e.g., “The pictures are
no longer up to date. Like the DVD, television, mailbox –
hardly anyone goes there anymore”).
Respondents described the following facilitators inherent
to the C-BiLLT: automatic scoring ability, which allowed them
to concentrate more on the assessment activity; the ease
with which the test could be administered; and lack of dis-
tracting test materials such as toys that could also be lost or
broken.
Factors related to the child
Barriers related to children with whom the C-BiLLT was used
were primarily associated with the complexity of the case
and level of visual impairment. Clinicians reported it could
be challenging to find the right access method, seating and
positioning, and table to properly administer the C-BiLLT
with some children with complex needs. Ensuring the chil-
dren with complex needs remained alert and motivated dur-
ing testing was also a challenge. Clinicians found the C-BiLLT
to be visually difficult for children with visual impairments,
even when the impairment was mild. Many clinicians had
suggestions for improving the test for children with visual
impairments, including creating better contrast between tar-
get images and the background screen, providing the ability
to choose different backgrounds, and having the option to
rearrange the way pictures are presented on the screen.
Child-related facilitators included increased interest in and
motivation to work with the computer screen and the lack
of requirement to use booklets and small objects such as
those required for traditional testing (e.g., Peabody Picture
Vocabulary Test, Preschool Language Scales), Both factors
were reported to improve concentration for all children who
were assessed using the C-BiLLT, including those who had
the speech and fine motor skills to participate in traditional
testing.
Figure 1. Barriers and facilitators to C-BiLLT use. Note. Barriers and facilitators
in the image include (a) factors inherent to the C-BiLLT (e.g., hardware, soft-
ware, and test characteristics), (b) child-related factors, (c) clinician-related fac-
tors, and (d) factors related to the environment.
6 J. N. BOOTSMA ET AL.
Factors related to the clinician
This theme relates to clinician skills, perceptions, resources,
goals, and context. Barriers included a perceived lack of
technological skills, lack of confidence in C-BiLLT results, and
insecurities regarding interpreting behaviors exhibited by cli-
ents during the assessment. Said one respondent: “The
patients that are relevant for me to test with the C-BiLLT
usually have extensive difficulties and it is often difficult to
decide whether test results are valid because of attention,
behavior, and communication challenges.”
Clinician-related facilitators included the perception that
the C-BiLLT meets a clinical need, which increased motiv-
ation for its use. For example, respondents commented on
how C-BiLLT enabled them to generate important language
comprehension assessment data for children with severe
motor problems and to assess children who were otherwise
untestable.
Factors related to the environment
Barriers included physical work space, organizational systems,
and social factors. For example, one respondent reported
that IT policies prevented them from connecting C-BiLLT
hardware in optimal ways, saying “We need to use stand-
alone computers if we want to connect devices to it, which
is often not possible in our workspace.”Lack of space to set
up the C-BiLLT was another barrier. Said one respondent:
“We have a set up away from practice space because of the
size of the screen, this is impractical.”Still others commented
on how lack of familiarity with the C-BiLLT either outside of
or within organizations prevented C-BiLLT results from being
used for official documentation (e.g., “lack of familiarity with
the C-BiLLT in some agencies, causes the C-BiLLT results not
to be recognized for admissions, for instance for education”).
Environmental facilitators included availability of adequate
space to leave the assessment equipment set up and ready
to use, and having a supportive team. A supportive team
was described as having the backing from colleagues, man-
agement, and workplace IT staff to use the C-BiLLT as well as
working with a dedicated team of people who administered
the C-BiLLT.
Discussion
This study investigated implementation of the C-BiLLT in the
Netherlands, Dutch-speaking parts of Belgium, and Norway.
In all, 90 clinicians shared details about their professional
background, caseload, C-BiLLT training and use, assessment
practices, and perceptions of the instrument. Despite identi-
fying barriers to its use, the respondents rated the C-BiLLT
highly in terms of appropriateness, acceptability, and feasibil-
ity. The present study adds to the nascent body of literature
examining the implementation of assessment tools in clinical
care, which is timely, given the increased availability but lack
of use of new assessment tools in pediatric rehabilitation
(O’Connor et al., 2016). Reasons for why uptake is slow are
largely unknown, because studies about the implementation
of evidence-based assessment tools are scarce (Colquhoun
et al., 2017).
The first aim of the study was to understand clinicians’
contexts, caseloads, and how they used the C-BiLLT. Many
clinicians reported using the test with children with cerebral
palsy, but also with children who had other diagnoses and
those who had the motor skills to participate in more trad-
itional language testing (i.e., the ability to point, speak, and
manipulate small objects). This finding suggests the C-BiLLT
was viewed favorably for its intended population, but for
other populations as well. The use of tests for unintended
purposes or populations is not uncommon in pediatric
rehabilitation (Hanna et al., 2007) and may suggest an unmet
need in the field. More research regarding use of the C-BiLLT
with other populations or in adapted ways is needed to test
validity in these situations (de Vet et al., 2011).
The second aim was to understand clinicians’perceptions
about the C-BiLLT’s appropriateness, acceptability, and feasi-
bility. Average ratings for each of these constructs were
above 4 on the previously noted 5-point scales, suggesting
positive perceptions among those who had used the instru-
ment in practice. This result was consistent across clinicians
from different countries, professions, and for all levels of
experience with the C-BiLLT. In interpreting this result, two
considerations are important. First, it is possible that high
ratings were due to perceptions of the clinical potential of C-
BiLLT results (e.g., assessment information it can yield) rather
than ease of implementation in everyday clinical contexts
and client satisfaction. Second, it should be noted that the
measures Acceptability of Intervention, Intervention
Appropriateness, and Feasibility of Intervention were devel-
oped to specifically explore implementation outcomes and
not service or client outcomes (Weiner et al., 2017).
The study’s final aim was to describe clinicians’perceived
barriers and facilitators to C-BiLLT use. Respondents
described barriers such as the high cost of the hardware
required to administer the test for some children, a lack of
practice using the instrument, and challenges completing
the test with children who had visual deficits. They noted
that while the C-BiLLT had the potential to enable assess-
ment of spoken language comprehension in children with
motor and speech impairments, the identified barriers may
hamper its implementation in some clinical contexts.
Clinicians highlighted that effective use of the C-BiLLT
requires advanced clinical skills and experience working with
children who have complex communication needs. In cases
where assessments could not be completed, clinicians ques-
tioned the validity of the child’s responses, particularly when
the child’s vision was impacted. The visual aspects of the
test are especially important to consider for children with
cerebral palsy because a significant percentage (65%) show
visual defects indicative of cerebral visual impairment
(Schenk-Rootlieb et al., 1994) and lower visual field impair-
ments are common (Philip & Dutton, 2014). Participants
offered many suggestions for improving the C-BiLLT’s suit-
ability for children with visual impairments regardless of
whether they also had cerebral palsy, such as the option to
present pictures against a black background; removing the
AUGMENTATIVE AND ALTERNATIVE COMMUNICATION 7
relatively large black buttons on the screen to make the first
30 items more eye-gaze friendly; and, during the second half
of the test, presenting the four images per screen horizon-
tally at the top of the screen rather in block format.
Participants also indicated a need to update some of the
images used in the C-BiLLT –both in terms of the content
and general appearance (some images have been replaced
in revised versions of the Norwegian and Canadian-English
instruments).
In terms of access methods used to complete the C-BiLLT,
the least used was computerized eye-tracking. This may in
part be because the C-BiLLT was often used with children
who could complete the test in other ways, but may also be
due to implementation barriers. In the current study, when
participants did use the eye tracking access method, they
identified challenges including difficulty interpreting a cli-
ent’s response based on their eye gaze, which led to doubts
about test reliability. Computerized eye-tracking was also
reported to be difficult for some children, which may have
been because it uses the same channel (eye gaze) for both
observation and control. The use of eye gaze for dual pur-
poses could make it difficult to differentiate when a child
selects a response deliberately from when they are simply
scanning the possible response options (Jacob, 1991). The lit-
erature on eye tracking also reports user fatigue (Bates et al.,
2005; Higginbotham et al., 2007), which may explain why
multiple access methods were often used to complete the C-
BiLLT. In general, the popularity of computerized eye track-
ing as an access method for AAC is increasing (Bates et al.,
2005), and research to better support this response method
is required.
An additional barrier associated with eye-tracking devices
was the high cost. To circumvent this, some participants who
liked using eye gaze as an access method reported using
low-tech eye gaze (i.e., their own interpretation of where the
child looked on the screen) as an access method. If compu-
terized eye trackers were more affordable, clinicians would
likely use this more reliable access method. This finding cor-
responds with previous literature that reports high costs as a
common barrier to the adoption of innovations in rehabilita-
tion care (Duncan & Murray, 2012).
Implications for practice
At the clinical level, effective use of the C-BiLLT will depend
on a variety of factors. These include (a) enhancing C-BiLLT
training by incorporating strategies for observing children
during testing and devising ways of identifying signs of lack
of attention or fatigue; (b) ensuring additional training mate-
rials are available to address changes and updates to the
technology; and (c) providing more advanced access meth-
ods for children with complex communication needs and
ensuring adequate levels of and access to training in these
methods. Furthermore, there is an ongoing need for more
advanced access methods, such as electroencephalography
and infrared sensing (Higginbotham et al., 2007; Tai et al.,
2008), which may help ensure children with complex com-
munication needs are able to express needs, wants, feelings,
and thoughts. Use of these and other technologies will
depend on adequate levels of and appropriate access to clin-
ician training. Finally, it was surprising that not many clini-
cians mentioned family involvement in the assessment
process given the emphasis on family-centred practice in
pediatric rehabilitation. For children who communicate non-
verbally, parents’or caregivers’knowledge about the child
and how they express themselves could help the clinician
interpret behaviors during the assessment (Lancioni et al.,
2007; Noyek et al., 2020). The C-BiLLT instructional course
could be enhanced with a section on family-centred assess-
ment practices to support clinicians.
There is a continuing need for more advanced access
methods for children with complex communication needs,
who are restricted in expressing their needs, wants, feelings,
and thoughts. There have been developments in access tech-
nology, including electroencephalography and infrared sens-
ing (Tai et al., 2008; Higginbotham et al., 2007), which may
be useful in practice in the future. Findings from the current
study suggest it will be critical to support clinicians in imple-
menting such technologies in practice, and efforts to
increase the uptake of evidence-based instruments should
directly target identified barriers such as lack of knowledge
or technical expertise to support clinicians in using the
access method that best meets their clients’needs.
Limitations and future directions
The current study has some limitations that must be consid-
ered when interpreting the results. Although the response
rate was high, because participation was voluntary and
anonymous, it is not possible to state that the results are
representative of all clinicians who have been trained to use
the C-BiLLT. Selection bias was also possible, and clinicians
who either really liked or disliked the C-BiLLT may have been
more inclined to participate. Survey respondents were highly
trained, specialized, and motivated professionals, and for
some, C-BiLLT use was encouraged, for instance through the
Dutch best practice guidelines for pediatric rehabilitation
professionals. These characteristics may have led to a specific
type of participant, which may not be representative of all
potential C-BiLLT users. It is also possible that the experien-
ces of clinicians who could not overcome implementation
barriers were not captured. Additionally, the survey lacked
follow-up questions, which could have been informative. For
example, it is not known whether clinicians who indicated
there were occasions when they could not complete the C-
BiLLT were able to finish the assessment in a follow-up
session.
Responses to this survey indicated clinicians’willingness
and professional enthusiasm for participating in research.
Future research will capitalize on this resource by actively
collaborating with clinicians. By including clinicians’experien-
ces and expertise, implementation efforts are more likely to
be relevant and successful (Kothari & Wathen, 2017), and
researchers may avoid the traditional education approaches
that have been shown to have a limited impact on practice
(Paley, 2007). Further research exploring clinicians’practices
8 J. N. BOOTSMA ET AL.
in different contexts and the impact of specific implementa-
tion interventions for the C-BiLLT may be particularly useful.
To bypass the identified challenges using the C-BiLLT with
children who have visual impairments, the use of a brain-
computer interface based on evoked potentials, such as the
brain wave P300 (Orlandi et al., 2021) could be helpful.
Brain-computer interfaces, where brain responses to stimuli
are picked up and interpreted, typically rely on visual stimuli;
however, research in adults shows good results with auditory
evoked potentials as well (Orlandi et al., 2021). Evidence is
emerging for the feasibility of such paradigms in children
who have vision impairments but intact hearing (Zhang
et al., 2019); and this may have potential as a future access
method for the C-BiLLT. However, parallel to the need for
even more advanced technology, is the need to address the
barrier of the high cost of the hardware necessary to admin-
ister the test. Future research could also investigate the reli-
ability of the low-tech eye access methods for the C-BiLLT to
support accessibility for both clinicians and children.
While not all suggestions may be readily adoptable, the
common barriers faced by clinicians will need to be consid-
ered in future implementation interventions. For example, C-
BiLLT trainings may be improved by incorporating strategies
for observing children during testing and providing tips for
identifying signs of lack of attention or fatigue. Updating C-
BiLLT content will be an ongoing process, and additional
training materials on the use of technology will be
considered.
Conclusion
Study participants rated the C-BiLLT as an acceptable, appro-
priate, and feasible instrument for children with cerebral
palsy, and clinicians were highly motivated to use it in their
practice. The identification of technological, clinical, and
resource-related barriers calls for a more detailed exploration
of how C-BiLLT users can be better supported.
Acknowledgements
The authors would like to thank all participating clinicians for sharing
their experiences and expertise, Johanna Geytenbeek for sharing her
database of names, Samantha Yimeng Dong for the illustration, and
Audr
ee Leclerc for her help with reliability coding.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Funding
This study was supported by funds awarded by CanChild Center for
Childhood Disability Research.
ORCID
Jael N. Bootsma http://orcid.org/0000-0002-7862-0109
Kristine Stadskleiv http://orcid.org/0000-0002-5478-5689
Michelle Phoenix http://orcid.org/0000-0002-6190-3997
Johanna J. M. Geytenbeek http://orcid.org/0000-0001-7821-7232
Jan Willem Gorter http://orcid.org/0000-0002-3012-2119
Dayle McCauley http://orcid.org/0000-0003-3863-7229
Natasha Crews http://orcid.org/0000-0003-0628-9311
Barbara Jane Cunningham http://orcid.org/0000-0002-2032-4999
References
Andersen, G. L., Mjøen, T., & Vik, T. (2010). Prevalence of speech prob-
lems and the use of augmentative and alternative communication in
children with cerebral palsy: A registry-based study in Norway.
Perspectives on Augmentative and Alternative Communication,19(1),
1–28. doi:10.1044/aac19.1.12
Barker, R. M., Akaba, S., Brady, N. C., & Thiemann-Bourque, K. (2013).
Support for AAC use in preschool, and growth in language skills, for
young children with developmental disabilities. Augmentative and
Alternative Communication,29(4), 334–346. doi:10.3109/07434618.
2013.848933
Bates, R., Istance, J., Oosthuizen, L., Majaranta, P. (2005). Survey of de-
facto standards in eye tracking. Retrieved June 6, 2022, from http://
wiki.cogain.org/images/1/1b/COGAIN-D2.1.pdf
Branson, D., & Demchak, M. (2009). The use of augmentative and alterna-
tive communication methods with infants and toddlers with disabil-
ities: A research review. Augmentative and Alternative Communication,
25(4), 274–286. doi:10.3109/07434610903384529
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology.
Qualitative Research in Psychology,3(2), 77–101. doi:10.1191/
1478088706qp063oa
Campbell, W. N., & Douglas, N. F. (2017). Supporting evidence-based
practice in speech-language pathology: A review of implementation
strategies for promoting health professional behavior change.
Evidence-Based Communication Assessment and Intervention,11(3–4),
72–81. doi:10.1080/17489539.2017.1370215
Clarke, M., Newton, C., Griffiths, T., Price, K., Lysley, A., & Petrides, K. V.
(2011). Factors associated with the participation of children with com-
plex communication needs. Research in Developmental Disabilities,
32(2), 774–780. doi:10.1016/J.RIDD.2010.11.002
Clarke, M., Newton, C., Petrides, K., Griffiths, T., Lysley, A., & Price, K.
(2012). An examination of relations between participation, communi-
cation and age in children with complex communication needs.
Augmentative and Alternative Communication,28(1), 44–51. doi:10.
3109/07434618.2011.653605
Colquhoun, H. L., Lamontagne, M. E., Duncan, E. A. S., Fiander, M.,
Champagne, C., & Grimshaw, J. M. (2017). A systematic review of
interventions to increase the use of standardized outcome measures
by rehabilitation professionals. Clinical Rehabilitation,31(3), 299–309.
doi:10.1177/0269215516644309
de Veer, A. J. E., Fleuren, M. A. H., Bekkema, N., & Francke, A. L. (2011).
Successful implementation of new technologies in nursing care: A
questionnaire survey of nurse-users. BMC Medical Informatics and
Decision Making,11(1), 67. doi:10.1186/1472-6947-11-67
de Vet, H. C., Terwee, C. B., Mokkink, L. B., & Knol, D. L. (2011).
Measurement in medicine: A practical guide. Cambridge University
Press.
Drager, K., Light, J., & McNaughton, D. (2010). Effects of AAC interven-
tions on communication and language in young children with com-
plex communication needs. Journal of Pediatric Rehabilitation
Medicine,3(4), 303–310. doi:10.3233/PRM-2010-0141
Duncan, E. A., & Murray, J. (2012). The barriers and facilitators to routine
outcome measurement by allied health professionals in practice: A
systematic review. BMC Health Services Research,12,96–105. doi:10.
1186/1472-6963-12-96
Engell, T., Follestad, I. B., Andersen, A., & Hagen, K. A. (2018). Knowledge
translation in child welfare –improving educational outcomes for
children at risk: Study protocol for a hybrid randomized controlled
pragmatic trial. Trials,19(1), 4. doi:10.1186/s13063-018-3079-4
Fiske, S. I., Haddeland, A. L., Skipar, I., Bootsma, J. N., Geytenbeek, J. J., &
Stadskleiv, K. (2020). Assessing language comprehension in motor
impaired children needing AAC: Validity and reliability of the
AUGMENTATIVE AND ALTERNATIVE COMMUNICATION 9
Norwegian version of the receptive language test C-BiLLT.
Augmentative and Alternative Communication,36(2), 95–106. doi:10.
1080/07434618.2020.1786857
Geytenbeek, J. J. M., Harlaar, L., Stam, M., Ket, H., Becher, J., Oostrom, K.,
& Vermeulen, R. J. (2010). Utility of language comprehension tests for
unintelligible or non-speaking children with cerebral palsy: A system-
atic review. Developmental Medicine and Child Neurology,52(12),
e267–e277. doi:10.1111/j.1469-8749.2010.03807.x
Geytenbeek, J. J. M., Mokkink, L. B., Knol, D. L., Vermeulen, R. J., &
Oostrom, K. J. (2014). Reliability and validity of the C-BiLLT: A new
instrument to assess comprehension of spoken language in young
children with cerebral palsy and complex communication needs.
Augmentative and Alternative Communication,30(3), 252–266. doi:10.
3109/07434618.2014.924992
Geytenbeek, J. J. M., Vermeulen, R. J., Becher, J. G., & Oostrom, K. J.
(2015). Comprehension of spoken language in non-speaking children
with severe cerebral palsy: An explorative study on associations with
motor type and disabilities. Developmental Medicine and Child
Neurology,57(3), 294–300. doi:10.1111/dmcn.12619
Graham, I., Logan, J., Harrison, M., Strauss, S., Tetroe, J., Caswell, W., &
Robinson, N. (2006). Lost in knowledge translation: Time for a map?
The Journal of Continuing Education in the Health Professions,26(1),
47. doi:10.1002/chp.47
Hanna, S. E., Russell, D. J., Bartlett, D. J., Kertoy, M. L., Rosenbaum, P. L.,
& Wynn, K. (2007). Measurement practices in pediatric rehabilitation:
A survey of physical therapists, occupational therapists, and speech-
language pathologists in Ontario. Physical & Occupational Therapy in
Pedatrics,27(2), 25–42. doi:10.1300/J006v27n02_03
Harris, P. A., Taylor, R., Minor, B. L., Elliott, V., Fernandez, M., O'Neal, L.,
McLeod, L., Delacqua, G., Delacqua, F., Kirby, J., & Duda, S. N. (2019).
The REDCap consortium: Building an international community of soft-
ware platform partners. Journal of Biomedical Informatics ,95, 103208.
doi:10.1016/j.jbi.2019.103208
Hidecker, M. J. C., Paneth, N., Rosenbaum, P. L., Kent, R. D., Lillie, J.,
Eulenberg, J. B., Chester, K., Johnson, B., Michalsen, L., Evatt, M., &
Taylor, K. (2011). Developing and validating the Communication
Function Classification System for individuals with cerebral palsy.
Developmental Medicine and Child Neurology,53(8), 704–710. doi:10.
1111/j.1469-8749.2011.03996.x
Higginbotham, D. J., Shane, H., Russell, S., & Caves, K. (2007). Access to
AAC: Present, past, and future. Augmentative and Alternative
Communication,23(3), 243–257. doi:10.1080/07434610701571058
Hustad, K. C., Allison, K., McFadd, E., & Riehle, K. (2014). Speech and lan-
guage development in 2-year-old children with cerebral palsy.
Developmental Neurorehabilitation,17(3), 167–175. doi:10.3109/
17518423.2012.747009
Jacob, R. K. (1991). The use of eye movements in human-computer inter-
action techniques. ACM Transactions of Information Systems,9, 152–
169. doi:10.1145/123078.128728
Kothari, A., & Wathen, C. N. (2017). Integrated knowledge translation:
Digging deeper, moving forward. Journal of Epidemiology and
Community Health,71(6), 619–623. doi:10.1136/JECH-2016-208490
Lancioni, G. E., O’Reilly, M. F., Oliva, D., Severini, L., Singh, N. N., Sigafoos,
J., & Groeneweg, J. (2007). Eye-and mouth-opening movements
replacing head and hand responses in a microswitch program for an
adolescent with deteriorating motor condition. Perceptual and Motor
Skills,105(1), 107–114. doi:10.2466/pms.105.1.107-114
Machalicek, W., Sanford, A., Lang, R., Rispoli, M., Molfenter, N., &
Mbeseha, M. K. (2010). Literacy interventions for students with phys-
ical and developmental disabilities who use aided AAC devices: A sys-
tematic review. Journal of Developmental and Physical Disabilities,
22(3), 219–240. doi:10.1007/s10882-009-9175-3
McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia
Medica,22(3), 276–282. doi:10.11613/bm.2012.031
Mei, C., Reilly, S., Reddihough, D., Mensah, F., Pennington, L., & Morgan,
A. (2016). Language outcomes of children with cerebral palsy aged 5
years and 6 years: A population-based study. Developmental Medicine
and Child Neurology,58(6), 605–611. doi:10.1111/dmcn.12957
Nederlandse Vereniging van Revalidatieartsen. (2015, April 01). Richtlijn
Spastische Cerebrale parese bij kinderen. https://richtlijnendatabase.nl/
richtlijn/spastische_cerebrale_parese_bij_kinderen/spastische_cere-
brale_parese_-_startpagina.html
Noyek, S., Vowles, C., Batorowicz, B., Davies, C., & Fayed, N. (2020). Direct
assessment of emotional well-being from children with severe motor
and communication impairment: A systematic review. Disability and
Rehabilitation,2020,1–14. doi:10.1080/17483107.2020.1810334
O’Connor, B., Kerr, C., Shields, N., & Imms, C. (2016). A systematic review
of evidence-based assessment practices by allied health practitioners
for children with cerebral palsy. Developmental Medicine and Child
Neurology,58(4), 332–347. doi:10.1111/dmcn.12973
Olswang, L. B., & Prelock, P. A. (2015). Bridging the gap between
research and practice: Implementation science. Journal of Speech
Language and Hearing Research,58(6), 1818–1826. doi:10.1044/2015
Orlandi, S., House, S. C., Karlsson, P., Saab, R., & Chau, T. (2021). Brain-
computer interfaces for children with complex communication needs
and limited mobility: A systematic review. Frontiers in Human
Neuroscience,15, 643294. doi:10.3389/fnhum.2021.643294
Oskoui, M., Coutinho, F., Dykeman, J., Jett
e, N., & Pringsheim, T. (2013).
An update on the prevalence of cerebral palsy: A systematic review
and meta-analysis. Developmental Medicine and Child Neurology,55(6),
509–519. doi:10.1111/dmcn.12080
Paley, J. (2007). Distributed cognition and the education reflex. In E.
Drummond & P. Standish (Eds.), The philosophy of nurse education
(pp. 129–142). Mcmillan Education UK.
Palisano, R. J., Rosenbaum, P., Walter, S., Russell, D., Wood, E., & Galuppi,
B. (1997). Development and reliability of a system to classify gross
motor function in children with cerebral palsy. Developmental
Medicine and Child Neurology,39(4), 214–223. doi:10.1111/j.1469-8749.
1997.tb07414.x
Pennington, L., Goldbart, J., & Marshall, J. (2004). Speech and language
therapy to improve the communication skills of children with cerebral
palsy. Cochrane Database of Systematic Reviews,2, CD003466 doi:10.
1002/14651858.CD003466.pub2
Philip, S. S., & Dutton, G. N. (2014). Identifying and characterising cere-
bral visual impairment in children: A review. Clinical & Experimental
Optometry,97(3), 196–208. doi:10.1111/CXO.12155
Pirila, S., van der Meere, J., Pentikainen, T., Ruusu-Niemi, P., Korpela, R.,
Kilpinen, J., & Nieminen, P. (2007). Language and motor speech skills
in children with cerebral palsy. Journal of Communication Disorders,
40(2), 116–128. doi:10.1016/j.jcomdis.2006.06.002
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A.,
Griffey, R., & Hensley, M. (2011). Outcomes for implementation
research: Conceptual distinctions, measurement challenges, and
research agenda. Administration and Policy in Mental Health,38(2), 65–
76. doi:10.1007/s10488-010-0319-7
Raghavendra, P., Olsson, C., Sampson, J., Mcinerney, R., & Connell, T.
(2012). School participation and social networks of children with com-
plex communication needs, physical disabilities, and typically devel-
oping peers. Augmentative and Alternative Communication,28(1), 33–
43. doi:10.3109/07434618.2011.653604
Romski, M., Sevcik, R. A., Barton-Hulsey, A., & Whitmore, A. S. (2015).
Early intervention and AAC: What a difference 30 years makes.
Augmentative and Alternative Communication,31(3), 181–202. doi:10.
3109/07434618.2015.1064163
Rosenbaum, P., Paneth, N., Leviton, A., Goldstein, M., Bax, M., Damiano,
D., Dan, B., & Jacobsson, B. (2007). A report: The definition and classi-
fication of cerebral palsy April 2006. Developmental Medicine and
Child Neurology. Supplement,109(suppl 109), 8–14. doi: 10.1111/j.
1469-8749.2007.tb12610.x
Schenk-Rootlieb, A. J., Van Nieuwenhuizen, O., Van Waes, P. F., & Van
der Graaf, Y. (1994). Cerebral visual impairment in Cerebral Palsy:
Relation to structural abnormalities of the cerebrum. Neuropediatrics,
25(2), 68–72. doi:10.1055/s-2008-1071588
Sennott, S. C., Light, J. C., & Mcnaughton, D. (2016). AAC modeling inter-
vention research review. Research and Practice for Persons with Severe
Disabilities,41(2), 101–115. doi:10.1177/1540796916638822
Sigurdardottir, S., & Vik, T. (2011). Speech, expressive language, and ver-
bal cognition of preschool children with cerebral palsy in Iceland.
Developmental Medicine and Child Neurology,53(1), 74–80. doi:10.
1111/j.1469-8749.2010.03790.x
10 J. N. BOOTSMA ET AL.
Simon, J. D. H. P., Dalington, A. S., & Grootenhuis, M. A. (n.d.). Dutch
translation of the acceptability of intervention measure (AIM), interven-
tion appropriateness measure (IAM), and feasibility of intervention meas-
ure (FIM).
Smits, D. W., Ketelaar, M., Gorter, J. W., Van Schie, P. E., Becher, J. G.,
Lindeman, E., & Jongmans, M. J. (2011). Development of non-verbal
intellectual capacity in school-age children with cerebral palsy.
Journal of Intellectual Disability Research,55(6), 550–562. doi:10.1111/j.
1365-2788.2011.01409.x
Solomon-Rice, P. L., & Soto, G. (2014). Facilitating vocabulary in toddlers
using AAC: A preliminary study comparing focused stimulation and
augmented Input. Communication Disorders Quarterly,35(4), 204–215.
doi:10.1177/1525740114522856
Stadskleiv, K. (2020). Cognitive functioning in children with cerebral
palsy. Developmental Medicine and Child Neurology,62(3), 283–289.
doi:10.1111/dmcn.14463
Stadskleiv, K., Jahnsen, R., Andersen, G. L., & von Tetzchner, S. (2018).
Neuropsychological profiles of children with cerebral palsy.
Developmental Neurorehabilitation,21(2), 108–120. doi:10.1080/
17518423.2017.1282054
Straus, S., Tetroe, J. & Graham, I. D. (Eds.). (2013). Knowledge translation
in health care: Moving from evidence to practice (2nd ed.). John Wiley
& Sons.
Tai, K., Blain, S., & Chau, T. (2008). A Review of Emerging Access
Technologies for Individuals With Severe Motor Impairments. Assistive
Technology, 20(4), 204–221. doi: 10.1080/10400435.2008.10131947
Vaillant, E., Geytenbeek, J. J. M., Jansma, E. P., Oostrom, K. J., Vermeulen,
R. J., & Buizer, A. I. (2020). Factors associated with spoken language
comprehension in children with cerebral palsy: A systematic review.
Developmental Medicine and Child Neurology,62(12), 1363–1373. doi:
10.1111/DMCN.14651
von Tetzchner, S., & Stadskleiv, K. (2016). Constructing a language in
alternative forms. In M. M. Smith & J. Murray (Eds.), The silent partner?
Language, interaction and aided communication (pp. 17–34). J&R Press
Limited.
Voorman, J. M., Dallmeijer, A. J., Van Eck, M., Schuengel, C., & Becher,
J. G. (2010). Social functioning and communication in children with
cerebral palsy: Association with disease characteristics and personal
and environmental factors. Developmental Medicine and Child
Neurology,52(5), 441–447. doi:10.1111/j.1469-8749.2009.03399.x
Vos, R. C., Dallmeijer, A., Verhoef, M., Van Schie, P., Voorman, J. M.,
Wiegerink, D. J. H., Geytenbeek, J. J. M., Roebroeck, M., & Becher, J. G.
(2014). Developmental trajectories of receptive and expressive com-
munication in children and young adults with cerebral palsy.
Developmental Medicine and Child Neurology,56(10), 951–959. doi:10.
1111/dmcn.12473
Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary,
A. S., Boynton, M. H., & Halko, H. (2017). Psychometric assessment of
three newly developed implementation science outcome measures.
Implementation Science,12(1), 1–12. doi:10.1186/s13012-017-0635-3
Zhang, J., Jadavji, Z., Zewdie, E., & Kirton, A. (2019). Evaluating if children
can use simple brain computer interfaces. Frontiers in Human
Neuroscience,13, 24. doi:10.3389/fnhum.2019.00024.
AUGMENTATIVE AND ALTERNATIVE COMMUNICATION 11
