Social Validity of Evidence-Based Practices and Emerging Interventions in Autism

Abstract

Although social validation of the goals, methods, and outcomes of evidence-based practices (EBPs) in autism treatment is a significant factor in their selection and effective use, EBPs are typically identified on the basis of the technical soundness of research without consideration of social validity. The authors investigated EBPs and emerging treatments identified by the National Autism Center (NAC) and National Professional Development Center on Autism Spectrum Disorders (NPDC) to determine which interventions have evidence of social validity, and the types of social validation addressed. A review of 828 articles cited by the NAC and NPDC determined that only 221 articles (26.7%) demonstrated direct evidence of the measurement of social validation. Of seven social validity categories analyzed, only consumer satisfaction, clinically significant behavioral change, and socially important dependent variables were consistently reported. A list of EBPs with varying levels of social and empirical validation is presented, and implications for future research are discussed.

Full text

Focus on Autism and Other
Developmental Disabilities
2017, Vol. 32(3) 188 –197
© Hammill Institute on Disabilities 2016
Reprints and permissions:
sagepub.com/journalsPermissions.nav
DOI: 10.1177/1088357616632446
focus.sagepub.com
Article
Although the field of autism intervention and treatment is
still emerging, a lack of knowledge about what to do should
no longer be considered a significant challenge facing ser-
vice providers. Recent large-scale, systematic, and robust
research reviews have identified interventions with strong
empirical evidence of effectiveness in some outcome areas.
The results of these reviews provide support for the use of
specific treatments with individuals who have autism spec-
trum disorder (ASD), especially within certain subpopula-
tions. Two initiatives, in particular, the National Autism
Center’s (NAC; 2009) National Standards Project (NSP)
and similar efforts by the National Professional Development
Center on Autism Spectrum Disorders (NPDC; Odom,
Collet-Klingenberg, Rogers, & Hatton, 2010; Wong et al.,
2014), have resulted in the dissemination of lists of empiri-
cally validated treatments that can be implemented by
teachers, parents, and therapists. Although differing some-
what in methodology, scope, and definitions (NPDC, 2013),
the NSP and NPDC studies independently and reliably
identified autism interventions and treatments considered to
be “established” and “confirmed,” respectively, as evi-
dence-based practices (EBPs) in autism. The NPDC list
originally specified 24 confirmed EBPs, 18 of which were
also identified as established treatments by the NSP.
Additional EBPs have recently been added (and one was
removed) by the NPDC (Wong et al., 2015).
Both the NSP and NPDC reviews developed criteria to
systematically analyze the content (i.e., the “quality, quan-
tity, and consistency,” NAC, 2009, p. 31) of research find-
ings and their corresponding strength of evidence. Both
research teams sought to ensure adequate empirical evi-
dence was available to reliably determine the efficacy of
treatments for individuals with ASD before labeling any
intervention as an established, confirmed, or emerging EBP.
Despite this encouraging development, research in spe-
cial education, applied behavior analysis (ABA), and other
therapeutic fields suggests empirical evidence of effective-
ness is often not sufficient to ensure the widespread use of
potentially effective interventions, treatments, and programs
(e.g., Bodfish, 2004; Carter, 2010). Although using EBPs in
the public education of children with autism and other dis-
abilities is mandated by federal regulations (e.g., Individuals
With Disabilities Education Improvement Act [IDEIA],
2004; No Child Left Behind [NCLB], 2002) and state and
632446FOAXXX10.1177/1088357616632446Focus on Autism and Other Developmental DisabilitiesCallahan et al.
research-article2016
1University of North Texas, Denton, USA
2Anadolu University, Eskisehir, Turkey
3National Taiwan Normal University, Taipei, Taiwan
Corresponding Author:
Kevin Callahan, Kristin Farmer Autism Center, University of North
Texas, 490 S. I-35 East, Denton, TX 76205-7768, USA.
Email: Callahan@unt.edu
Social Validity of Evidence-Based Practices
and Emerging Interventions in Autism
Kevin Callahan, PhD1, Heather L. Hughes, PhD1,
Smita Mehta, PhD1, Karen A. Toussaint, PhD1,
Susan M. Nichols, PhD1, Phoenix S. Ma, EdD1,
Metehan Kutlu, MEd2, and Hui-Ting Wang, PhD3
Abstract
Although social validation of the goals, methods, and outcomes of evidence-based practices (EBPs) in autism treatment is a
significant factor in their selection and effective use, EBPs are typically identified on the basis of the technical soundness of
research without consideration of social validity. The authors investigated EBPs and emerging treatments identified by the
National Autism Center (NAC) and National Professional Development Center on Autism Spectrum Disorders (NPDC)
to determine which interventions have evidence of social validity, and the types of social validation addressed. A review
of 828 articles cited by the NAC and NPDC determined that only 221 articles (26.7%) demonstrated direct evidence of
the measurement of social validation. Of seven social validity categories analyzed, only consumer satisfaction, clinically
significant behavioral change, and socially important dependent variables were consistently reported. A list of EBPs with
varying levels of social and empirical validation is presented, and implications for future research are discussed.
Keywords
autism, autism spectrum disorder, evidence-based practices, social validation, applied behavior analysis, special education

Callahan et al. 189
local education policies, many service providers resist their
use (Bodfish, 2004) or continue to implement practices dem-
onstrated by research to be ineffective (Cook & Odom,
2013; Lilienfeld, Marshall, Todd, & Shane, 2015). And,
even though the delivery of effective programming based on
scientific evidence is emphasized by professional standards
and codes of ethics in special education (Council for
Exceptional Children, 2013), ABA (Behavior Analyst
Certification Board, 2010), and most other therapeutic pro-
fessions providing autism services (e.g., American
Occupational Therapy Association, 2010), significant chal-
lenges remain in the widespread effective implementation of
research-demonstrated EBPs in public schools and other set-
tings, including homes, clinics, and centers (Cook & Cook,
2011; Mesibov & Shea, 2011). A variety of factors related to
the users of EBPs, the practices themselves, and their insti-
tutional context (Cook & Odom, 2013; Rapp et al., 2010)
have been reported to negatively affect the effective use of
EBPs. These factors include a lack of knowledge of EBPs,
limited availability of training, low competence of service
providers, high costs, lack of time for planning and prepara-
tion, and difficulties using some interventions, among many
others. Limited social validation is another possible factor
affecting the routine application of EBPs.
Social validity can generally be defined as consumer sat-
isfaction with the goals, procedures, and outcomes of pro-
grams and interventions (Alberto & Troutman, 2008; Wolf,
1978). The extent to which a particular intervention or treat-
ment receives social validation plays a significant role in
whether the treatment is adopted and implemented (Kazdin,
1977; Kern & Manz, 2004). According to Carter (2010),
“. . . just because a program is considered effective does not
mean that it will be considered appropriate by those closely
involved in implementation of the program, or that it should
even be implemented in the first place” (p. 7). Thus, without
social validation, there is perhaps less chance autism EBPs
will be used effectively in homes, schools, and clinics
(Bodfish, 2004; Carter, 2010; Kazdin, 1977; Schwartz &
Baer, 1991; Wolf, 1978). It can be argued that social valid-
ity is a necessary additional consideration in the selection
and effective use of EBPs in autism.
Extending previous seminal work investigating the qual-
ity of intervention research for individuals with disabilities
(Horner et al., 2005; Gersten et al., 2005), Reichow and col-
leagues (Reichow, Doehring, Cicchetti, & Volkmar, 2011;
Reichow, Volkmar, & Cicchetti, 2008) identified quality
indicators of robust autism research, including seven spe-
cific components of social validation. These social validity
indicators include (a) socially important dependent vari-
ables (i.e., society would value the changes in outcome of
the study), (b) time- and cost-effective intervention (i.e., the
ends justify the means), (c) comparisons between individu-
als with and without disabilities, (d) a behavioral change
that is large enough for practical value (i.e., it is clinically
significant), (e) consumers who are satisfied with the
results, (f) independent variable manipulation by people
who typically come into contact with the participant, and
(g) a natural context. Further research investigating key
aspects of social validation can provide important addi-
tional evidence to support the identification and use of spe-
cific interventions for individuals with ASD across the
autism spectrum and life span.
Fortunately, increased attention has been given to the
social validation of autism interventions and treatments.
Similar to the NSP and NPDC results, researchers have
identified autism interventions and treatment components,
as well as specific target skills, which have been socially
validated by parents, special educators, and public school
administrators (Callahan, Henson, & Cowan, 2008) and by
behavior analysts and other service providers (Dickson
et al., 2014). Callahan et al. (2008) asked parents, special
education teachers, and administrators to socially validate
essential components of autism programs based on their
perceived importance as part of comprehensive school-
based programming. These authors reported a rank-ordered
list of intervention components with broad-based support
for their use organized into a comprehensive treatment
model referred to as IDEAL, an acronym of the model’s
five functional areas, including (a) individualized program-
ming, (b) data-based decision making, (c) use of empiri-
cally demonstrated interventions, (d) active collaboration,
and (e) a focus on long-term outcomes. The IDEAL Model
components (see Table 1) were further reliably analyzed to
determine whether they aligned with practices typically
associated with ABA, the Treatment and Education of
Autistic and Related Communication-Handicapped
Children Model (TEACCH), both ABA and TEACCH, or
neither approach (Callahan, Shukla-Mehta, Magee, & Wie,
2010). A total of 24 subject matter experts with certification
and/or extensive training and experience (eight board certi-
fied behavior analysts, eight TEACCH-trained practitio-
ners, eight experts with training in both models) rated which
IDEAL Model components represented a “defining feature”
of ABA or TEACCH, or both models, as they are commonly
recognized and practiced with students with ASD. Overall
agreement with the authors’ assigned categories was 70.2%
(Callahan et al., 2010, p. 76).
More recently, Dickson et al. (2014) investigated core
skills important to individuals with autism across the life
span using an assessment instrument consisting of direct
and indirect measures of skills. Two hundred forty-five par-
ents, teachers, and therapeutic staff responsible for imple-
menting effective treatment programs were asked to indicate
which items they considered to be foundational. The results
provided a measure of social validation for specific treat-
ment targets in the domains of discrimination, communica-
tion, social skills, self-help, health and safety, and recreation/
physical education. The authors concluded these socially

190
Table 1. Summary of Evidence of SV.
IDEAL Model intervention
components/SV rating (SD)/model
affiliation: ABA, TEACCH, or BOTH
EBP/emerging practice/
articles cited (n)
Evidence of SV within NSP
and NPDC articles
Articles reporting social validity across Reichow, Doehring,
Cicchetti, and Volkmar (2011) SV categories (n)
NSP NPDC
Articles
revieweda (n)
Articles
reporting
social validity
(n/%)
Socially
important
dependent
variables
Time- and
cost-effective
Individual with
and without
disabilities
Clinically
significant
behavioral
change
Consumer
satisfied with
results
Independent
variable ×
People in typical
context
Natural
context
Social skills strategies/6.69 (0.56)/
BOTH
Social skills packageb/16 Social skills
training/15
28 13/46.4% 3 4 5 5 11 1 4
Structure and routines/6.62 (0.73)/
TEACCH
Schedules/12 NA/Structured
work systemsc
12 6/50.0% 3 3 1 5 5 1 1
Communication skills strategies/6.57
(0.70)/BOTH
Behavioral package/228 Functional
communication
training/12
234 35/15.0% 14 7 1 23 26 10 10
FBA/functional analysis/6.54 (0.69)/
ABA
Behavioral package/228 FBA/10 236 35/14.8% 14 9 1 23 26 8 8
Motivational proceduresd/6.53 (0.64)/
BOTH
Antecedent package/99 Antecedent-based
intervention/32
121 22/18.2% 10 4 3 5 16 3 3
Effective reinf./6.42 (0.80)/BOTH Behavioral package/ Reinf./43 253 35/13.8% 15 8 1 24 26 9 9
228
Task analysis/6.40 (0.79)/BOTH Behavioral package/ Task analysis/7 235 33/14.0% 15 6 2 23 24 8 9
228
Prompting, shaping, chaining, and
fading/6.39 (0.79)/ABA
Antecedent package/99 Prompting/33 128 23/18.0% 16 3 4 7 19 2 5
Incidental teaching/natural.
teach./6.32 (0.78)/BOTH
Natural. teaching
strategies/32
Natural.
intervention/10
37 17/45.9% 3 6 4 14 13 3 3
Peer-mediated intervention/6.32
(0.82)/BOTH
Peer training
package/33
Peer-mediated
instruction and
intervention/15
42 15/35.7% 8 5 6 13 3 5 4
Incentive programs based on positive
reinf./6.18 (0.94)/ABA
Behavioral package/228 Reinf./43 253 35/13.8% 15 8 1 24 26 9 9
Intensive 1:1 instruction trials/6.17
(1.00)/ABA
Behavioral package/228 Discrete trial
training/13
240 32/13.3% 14 6 1 22 23 8 8
Preferred reinforcers/6.16 (1.06)/
ABA
Antecedent package/99 Antecedent-based
intervention/32
121 22/18.2% 10 4 3 5 16 3 3
Visual activity schedules/visual
supports/6.16 (1.09)/TEACCH
Schedules/12 Visual supports/18 26 17/65.4% 13 7 4 7 16 1 2
PRT/6.15 (0.90)/BOTH PRT/14 PRT/7 17 6/35.3% 0 1 0 4 5 2 0
Individual. desks and furniture, other
classroom materials/6.10 (1.03)/
BOTH
Antecedent package/99 Antecedent-based
intervention/32
121 22/18.2% 10 4 3 5 16 3 3
Formal reinf. assessments/6.08 (0.95)/
ABA
Behavioral package/228 Reinf./43 253 35/13.8% 15 8 1 24 26 9 9
(continued)

191
IDEAL Model intervention
components/SV rating (SD)/model
affiliation: ABA, TEACCH, or BOTH
EBP/emerging practice/
articles cited (n)
Evidence of SV within NSP
and NPDC articles
Articles reporting social validity across Reichow, Doehring,
Cicchetti, and Volkmar (2011) SV categories (n)
NSP NPDC
Articles
revieweda (n)
Articles
reporting
social validity
(n/%)
Socially
important
dependent
variables
Time- and
cost-effective
Individual with
and without
disabilities
Clinically
significant
behavioral
change
Consumer
satisfied with
results
Independent
variable ×
People in typical
context
Natural
context
Differential reinf. strategies/6.06
(1.00)/ABA
Behavioral package/228 Differential reinf.
of alternative
incompat.
or other
behavioral/26
241 32/13.3% 14 6 1 22 23 8 8
Stimulus control/6.06 (1.10)/ABA Antecedent package/99 Antecedent-based
intervention/32
121 22/18.2% 10 4 3 5 16 3 3
Progress. and constant time
delay/6.05 (0.99)/ABA
Antecedent package/99 Time delay/12 107 18/16.8% 11 2 3 3 14 2 4
Adult-directed strategies/6.02 (0.97)/
ABA
Modeling/50 Prompting/33 83 17/20.5% 5 2 2 14 8 1 3
Student preferences and/or
interests/6.02 (1.03)/BOTH
Antecedent package/99 Antecedent-based
intervention/32
121 22/18.2% 10 4 3 5 16 3 3
Systematic prompts and prompt
hierarchies/5.99 (1.10)/ABA
Antecedent package/99 Prompting/33 128 23/18.0% 16 3 4 7 19 2 5
Modifying physical environment/5.95
(1.06)/TEACCH
Antecedent package/99 Antecedent-based
intervention/32
121 22/18.2% 10 4 3 5 16 3 3
Modeling/video modeling/5.93 (1.09)/
ABA
Modeling/50 Video modeling/32 66 26/39.4% 10 5 2 17 11 2 1
Extinction/5.91 (1.24)/ABA Reductive packageb/33 Extinction/11 44 1/2.3% 1 1 0 1 1 0 0
Physical classroom organizers/5.75
(1.28)/TEACCH
Schedules/12 NA/Structured
work systemsc
12 6/50.0% 3 3 1 5 5 1 1
Technology and other augmentative
communication devices/5.74 (1.30)/
BOTH
Technology-based
treatmentb/19
Technology-aided
instruction and
intervention/20
37 7/18.9% 6 2 0 3 6 2 1
Pictorial storyboard simulations
and/or social stories/5.72 (1.15)/
TEACCH
Story-based
intervention
package/21
Social
narratives/17
31 23/74.2% 13 5 2 14 13 3 7
PECS/5.71 (1.31)/TEACCH PECSb/13 PECS/6 17 6/35.3% 3 3 0 3 5 1 2
Self-management strategies/5.70
(1.10)/BOTH
Self-management/21 Self-
management/10
27 9/33.3% 5 4 1 6 6 2 4
Note. ABA = applied behavior analysis; TEACCH = Treatment and Education of Autistic and related Communication-handicapped Children; EBP = evidence-based practices; SV = social validation; NSP = National Standards
Project (National Autism Center); NPDC = National Professional Development Center on Autism Spectrum Disorders; FBA = functional behavior assessment; PRT = pivotal response training; PECS = Picture Exchange
Communication System; NAC = National Autism Center.
aThe number of articles reviewed equals the total number of articles cited by the NSP and NPDC minus the number of articles appearing on both reviews. bConsidered an “emerging practice” by the NAC. cNot applicable (NA)
because “more stringent evidence reduced previous evidence” and the intervention is no longer considered an EBP by the NPDC. dPreferred items and natural reinforcers.
Table1. (continued)

192 Focus on Autism and Other Developmental Disabilities 32(3)
validated skills, such as making requests for assistance or
desired items, responding to one’s name, and following
one-step directions, can be used as a starting point for
instructional programming for individuals with ASD
(Dickson et al., 2014).
Now that researchers have reliably identified potentially
effective interventions from which parents, teachers/thera-
pists, and administrators can choose, research attention can
begin to address specific factors affecting how these con-
sumers select and implement EBPs (Cook & Odom, 2013).
A critical first step in this process is to help service provid-
ers identify from among the relatively large list of available
interventions and treatments those that, ostensibly, have the
greatest likelihood of making a positive impact on specific
target deficits of their children, students, and clients (Odom,
2009; Odom, Cox, Brock, & the NPDC, 2013). The results
of additional social validity research can help inform this
process. For example, identifying which specific social val-
idation components are most critical for the successful
implementation of EBPs, and which aspects of social valid-
ity are explored most and least frequently in the literature,
can provide new insights for practitioners and researchers.
The purpose of this study is to identify the extent to
which autism interventions and treatment components are
reported to have both social and empirical validation and to
investigate the types of social validity commonly addressed
by autism researchers and practitioners. The resulting list of
EBPs could serve as a starting point for service providers
designing and implementing autism programming.
Identifying socially validated EBPs could help address (and
possibly eliminate) some of the barriers to effective imple-
mentation of EBPs, and address the pervasive research-to-
practice gaps in autism programming (Dingfelder &
Mandell, 2011). Most importantly, a prioritized list of vali-
dated interventions and training resources could ultimately
result in improved outcomes for individuals with ASD and
their families.
Method
Participants and Procedures
Preliminary analysis. Two analyses were conducted to iden-
tify the extent to which empirically demonstrated EBPs also
have evidence of social validation. For the first analysis, a
team of three researchers representing doctoral-level exper-
tise in special education, ABA, autism assessment, and
early childhood education compared the correspondence of
empirically validated treatments and interventions using the
NSP and NPDC lists of established, confirmed, and emerg-
ing EBPs with a list of 60 evidence-based intervention com-
ponents that were socially validated using a mail survey by
187 parents, special education teachers, and public school
administrators (Callahan et al., 2008). The authors included
treatments categorized as “emerging” by the NSP. These
treatments are supported by a minimum of one group design
or two single-subject design studies with a minimum of six
participants, with adequate scientific merit, and no conflict-
ing results reported. In addition, beneficial treatment effects
have been reported for these interventions for at least one
dependent variable/specific target behavior (NAC, 2009).
Previously, survey respondents rated each of the 60 inter-
vention components on a 7-point Likert-type scale based on
their perceived importance of the intervention as part of a
high-quality autism program, where a rating of 7 indicated
the component was “Absolutely Important (the component
is an indispensable part of a high-quality autism program)”
and a rating of 1 meant the component was “not at all
important (the component is totally irrelevant and/or unre-
lated to a high-quality autism program).”
Inter-rater agreement. For this preliminary study, the
research team independently analyzed each of the socially
validated intervention components of the IDEAL Model to
identify corresponding treatments empirically validated by
the NSP and/or NPDC reports. Each member of the research
team was instructed to thoroughly read designated NSP and
NPDC materials to become knowledgeable about the
research and analysis procedures and the treatment defini-
tions used. The researchers were directed to independently
review each of the 60 IDEAL Model components and deter-
mine whether the components directly aligned with any of
the 24 treatments identified as “established” or “emerging”
by the NSP and/or those “confirmed” as an EBP by the
NPDC. Descriptions of the identified EBPs and IDEAL
Model components were used by the study authors during
the review and alignment phase of this analysis. The NSP
information included the name of the intervention package,
if applicable, as well as descriptive information about the
various interventions making up the package. For example,
the NSP’s “Antecedent Package” was defined as including
behavior chain interruption, behavioral momentum, choice,
and cueing/prompting, among other components. The
results of the independent ratings were analyzed for agree-
ment. Twenty-three of the 24 NSP/NPDC EBPs aligned
directly with one or more of the socially validated IDEAL
Model intervention components. Inter-rater agreement
across all intervention components was 89.4%. The research
team met to address all items for which there was less than
100% agreement among the three raters. This process clari-
fied definitions and resulted in 100% agreement among all
raters for all items.
Outcomes. A total of 31 IDEAL Model components were
determined to align directly with an NSP/NPDC EBP.
Although typically a single IDEAL Model component
aligned with just one EBP, in some cases, more than one
IDEAL Model component corresponded with a particular

Callahan et al. 193
EBP. For example, two IDEAL Model components, “The
use of visual activity schedules . . .,” and “The use of physi-
cal classroom organizers . . .,” were determined to align
with the EBP “Visual Supports” (NPDC)/”Schedules”
(NSP).
Expert validation process. The corresponding IDEAL Model
components and NSP/NPDC EBPs were systematically
validated by experts in the field of autism. Using Qualtrics
software, Version 13, of the Qualtrics Research Suite (Qual-
trics, 2013), a link to an online survey was sent to seven
persons with extensive experience and training in autism
intervention, including university faculty members teaching
autism graduate courses, and public school and center-
based practitioners with board certification in ABA and/or
formal training in the TEACCH Model. The experts were
asked to validate that the socially validated IDEAL Model
components represented the same interventions and treat-
ments within the NSP/NPDC studies. Respondents were
instructed to indicate agreement (“Yes”) or disagreement
(“No”) that the NSP/NPDC EBP referred to the same treat-
ment or intervention as conceptualized within the IDEAL
Model. For example, the experts were asked to determine
whether the IDEAL Model intervention component, “The
use of incidental teaching or naturalistic teaching . . .,” rep-
resented the same intervention as the NSP’s “Naturalistic
Teaching Strategies” and the NPDC’s “Naturalistic inter-
ventions.” All but one expert validated 100% of the items as
representing the same interventions. The overall mean per-
centage agreement of the expert validators was 99.6%
(range = 96.7%–100%), indicating a very high level of
agreement that the socially validated IDEAL Model inter-
vention components represent the same treatments empiri-
cally validated by the NSP and NPDC.
Secondary analysis. Because the procedures implemented to
determine the social validity of the IDEAL Model compo-
nents addressed only a limited aspect of social validation—
that is, the perceived value of specific intervention
components by consumers within public school autism
programs—a second analysis was conducted to determine
whether broader evidence of social validation exists for
EBPs and emerging practices in autism. For this analysis,
an expanded research team consisting of the original mem-
bers and four additional PhD-level researchers reviewed all
articles cited within the NSP and NPDC reviews for the cor-
responding socially validated IDEAL Model components.
The purpose was to determine whether the author(s) directly
addressed social validation, and to identify the types of
social validity reported. A total of 828 unique articles within
the reference lists for the relevant NSP and NPDC EBPs
were reviewed.
Articles were searched using an automated search func-
tion and at least the following search terms: valid, social,
and satis. In situations where the automated search was
inoperable or failed to locate social validity information, the
researchers reviewed the full article to identify possible ref-
erences to social validity and social validation. Researchers
were directed to search every article for language and termi-
nology not captured in the automated searches.
The research team used an intentionally conservative
approach to reliably identify direct evidence of measure-
ment of social validation. That is, the researchers coded
articles as demonstrating direct evidence of social valida-
tion only if the author(s) specifically collected social valid-
ity data. The research team only considered the explicit
language of the author(s) using the words, headings,
phrases, and/or concepts appearing in the articles which
clearly addressed or referred to social validity or social vali-
dation. Most often, this information was located within sec-
tions of the article labeled “Social Validation,” or “Parent
Satisfaction,” or similar headings. In rare cases, the review-
ers located relevant information embedded within other
sections of the articles. In all instances in which the article
authors examined social validity, at least one Reichow et al.
(2011) category was identified and coded.
Finally, to determine the types of social validation most
often addressed by autism researchers, article reviewers
assessed the author(s)’ stated evidence of social validation
and assigned the information to one or more of the Reichow
et al. (2011) categories of social validity. For example, the
statement “Social validation measures provided by 30 raters
indicated that perceived positive changes in performance
had occurred in child behaviors, and training procedures
were acceptable” could be assigned to Reichow et al.’s cat-
egories of “Clinically significant behavioral change” and
“Consumers satisfied with the results,” respectively.
Inter-rater agreement. Of the 828 articles reviewed within
the reference lists of the NSP and NPDC EBP summary
reports, a total of 201 articles (24.3%) were reviewed and
coded by at least two different researchers to determine
inter-rater reliability in assessing evidence of social validity
and agreement among the reviewers for assignments of
social validity evidence to the Reichow et al. (2011) catego-
ries. A reliability coefficient was calculated for each article
by dividing the number of rater agreements by the number of
agreements plus disagreements and multiplying the results
by 100 to obtain a percentage of agreement. Agreements and
disagreements were compared for each of the seven Reichow
et al. (2011) categories, as well as for the basic question,
“Did the article contain evidence of social validity?” An
agreement was defined as both reviewers scoring the
Reichow et al. category and answering the evidence of social
validity question the same. There was 100% agreement
regarding whether articles contained evidence of social
validity. Across all articles in which social validation was
identified and those articles in which social validity was not

194 Focus on Autism and Other Developmental Disabilities 32(3)
identified, overall inter-rater agreement on the assignment of
social validity evidence to the Reichow et al. categories was
97.4%. For those articles only in which social validity was
identified, overall inter-rater agreement was 88.4% (range =
62.5%–100%). Within these articles, inter-rater agreement
for each Reichow et al. social validity category was as fol-
lows: (a) socially relevant Dependent Variables = 84.4%, (b)
time- and cost-effective = 88.8%, (c) persons with and with-
out disabilities = 93.3%, (d) clinically significant behavioral
change = 82.2%, (e) consumers satisfied with the results =
88.8%, (f) Independent Variable by people in typical context
= 86.6%, and (g) a natural context = 84.4%. When inter-rater
agreement was not reached, the raters discussed how they
coded the specific information in the article and whether it
fit the established coding criteria. After independently re-
rating these articles, consensus was reached in all cases.
Results
IDEAL Model Alignment With NSP and NPDC
Table 1 contains a list of the 31 socially validated IDEAL
Model autism interventions that correspond directly with
EBPs and emerging treatments empirically demonstrated to
be effective for individuals with ASD. As noted above, each
IDEAL Model intervention component was systematically
determined to represent the same specific EBP or treatment/
package validated by the NPDC and NAC, respectively.
Complete verbatim descriptions of the IDEAL Model
Intervention Components in Table 1 may be found in
Callahan et al. (2008). The social validation ratings on the
31 interventions range from 5.70 to 6.69 (M = 6.17) on a
7-point scale, indicating a high to very high level of support
by key consumers for the use of these treatments in autism
programming within schools, clinics, and homes.
Table 1 includes a designated model affiliation associ-
ated with each intervention, based on earlier research
(Callahan et al., 2010). Callahan and colleagues previously
analyzed whether socially validated autism interventions
were associated with ABA, TEACCH, both models, or nei-
ther model. Prior results suggested the primary consumers
of autism programming value intervention components
inherent within both ABA and TEACCH above treatments
associated with either model alone.
Several of the socially validated EBPs were identified
by the NSP as emerging, versus established, practices.
These EBPs were “Social Skills Training Groups/Social
Skill Package,” “Computer Aided Instruction/Technology-
Aided Instruction and Intervention,” “Picture Exchange
Communication,” and “Extinction.”
Four established EBPs had more than one corresponding
IDEAL Model intervention component. “Antecedent-Based
Intervention/Antecedent Package” was represented by six
practices socially validated within the IDEAL Model,
including the use of specific motivational procedures, stu-
dent preferences, stimulus control, physical modifications
to the instructional environment, and others. The effective
use of “Reinforcement/Behavioral Package” was supported
by three socially validated interventions.
Social Validity Within NSP and NPDC EBPs
As can be seen in Table 1, the secondary analysis identified
additional evidence of social validity for all of the NSP and
NPDC EBPs aligning with the IDEAL Model. Across all
EBPs, the mean percentage of articles in which the authors
directly reported evidence of social validity was 26.7%.
However, the percentage of articles in which the authors
reported social validity was variable, ranging from a low of
2.3% for the use of extinction to 74.2% for the use of Pictorial
Story Board Simulations/Social Stories (see Table 1).
Of the Reichow et al. (2011) categories, researchers
most often addressed social validity by reporting evidence
within the categories of “Consumers Satisfied with the
Results,” “Clinically Significant Behavioral Change,” and
“Socially Important Dependent Variables.” Across all
EBPs, more than 73% of the research articles in which
social validation evidence was present addressed consumer
satisfaction with the specific EBP, while approximately half
of the articles directly addressed whether the intervention
targeted socially important variables, and whether the treat-
ment had clinically significant benefits. Relatively few
researchers addressed the time and cost-effectiveness of
interventions, and where or with whom the interventions
were implemented. Table 2 shows the overall mean percent-
age of articles containing social validation evidence for
each of the Reichow et al. (2011) categories.
Analyses of social validity of the EBPs and emerging
treatments by their affiliated model (i.e., ABA, TEACCH, or
BOTH ABA and TEACCH) were conducted. EBPs associ-
ated with the TEACCH Model (N = 6; M = 48.9%) had a
much higher overall percentage of articles addressing social
validity than treatments associated with the ABA approach
alone (N = 13; M = 17.0%) or those representing BOTH
ABA and TEACCH (N = 12; M = 26.1%). The results of a
one-way, between-subjects analysis of variance indicated a
statistically significant difference between TEACCH and
ABA, and between TEACCH and the BOTH group of inter-
ventions, F(2, 28) = 12.949, p < .000, MSerror = 161.522, α =
.05. The analysis of variance did not identify a statistically
significant difference between the ABA treatments and those
in the BOTH category.
Discussion
This research provides new information for consumers of
autism interventions. For the first time, the literature
addressing EBPs and social validity has been systematically

Callahan et al. 195
integrated to help parents, educators, therapists, and others
access research findings that include both evidence of effec-
tiveness and validation of support for the goals, procedures,
and outcomes of autism treatments. As a result, service pro-
viders and administrators now have a menu of research-
validated interventions that can serve as a blueprint for
treatment implementation, strategic planning, and program
evaluation to improve outcomes, establish training priori-
ties (Hughes, Combes, & Mehta, 2012), and build effective
communities of practice (Cook & Odom, 2013; Hall, 2015).
Adding a social validation component to efforts by the
NAC, NPDC, and others to identify and define EBPs sup-
ports long-standing and renewed calls for making social
validity a routine part of program application and evalua-
tion (Carter, 2010; Foster & Mash, 1999; Kazdin, 1977;
Reichow, 2011; Schwartz & Baer, 1991; Wolf, 1978).
Recommendations to refine and expand social validation in
behavior change research have been made and, some might
argue, ignored, for decades beginning with the seminal
work of Kazdin (1977) and Wolf (1978). Fifteen years later,
Kennedy (1992) assessed that only 20% of articles pub-
lished in the journals Behavior Modification and the Journal
of Applied Behavior Analysis addressed measures of social
validity, leading to additional questions about how best to
define, conceptualize, and measure the construct of social
validation. The results of our research suggest the leading
behavioral journals continue to report a low rate of social
validity measurement in their autism treatment studies. For
example, of 182 articles published in the Journal of Applied
Behavior Analysis, only 19 studies (10.4%) directly reported
the measurement of social validation. If social validation
research is to have a maximum impact on improving the
effectiveness of applied interventions, researchers must
make renewed efforts to address fundamental issues about
“when to use social validity measures, and what measures
are most appropriate” (Kennedy, 1992, p. 154). In this
study, the low rates of social validity observed in several of
the Reichow categories underscores the continuing need for
exploring potentially important differences in the way
researchers value, select, implement, and report social vali-
dation measures.
It is important to note that establishing the social validity
of any particular treatment is not a substitute for fully exam-
ining the intervention’s efficacy in other relevant ways
(Schwartz & Baer, 1991). Further evaluation is necessary to
help specify the salient steps or components of EBPs which
result in maximum impact and widespread adoption by
practitioners—important elements of implementation sci-
ence (Odom, 2009). In addition, research to investigate
interactions between key factors, such as the age and sever-
ity of students and clients, and the types of treatment set-
tings, is warranted.
Limitations and Future Directions
Given the processes implemented in previous studies lead-
ing to this research, wherein researchers solicited social
validation ratings of autism intervention components with
widely disseminated positive outcomes, it is not surprising
key stakeholders would broadly support their use. Thus,
additional research is needed to extend these preliminary
findings to assess consumer satisfaction with the goals, pro-
cedures, and outcomes of a larger array of practices in
homes, schools, and clinics. In addition, the conservative
approach used by this research team to identify evidence of
social validity undoubtedly resulted in under-reporting the
percentage of NSP and NPDC articles containing social
validity evidence. For example, most of the articles
reviewed detailed the results of interventions with outcomes
that could be considered to be clinically significant.
However, because no effort was made by the authors to
directly address the concept of social validity, these articles
were not coded as providing evidence of social validation.
If future research was conducted with a broader definition
of what constitutes social validity, the results would likely
indicate higher levels of social validation than reported
here. Furthermore, these results are limited to only those
studies included in the NSP and NPDC reviews. It is possi-
ble different levels of social validation could be reported in
autism research not cited by the NSP and NPDC. In addi-
tion, our analyses of the combined articles including both
the NAC packages (e.g., the 228 articles within the
Behavioral Package) with the more discrete NPDC treat-
ments (e.g., the 10 articles for Functional Behavior
Assessment [FBA]) are problematic. In this example, with
only two articles in common, the majority of the Behavioral
Package studies clearly targeted interventions other than
FBA. Future research should attempt to identify specific
Table 2. Mean Percentage of Articles Reporting Social Validity Within Reichow, Doehring, Cicchetti, and Volkmar (2011) Categories
Across All EBPs.
Socially important
dependent
variables
Time- and
cost-effective
Individuals
with/without
disabilities
Clinically
significant
behavioral change
Consumers
satisfied with
the results
IV by people
in typical
contact
A natural
context
Mean % of articles 48.4 27.4 11.5 55.1 73.3 17.4 19.7
Note. EBPs = evidence-based practices.

196 Focus on Autism and Other Developmental Disabilities 32(3)
articles within the larger NAC packages, which more
closely match the corresponding treatments in the NPDC
and IDEAL research. Finally, an important future direction
for autism treatment research is to maximize the potential
value of lists of effective interventions such as those pre-
sented here and disseminated by the NAC and NPDC. It is
potentially problematic for service providers that the listed
interventions have been classified using a variety of con-
ceptualizations and methods. Autism researchers should
identify inherent problems with such classification schemes
and move toward investigating common principles and
practices that underlie the effective application of evidence-
based treatments.
Conclusion
These results affirm that empirically demonstrated autism
treatments believed to be important by parents, teachers,
therapists, and administrators have evidence of social valid-
ity. It is encouraging that all of the EBPs identified as estab-
lished, confirmed, or emerging by the NAC and NPDC
have at least a minimal level of social validation to comple-
ment the empirical evidence of effectiveness. At the same
time, with less than 27% of the reviewed research articles
directly addressing social validity, and in light of the wide
range of reported social validity among EBPs, it is clear that
the overall evidence for social validation remains limited.
In addition, the relatively low rates of social validity for
some treatments considered to be standard parts of compre-
hensive autism programming (e.g., the use of positive and
differential reinforcement, Discrete Trial Training, and
extinction) suggests much more work needs to be accom-
plished in this important area.
Although it is beyond the scope of this article, we sup-
port the future development of improved processes, includ-
ing standardized frameworks for socially validating EBPs
in autism intervention (Schlosser, 1999). For now, however,
immediate actions can include consideration of policy
changes by the editors of journals in applied research, which
would require authors to directly and clearly report how
social validity was measured. Researchers themselves
should also begin to include descriptive social validity
information within all intervention research manuscripts
submitted for publication.
As part of the evolution of autism research and interven-
tion from a focus on what works to why and how specific
treatments can be implemented with fidelity to bridge the
research-to-practice gap, it is important that researchers and
practitioners consider social validity when making instruc-
tional and therapeutic decisions.
Acknowledgments
The authors wish to thank the expert validators for their valuable
assistance with this research.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with
respect to the research, authorship, and/or publication of this
article.
Funding
The author(s) received no financial support for the research,
authorship, and/or publication of this article.
References
Alberto, P. A., & Troutman, A. C. (2008). Applied behavior analy-
sis for teachers (8th ed.). Columbus, OH: Pearson.
American Occupational Therapy Association. (2010).
Occupational therapy code of ethics and ethics standards
(2010). Retrieved from http://www.aota.org/Practice/Ethics.
aspx
Behavior Analyst Certification Board. (2010). BACB guidelines
for responsible conduct for behavior analysts. Retrieved from
http://www.bacd.com/index.php?page=57
Bodfish, J. W. (2004). Treating the core features of autism: Are we
there yet? Mental Retardation and Developmental Disabilities
Research Reviews, 10, 318–326.
Callahan, K., Henson, R., & Cowan, A. (2008). Social validation
of evidence-based practices in autism by parents, teachers,
and administrators. Journal of Autism and Developmental
Disorders, 38, 678–692.
Callahan, K., Shukla-Mehta, S., Magee, S., & Wie, M. (2010).
ABA versus TEACCH: The case for defining and validating
comprehensive treatment models in autism. Journal of Autism
and Developmental Disorders, 40, 74–88.
Carter, S. L. (2010). The social validity manual: A guide to sub-
jective evaluation of behavioral interventions. Boston, MA:
Academic Press.
Cook, B. G., & Cook, C. C. (2011). Unraveling evidence-based
practices in special education. The Journal of Special
Education, 47, 71–82.
Cook, B. G., & Odom, S. L. (2013). Evidence-based practices
and implementation science in special education. Exceptional
Children, 79, 135–144.
Council for Exceptional Children. (2013). Special education pro-
fessional ethical principles and practice standards for spe-
cial education professionals. Retrieved from http://www.
cec.sped.org/Standards/Ethical-Principles-and-Practice-
Standards?sc_lang=en
Dickson, C. A., MacDonald, R. P. F., Mansfield, R., Guilhardi, P.,
Johnson, C., & Ahearn, W. H. (2014). Social validation of the
New England Center for Children-Core Skills Assessment.
Journal of Autism and Developmental Disorders, 44, 65-74.
doi:10.1007/s10803-013-1852-5
Dingfelder, H. E., & Mandell, D. S. (2011). Bridging the
research-to-practice gap in autism intervention: An applica-
tion of diffusion of innovation theory. Journal of Autism and
Developmental Disorder, 41, 597–609.
Foster, S. L., & Mash, E. J. (1999). Assessing social validity in
clinical treatment research: Issues and procedures. Journal of
Consulting and Clinical Psychology, 67, 308–319.
Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood,
C., & Innocenti, M. S. (2005). Quality indicators for group

Callahan et al. 197
experimental and quasi-experimental research in special edu-
cation. Exceptional Children, 71, 149–164.
Hall, L. J. (2015). Sustaining evidence-based practices by gradu-
ated special educators of students with ASD: Creating a com-
munity of practice. Teacher Education and Special Education,
38, 28–43. doi:10.1177/0888406414558883
Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., &
Wolery, M. (2005). The use of single-subject research to iden-
tify evidence-based practice in special education. Exceptional
Children, 71, 165–179.
Hughes, H. L., Combes, B. H., & Mehta, S. S. (2012). Managing
autism: Knowledge and training in autism spectrum disorders
among special education administrators in Texas. Journal of
Special Education Leadership, 25(2), 90–98.
Individuals With Disabilities Education Improvement Act (2004).
20 U.S.C., § 1415 (reauthorization of the Individuals With
Disabilities Education Act of 1990).
Kazdin, A. E. (1977). Assessing the clinical or applied signifi-
cance of behavior change through social validation. Behavior
Modification, 1, 427–452.
Kennedy, C. H. (1992). Trends in the measurement of social valid-
ity. The Behavior Analyst, 15, 147–156.
Kern, L., & Manz, P. (2004). A look at current validity issues of
school-wide behavior support. Behavioral Disorders, 30, 47–59.
Lilienfeld, S. O., Marshall, J., Todd, J. T., & Shane, H. C. (2015).
The persistence of fad interventions in the face of negative
scientific evidence: Facilitated communication for autism as a
case example. Evidence-Based Communication Assessment and
Intervention, 8, 62–101. doi:10.1080/17489539.2014.976332
Mesibov, G. B., & Shea, V. (2011). Evidence-based practices and
autism. Autism, 15, 114–133.
National Autism Center. (2009). Addressing the need for evi-
dence-based practice guidelines for autism spectrum dis-
orders (National Standards Report: The National Standards
Project). Randolph, MA: Author.
National Professional Development Center. (2013). The NPDC
on ASD and the National Standards Project. Retrieved from
http://autismpdc.fpg.unc.edu/evidence-based-practices
No Child Left Behind (NCLB) Act of 2001, Publ. L. No. 107–110,
§ 115, Stat.1425 (2002).
Odom, S. L. (2009). The tie that binds: Evidence-based prac-
tice, implementation science, and outcomes for children.
Topics in Early Childhood Special Education, 29, 53–61.
doi:10.1177/0271121408329171
Odom, S. L., Collet-Klingenberg, L., Rogers, S., & Hatton, D.
(2010). Evidence-based practices in interventions for children
and youth with autism spectrum disorders. Preventing School
Failure, 54, 275–282.
Odom, S. L., Cox, A. W., Brock, M. E., & the National
Professional Development Center. (2013). Implementation
science, professional development, and autism spectrum dis-
orders. Exceptional Children, 79, 233–251.
Qualtrics. (2013). Qualtrics (Version 13). Provo, UT: Author.
Rapp, C. A., Etzel-Wise, D., Marty, D., Coffman, M., Carlson,
L., Asher, D., . . . Holter, M. (2010). Barriers to evidence-
based practice implementation: Results of a qualitative study.
Community Mental Health Journal, 46, 112–118.
Reichow, B. (2011). Developments, procedures, and application of
the evaluative method for determining evidence-based prac-
tices in autism. In B. Reichow, P. Doehring, D. V. Cicchetti
& F. R. Volkmar (Eds.), Evidence-based practices and treat-
ments for children with autism (pp. 25–39). New York, NY:
Springer.
Reichow, B., Doehring, P., Cicchetti, D. V., & Volkmar, F. R.
(2011). Evidence-based practices and treatments for children
with autism. New York, NY: Springer.
Reichow, B., Volkmar, F. R., & Cicchetti, D. V. (2008).
Development of the evaluative method for evaluating and
determining evidence-based practices in autism. Journal
of Autism and Developmental Disorders, 38, 1311–1319.
doi:10.1007/s10803-007-0517-7
Schlosser, R. W. (1999). Social validation of interventions in aug-
mentative and alternative communication. Alternative and
Augmentative Communication, 15, 234–247.
Schwartz, I. S., & Baer, D. M. (1991). Social validity assessments:
Is current practice the state of the art? Journal of Applied
Behavior Analysis, 24, 189–204.
Wolf, M. M. (1978). Social validity: The case for subjective mea-
surement or how applied behavior analysis is finding its heart.
Journal of Applied Behavior Analysis, 11, 203–214.
Wong, C., Odom, S. L., Hume, K. A., Cox, A. W., Fettig, A.,
Kucharczyk, S., . . . Schultz, T. R. (2014). Evidence-based
practices for children, youth, and young adults with autism
spectrum disorder. Chapel Hill: Frank Porter Graham Child
Development Institute, Autism Evidence-Based Practice
Review Group, The University of North Carolina.
Wong, C., Odom, S. L., Hume, K. A., Cox, A. W., Fettig, A.,
Kucharczyk, S., . . . Schultz, T. R. (2015). Evidence-based
practices for children, youth, and young adults with autism
spectrum disorder: A comprehensive review. Journal of
Autism and Developmental Disorders, 45, 1951–1966.
doi:10.1007/s10803-014-2351-z