Content uploaded by Gerald P Koocher
Author content
All content in this area was uploaded by Gerald P Koocher on Nov 24, 2014
Content may be subject to copyright.
This article was downloaded by: [DePaul University]
On: 25 April 2014, At: 13:08
Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,
37-41 Mortimer Street, London W1T 3JH, UK
Journal of Clinical Child & Adolescent Psychology
Publication details, including instructions for authors and subscription information:
http://www.tandfonline.com/loi/hcap20
Discredited Assessment and Treatment Methods Used
with Children and Adolescents: A Delphi Poll
Gerald P. Koocher a , Madeline R. McMann a , Annika O. Stout a & John C. Norcross b
a Department of Psychology , Simmons College
b Department of Psychology , University of Scranton
Published online: 25 Apr 2014.
To cite this article: Gerald P. Koocher , Madeline R. McMann , Annika O. Stout & John C. Norcross (2014): Discredited
Assessment and Treatment Methods Used with Children and Adolescents: A Delphi Poll, Journal of Clinical Child & Adolescent
Psychology, DOI: 10.1080/15374416.2014.895941
To link to this article: http://dx.doi.org/10.1080/15374416.2014.895941
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained
in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the
Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and
are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and
should be independently verified with primary sources of information. Taylor and Francis shall not be liable for
any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever
or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of
the Content.
This article may be used for research, teaching, and private study purposes. Any substantial or systematic
reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any
form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://
www.tandfonline.com/page/terms-and-conditions
Discredited Assessment and Treatment Methods Used
with Children and Adolescents: A Delphi Poll
Gerald P. Koocher, Madeline R. McMann, and Annika O. Stout
Department of Psychology, Simmons College
John C. Norcross
Department of Psychology, University of Scranton
In the context of intense interest in identifying what works in mental health, we sought to
establish a consensus on what doesnot work—discredited psychological assessments and
treatments used with children and adolescents. Applying a Delphi methodology, we
engaged a panel of 139 experts to participate in a two-stage survey. Participants reported
their familiarity with 67 treatments and 35 assessment techniques and rated each on a
continuum from not at all discredited to certainly discredited. The composite results
suggest considerable convergence in what is considered discredited and offer a first step
in identifying discredited procedures in modern mental health practice for children and
adolescents. It may prove as useful and easier to identify what does not work for youth
as it is to identify what does work—as in evidence-based practice compilations. In
either case, we can simultaneously avoid consensually identified discredited practices
to eradicate what does not work and use inclusively defined evidence-based practices
to promote what does work.
Which psychological tests and assessment procedures
give us the most accurate data when assessing child and
adolescent clients? Which psychotherapies consistently
prove effective in treating the conditions we diagnose?
The era of evidence-based practice (EBP) has inundated
clinicians with lists of best practices, treatment guidelines,
empirically supported therapies, practice guidelines, and
reimbursable procedure codes. Dozens of compilations
now offer, to varying degrees, evidence-based methods
to employ with youth (e.g., American Psychiatric Associ-
ation, 2006; Hersen & Sturmey, 2012; LeCroy, 2008;
Rubin, 2011; Spirito & Kazak, 2005; Weisz & Kazdin,
2010). All are noble attempts to identify and disseminate
what works in mental health.
At the same time, relatively little attention has focused
on identifying ineffective treatments and invalid tests for
youth. That is, what does not work beyond the passage
of time alone, expectancy, base rates, or credible placebo.
In those clinical circumstances when few validation
studies or few randomized clinical trials exist, how can
we, as practitioners, educators, and as an entire
discipline, draw a line between methods that enjoy the
confidence of the experts and those that experience
widespread skepticism?
Several authors have attempted to identify pseudos-
cientific, unvalidated, potentially harmful, or ‘‘quack’’
psychotherapies (e.g., Carroll, 2003; Della Sala, 1999;
Eisner, 2000; Lilienfeld, 2007; Lilienfeld, Lynn, & Lohr,
2003; Singer & Lalich, 1996), including those for select
youth disorders (e.g., Jacobson, Foxx, & Mulick,
2005). Parallel efforts have focused on identifying assess-
ment measures of questionable validity on psychometric
grounds (e.g., Hunsley, Crabb, & Mash, 2004; Hunsley
& Mash, 2005).
These pioneering efforts suffered from at least
two prominent limitations. First, none of the efforts
systematically relied on expert consensus to reach their
conclusions. Instead, the authors assumed that a
professional consensus already existed, or they selected
entries on the basis of their own opinions. Second, these
Correspondence should be addressed to Gerald P. Koocher,
College of Science and Health, DePaul University, 1110 West Belden
Avenue, Chicago, IL 60614. E-mail: gkoocher@depaul.edu
Journal of Clinical Child & Adolescent Psychology, 0(0), 1–8, 2014
Copyright #Taylor & Francis Group, LLC
ISSN: 1537-4416 print=1537-4424 online
DOI: 10.1080/15374416.2014.895941
Downloaded by [DePaul University] at 13:08 25 April 2014
authors provided little logical differentiation between
credible and noncredible treatments and between vali-
dated and unvalidated tests. This demarcation problem
(Gardner, 2000)—the challenge of formulating sharp
distinctions between validated and unvalidated—lead
to rather crude and dichotomous judgments. Previous
efforts were often less than systematic.
We took a different tack to identifying discredited
procedures in mental health. We chose to conduct
Delphi polls of mental health experts to secure a consen-
sus and to establish more refined characterizations of
treatments and tests ranging along a continuum from
not at all discredited to certainly discredited. Having
previously focused on discredited procedures in both
adults and the addictions (Norcross, Koocher, Fala, &
Wexler, 2010; Norcross, Koocher, & Garofalo, 2006),
we sought in this study to apply the same tack to ident-
ify discredited assessment and treatment methods used
with youths.
METHODS
The Delphi Poll
We searched broadly to collect nominations for discre-
dited mental health treatments and tests via literature
searches, electronic list requests, and peer consultations.
Our inclusion criteria included treatments and tests used
professionally for mental health purposes during the past
100 years in the United States or Western Europe.
Exclusion criteria were controversial theories of psy-
chology that did not directly involve mental health (e.g.,
maternal employment as a cause of child maladjustment,
intrauterine learning), unusual phenomena regarding
youth (e.g., imaginary playmates, extra sensory percep-
tion) that have not yielded pertinent treatments, treat-
ments or assessments that have never found advocacy
in mental health (e.g., astrology, numerology), medica-
tions or biochemical substances (including conventional,
herbal, naturopathic, or homeopathic preparations),
and practices used primarily outside the United States
and Western Europe.
Using these criteria, we compiled and listed separately
59 candidate treatments and 30 candidate assessment
procedures on a questionnaire. In the interest of inclu-
siveness, we listed all nominations received, even though
some of the methods have acquired a body of published
peer-reviewed support. The poll listed the 89 therapy and
assessment methods and asked each participant to rate
them using a 5-point Likert-type scale (per instructions
provided next). Items were presented alphabetically
and with reference to a particular purpose or condition.
For example, ‘‘acupuncture for treatment of childhood
mental=behavioral disorders’’ and ‘‘applied kinesiology
for treatment of ADHD’’ were listed under the treatment
section, and ‘‘anatomically detailed dolls for use in
diagnosing child sexual abuse’’ and ‘‘Blacky Test for
personality assessment with children’’ were listed under
the test section.
The Expert Panel
In October 2012, we invited approximately 150 doctoral-
level mental health experts to participate in our Delphi
poll using personalized e-mail messages. The adjective
‘‘approximately’’ references the fact that in order to com-
ply with antispam policies we contacted each potential
participant individually to ensure that we had an accu-
rate e-mail address and that the person would entertain
our solicitation. This involved probing more than 150
potential e-mail addresses to connect electronically with
the participants. All were mental health professionals
with demonstrated expertise in working with children
and adolescents. Once 150 valid addresses were con-
firmed, we solicited those identified to serve on our panel.
Specifically, we invited
.editors and associate editors of scholarly journals
focused on child and adolescent mental health;
.premier researchers in the same arena (as deter-
mined by Web of Science citation counts);
.authors and editors of books on youth psycho-
pathology, assessment, or mental health;
.experts on psychodiagnostic assessment with
children and adolescents (drawn from among
journal editors and book authors);
.randomly selected psychologists holding certifi-
cation from the American Board of Professional
Psychology in the specialty of clinical child clinical
and adolescent psychology; and
.randomly selected psychiatrists certified by the
American Academy of Child and Adolescent
Psychiatry.
Many of the invited participants met more than one
selection criterion but are categorized here using the
initial mechanism for identifying them. The 139 who
ultimately agreed to participate were sent a link to a
SurveyMonkey online questionnaire. All 139 completed
and submitted the first round of the questionnaire.
Following the standard Delphi procedure, our panel
of experts answered the same items twice. In the first
round, the experts answered the questions anonymously
and without knowledge of the responses of their peers.
During subsequent rounds, the experts were provided
with anonymous data summarizing the responses of the
entire panel and were given the opportunity to revise
their ratings in light of the group judgment. The accuracy
of probability forecasts increases over Delphi rounds, up
2KOOCHER ET AL.
Downloaded by [DePaul University] at 13:08 25 April 2014
to the second round (Ascher, 1978; Martino, 1972), and
when statistical summaries are provided to the experts
(Rowe, Wright, & McColl, 2005).
Following the initial mailing and a subsequent
reminder to the 139 responders to round one, we received
67 responses to Round 2. The response rate to Round 1
was 93% (139=150) and the Round 2 response was 48%.
The experts were primarily child psychologists living
in the United States. Eighty-five percent described them-
selves as child=adolescent psychologists and 8% as child=
adolescent psychiatrists. More than half (53%) reported
earning board certification in their specialty, 65%
authored or edited a book in child mental health, and
65% served as an editorial board member of a peer-
reviewed journal in child psychology or psychiatry.
Approximately one third (37%) served as an editor or
associate editor of a peer-reviewed journal in child psy-
chology or psychiatry, and 55% currently or previously
held peer-reviewed grant funding in child psychology
or psychiatry. These categories are not mutually
exclusive, of course; many panelists fit more than one.
The average number of years of clinical experience was
26.3. Women accounted for 41% of respondents. In
terms of ethnicity, participants providing their race=
ethnicity characterized themselves as follows: 57 White=
Caucasian; three Black=African American; and one each
Native American, Hispanic, Asian American, and other.
Instructions to Experts
We presented the following instructions to the panelists
when they linked to the SurveyMonkey site:
For the purpose of this Delphi poll of experts, we oper-
ationally define discredited treatments and tests as those
unable to consistently generate treatment outcomes
(treatments) or valid assessment data (tests) beyond that
obtained by the passage of time alone, expectancy, base
rates, or credible placebo.
Our use of the term ‘‘discredited’’ subsumes ineffec-
tive and detrimental interventions but forms a broader
and more inclusive characterization. We are interested
in identifying disproven practices. Please rate the extent
to which you view the treatment or test as discredited
along a continuum from ‘‘not at all discredited’’ to
‘‘certainly discredited.’’
A treatment or assessment tool can be discredited
according to several types of evidence: peer-reviewed
controlled research, clinical practice, and=or professional
consensus. Please think in terms of the criteria for expert
opinions as delineated in well-known court decisions
such as Daubert v. Merrell Dow Pharmaceuticals
(1993) or Kumho Tire Co. v. Carmichael (1999). In these
cases the federal courts cited factors, such as experi-
mental testing, peer review, error rates, and acceptability
in the relevant scientific community, some or all of
which might prove helpful in determining the validity
of a particular scientific theory or technique.
We use a 5-point, Likert-type format with the follow-
ing ratings:
1- Not at all discredited, 2 - Not likely discredited, 3 -
Possibly discredited, 4 - Probably discredited, and 5-
Certainly discredited.
If you cannot make a rating because of unfamiliarity
with the treatment or test, then kindly check the not
familiar with treatment=test column. If you lack
familiarity with the treatment=test’s research or clinical
use, then kindly check the not familiar with research or
clinical use column. You may also mark both.
If experts indicated that they were unfamiliar with a
particular test or treatment, they could not numerically
rate it. In this way, ratings were contributed by only
those professionals who felt sufficiently cognizant of
the procedure and its evidence base.
RESULTS
Our Delphi poll results are summarized in Tables 1 and 2,
which display the results from both rounds for
treatments and assessments, respectively. The data in
the tables are ranked in descending order from those
regarded as most likely to least likely discredited in the
second round of ratings (i.e., from high to low in column
6). The tables display the mean ratings and standard
deviations of each item for both rounds, along with the
percentage of panelists indicating unfamiliarity with the
particular method.
As expected with consensus-building procedures, the
mean ratings in Round 2 tended toward less variability
than in Round 1. Only 18 of the standard deviations
for the 89 items (11 tests, seven treatments) evidenced
an increase from Round 1 to Round 2. The panelists
developed a greater consensus in their ratings on what
comprised discredited procedures.
Before proceeding to the treatment and assessment
methods judged by the panel as discredited, we should
note that several of those proposed as such in the
public literature or in private discussions did not merit
such condemnation according to the expert consensus.
We would characterize as not discredited those methods
receiving mean ratings in the second round between 1.0
and 2.5. Among the assessment methods were the
Balthazar Scale of Adaptive Behavior, Bender Visual
Motor Gestalt Test, Connor’s Symptom Checklist,
Devereaux Child Behavior Checklist, Finger Localiza-
tion Test, Jesness Inventory, Raven Standard Progressive
Matrices, Tactile Localization Test, Vineland Adaptive
Behavior Scales, and Wepman’s Auditory test for the
uses or purposes stated on the questionnaire. The expert
panelists were not necessarily recommending these tools
DISCREDITED CHILD ASSESSMENT AND TREATMENT METHODS 3
Downloaded by [DePaul University] at 13:08 25 April 2014
for their identified purpose or other purposes, but as of
2012 they did not consider them as discredited. Among
the psychological treatments, the not discredited
methods included Communication Cards to improve
social skills, Picture Exchange Communication, System
Self-Control Training for treatment of ADHD, and the
TEACCH approach for treatment of autism.
Table 1 presents the assessment tools and tests in
ranked order from the most to least discredited accord-
ing to the expert panel. Eleven earned mean ratings
above 4.25 on the 5-point scale (4 ¼probably discredited,
5¼certainly discredited). The expert consensus held that
enneagrams, Szondi Test, Brain Balance, biorhythms,
Hand Test, handwriting analysis, Animal Naming
Test, Fairy Tale Test, Blacky Test, IQ test scale scores,
and Holtzman Inkblot Test were discredited for
their purported assessment uses among children and
adolescents.
Forty-two of the 59 listed treatment methods received
average ratings above 4.25 in the second round. There
TABLE 1
Mean Discredited Ratings of Psychological Tests and Assessments Used with Children and Adolescents Ranked by Round 2 Mean Ratings
Test
Round 1 Round 2
MSD
%Not
Familiar
With
Technique
%Not
Familiar
With
Research M SD
%Not
Familiar
With
Technique
%Not
Familiar
With
Research
Szondi Test for Personality Assessment 4.80 0.40 56% 31% 4.83 0.82 47% 24%
Enneagrams Model of Personality Assessment 4.69 0.61 62% 28% 4.83 0.50 54% 35%
Brain Balance for Assessment of ADHD 4.38 0.74 54% 36% 4.81 0.47 38% 29%
Biorhythms for Personality Assessment 4.46 0.91 34% 26% 4.78 0.41 25% 21%
Hand Test for Personality Assessment 4.23 1.20 37% 37% 4.75 0.43 38% 29%
Handwriting Analysis for Personality Assessment 4.76 0.52 10% 13% 4.73 0.67 4% 10%
Animal Naming Test for Personality Assessment 4.50 0.80 42% 29% 4.59 0.74 21% 15%
Fairy Tales Test for Personality Assessment 4.65 0.68 55% 32% 4.55 0.81 40% 25%
IQ Scale Scores as Personality Assessment Tools 4.32 1.02 0% 8% 4.49 0.95 0% 8%
Blacky Test for Personality Assessment 4.46 0.82 44% 23% 4.47 0.93 31% 19%
Holtzman Inkblot Test for Personality Assessment 4.34 0.98 18% 23% 4.39 0.88 8% 12%
Machover Human Figure Test for Personality Assessment 4.04 1.10 38% 26% 4.22 0.92 39% 23%
Word Association Test for Personality Assessment 3.77 1.10 13% 26% 3.98 1.13 12% 17%
QEEG Brain Mapping for Diagnosing ADHD 3.70 1.23 49% 30% 3.96 1.07 31% 29%
Projective Storytelling Cards for Personality Assessment 3.63 1.20 14% 11% 3.80 0.99 13% 10%
Make A Picture Story Test for Personality Assessment 3.70 1.22 34% 22% 3.78 1.04 21% 13%
Tell Me A Story Test for Personality Assessment 3.60 1.25 38% 23% 3.76 1.30 33% 16%
House Tree Person Test for Personality Assessment 3.48 1.28 3% 8% 3.69 1.14 6% 8%
Kinetic Family Drawings for Personality Assessment 3.47 1.14 11% 13% 3.64 1.06 9% 9%
Rorschach Inkblots for Personality Assessment 3.64 1.18 7% 5% 3.55 1.27 0% 2%
Draw a Person Test for Personality Assessment 3.43 1.26 3% 3% 3.53 1.14 0% 1%
Thematic Apperception Tests for Personality Assessment 3.38 1.19 3% 3% 3.42 1.19 4% 2%
Robert’s Apperception Test for Personality Assessment 3.19 1.17 11% 13% 3.33 1.15 16% 16%
Children’s Apperception Test for Assessment of Personality 3.45 1.16 3% 6% 3.19 1.11 2% 6%
Anatomically Detailed Dolls for Use in Diagnosing Child Sexual
Abuse
3.13 1.32 3% 11% 3.12 1.08 0% 8%
Porteus Maze Test for Assessing Intelligence 3.18 1.27 31% 18% 2.65 1.05 17% 19%
Jesness Inventory for Predicting Later Delinquent Behavior 2.54 1.16 49% 41% 2.45 1.10 38% 21%
Balthazar Scale of Adaptive Behavior for Assessment of Self-Care
in the Developmentally Disabled
2.78 1.40 63% 35% 2.31 0.46 56% 37%
Tactile Localization Test for Neuropsychological Assessment 2.20 1.29 48% 31% 2.11 0.85 25% 15%
Bender Visual Motor Gestalt Test for Assessment of
Neuropsychological Impairment
2.34 1.05 4% 11% 2.11 0.63 8% 8%
Finger Localization Test in Neuropsychological Assessment 2.23 1.13 38% 38% 2.03 0.85 29% 25%
Raven Standard Progressive Matrices for Assessing Intelligence 1.93 0.75 12% 22% 1.98 0.90 8% 15%
Wepman’s Auditory test for Auditory Discrimination 1.95 1.28 41% 36% 1.85 1.01 35% 23%
Devereaux Child Behavior Checklist for Assessing ADHD 1.60 0.71 15% 13% 1.40 0.72 7% 9%
Connor’s Symptom Checklist for Diagnosing ADHD 1.38 0.52 2% 0% 1.37 0.80 2% 6%
Vineland Adaptive Behavior Scales for Assessment of Adaptive
Behavior
1.15 0.36 0% 3% 1.16 0.61 2% 0%
Note: 1¼not at all discredited,3¼possibly discredited,5¼certainly discredited. ADHD ¼attention deficit=hyperactivity disorder; QEEG ¼
quantitative electroencephalography.
4KOOCHER ET AL.
Downloaded by [DePaul University] at 13:08 25 April 2014
TABLE 2
Mean Discredited Ratings of Psychological Treatments Used With Children and Adolescents Ranked by Round 2 Mean Ratings
Treatment
Round 1 Round 2
MSD
%Not
Familiar
With
Technique
%Not
Familiar
With
Research M SD
%Not
Familiar
With
Technique
%Not
Familiar
With
Research
Magnet Therapy for Treatment of Child Psychopathology 4.65 0.73 25% 17% 4.97 0.18 27% 25%
Past Life Regression Therapy for Treatment of Child Psychopathology 4.91 0.29 8% 13% 4.96 0.20 6% 10%
Rebirthing Therapy for Treatment of Child Psychopathology 4.91 0.29 3% 11% 4.96 0.19 0% 4%
Crystal Healing for Treatment of Child Psychopathology 4.95 0.22 22% 20% 4.95 0.22 11% 19%
Bio-Ching for Treatment of Child Psychopathology 4.81 0.39 55% 39% 4.94 0.24 54% 42%
JoyTouch for Treatment of Child Psychopathology 4.85 0.36 58% 41% 4.94 0.24 56% 42%
Kirlian Therapy for Treatment of Child Psychopathology 4.69 0.46 57% 41% 4.94 0.23 29% 45%
Penduluming for Treatment of Child Psychopathology 4.88 0.32 47% 30% 4.94 0.23 48% 38%
Withholding Food=Water for Treatment of Child Psychopathology 4.98 0.14 13% 17% 4.93 0.26 15% 9%
Aura Therapy for Treatment of Child Psychopathology 4.81 0.46 36% 28% 4.92 0.27 40% 36%
Orgone Therapy for Treatment of Child Psychopathology 4.75 0.51 42% 33% 4.90 0.30 25% 31%
Astrotherapy for Treatment of Child Psychopathology 4.86 0.34 38% 32% 4.89 0.31 36% 38%
Conversion Therapy=Reparative Therapy for Adolescent
Homosexuality
4.87 0.44 9% 11% 4.89 0.38 4% 15%
Prism Glasses=Colored Glasses for Treatment of Autism 4.75 0.43 35% 27% 4.89 0.31 23% 13%
Triggering Anger Therapy for Treatment of Reactive Attachment
Disorder
4.65 0.79 48% 34% 4.88 0.32 46% 29%
Bach’s Flower Therapy for Treatment of Child Psychopathology 4.73 0.44 57% 20% 4.86 0.34 46% 42%
Tap Therapies for Treatment of Child Psychopathology 4.50 0.71 55% 35% 4.86 0.35 56% 44%
Bioenergetic Therapy for Treatment of Child Psychopathology 4.63 0.70 42% 38% 4.85 0.36 45% 39%
Energy Field Therapy for Treatment of Child Psychopathology 4.61 0.83 35% 31% 4.83 0.58 25% 29%
Irlen Lenses for Treatment of ‘‘Processing Problems’’ 4.68 0.55 47% 34% 4.81 0.39 40% 32%
Chelation Therapy for Uses Other Than Lead Poisoning 4.65 0.65 23% 20% 4.80 0.40 19% 21%
Angel Therapy for Treatment of Child Psychopathology 4.87 0.34 52% 45% 4.79 0.41 51% 49%
Hyerbaric Oxygen Treatment for Treatment of Child Psychopathology 4.62 0.69 27% 29% 4.77 0.49 23% 25%
Psychodynamic Psychotherapy for Treatment of Autism 4.67 0.77 0% 7% 4.76 0.55 0% 4%
Bettleheim’s Psychogenic Treatment for Autistic Children 4.71 0.54 18% 12% 4.74 0.54 15% 13%
Color Therapy for Treatment of Child Psychopathology 4.44 0.99 44% 25% 4.73 0.51 23% 33%
Aroma Therapy for Treatment of Child Psychopathology 4.63 0.72 5% 29% 4.68 0.51 8% 17%
Psychomotor Patterning for Treatment of Child Psychopathology 4.42 1.07 32% 25% 4.67 0.65 32% 26%
Qigong for Treatment of Child Psychopathology 4.39 1.25 51% 35% 4.67 0.47 53% 40%%
Chiropractic Skull Manipulation=Cranio-Sacral Therapy 4.65 0.63 17% 15% 4.65 0.74 8% 15%
Interactive Metronome Training for Treatment of ADHD 4.38 0.86 31% 30% 4.65 0.64 23% 21%
Holding Therapy for Treatment of Reactive Attachment Disorder 4.40 0.84 11% 11% 4.62 0.70 8% 11%
Attachment-Promoting Therapies that Use Holding for Treatment
of Child Psychopathology
4.38 0.91 2% 8% 4.60 0.60 2% 12%
Reiki for Treatment of Child Psychopathology 4.75 0.49 30% 24% 4.59 0.84 23% 26%
Safety Seeking Psychotherapy for Treatment of Substance Abuse 4.22 1.18 51% 35% 4.59 0.77 53% 45%
Equine Therapy as Stand-Alone Treatment for Treatment of Child
Psychopathology
4.40 1.20 14% 23% 4.55 0.80 8% 21%
Play Therapy for the Treatment of ADHD 4.18 1.13 0% 6% 4.51 0.86 0% 8%
Applied Kinesiology for Treatment of ADHD 4.37 0.72 21% 33% 4.50 0.59 11% 17%
Thoughtful Ed Therapy for Treatment of Child Psychopathology 4.00 1.36 57% 38% 4.50 0.63 60% 47%
Jungian Sand Tray Therapy for Treatment of Child Psychopathology 4.13 1.12 22% 27% 4.45 0.92 19% 15%
Vision Therapy or Vision Training for Treatment of Reading Problems 4.15 1.19 27% 29% 4.40 0.83 25% 17%
Behavioral Vision Therapy for Treatment of Reading Problems 4.19 1.11 32% 23% 4.32 0.87 23% 23%
Brain Balance for Treatment of ADHD 4.12 1.09 48% 18% 4.23 0.71 36% 30%
Brushing and Joint Compression Treatment for Autism 4.08 1.13 29% 20% 4.22 0.84 23% 21%
Auditory Integrative Therapy for Treatment of Autism 3.94 1.16 15% 16% 4.18 0.78 9% 11%
Music Therapy as Stand-Alone Treatment for Treatment of Child
Psychopathology
4.04 0.98 2% 11% 4.18 0.72 4% 17%
Art Therapy for TREATMENT of Childhood Schizophrenia 4.00 1.07 0% 15% 4.16 0.89 0% 15%
Facilitated Communication for Treatment of Autism 3.98 1.19 5% 12% 4.13 1.09 4% 9%
Dolphin Assisted Therapy for Treatment of Developmental Disorders 4.00 1.41 15% 26% 4.08 0.82 8% 26%
(Continued )
DISCREDITED CHILD ASSESSMENT AND TREATMENT METHODS 5
Downloaded by [DePaul University] at 13:08 25 April 2014
was greater unanimity in the proportion of candidates
treatments considered discredited than in the assess-
ments. Table 2 presents them in ranked order according
to the Round 2 mean rating. Those receiving the highest
discredited ratings (4.88 or higher) were magnet therapy,
rebirthing therapy, past life regression therapy, crystal
healing, Kirlian therapy, penduluming, Bio-Ching,
JoyTouch, withholding food=water, aura therapy,
Orgone Therapy, Astrotherapy, Conversion Therapy=
Reparative Therapy for adolescent homosexuality, and
Triggering Anger Therapy for treatment of reactive
attachment disorder.
DISCUSSION
We designed our research to identify a professional
consensus concerning discredited assessment and treat-
ment methods for youth. We conducted a Delphi poll
to secure such a consensus and to establish more refined
characterizations of treatments and tests.
The results do suggest a continuum from not discre-
dited to certainly discredited, according to our experts.
It is useful to identify and avoid those practices pro-
fessionally judged as ineffective, perhaps even detrimen-
tal; it is also useful to delineate those that are not. One
person’s opinion in a review article or book chapter does
not constitute a collective or definitive judgment, nor is
professional consensus a guarantee of truth. But insofar
as science is a process and product of collecting repli-
cated provisional ‘‘facts,’’ expert consensus is probably
superior to individuals’ judgment. Even experts can be
wrong, but less so than single practitioners and especially
those marketing mental health practices.
Despite the presence of a number of projective per-
sonality instruments in the ‘‘top eleven,’’ a number of
other more popular or better known projective tools—
such as the Children’s Apperception Test, Roberts
Apperception Test, and Rorschach Inkblot technique—
drew less harsh ratings. In addition, some objective or
inventory-style assessments, including the Connor’s
Symptom Checklist for diagnosing ADHD, the Dever-
eaux Child Behavior Checklist for assessing ADHD,
and the Vineland Adaptive Behaviors Scales, were rated
by the experts as ‘‘not likely discredited.’’ The instru-
ments faring worst tended to be those with obsolete or
sparse research and those relying on narrow theoretical
approaches (e.g., the Blacky Test, which relies on
Freud’s psychosexual theory of development for con-
struct validity) or those with highly suspect theoretical
underpinnings (e.g., the Szondi Test’s foundation in
‘‘hereditobiology’’).
TABLE 2
Continued
Treatment
Round 1 Round 2
MSD
%Not
Familiar
With
Technique
%Not
Familiar
With
Research M SD
%Not
Familiar
With
Technique
%Not
Familiar
With
Research
DARE Programs for Prevention of Substance Abuse=Dependence 3.93 1.13 3% 11% 4.00 0.92 6% 9%
Boot Camps for Treating Adolescents With Conduct Disorders 3.81 1.08 1% 16% 3.83 0.99 4% 11%
Sensory Integrative Treatment for ADHD 3.79 1.09 8% 16% 3.80 1.04 6% 10%
Brain Gym for Concentration and Memory Problems 3.62 0.96 42% 16% 3.77 0.89 36% 34%
Boot Camps for Treating Adolescents With Substance Abuse
Problems
3.82 1.03 2% 18% 3.76 0.98 6% 8%
Acupuncture for Treatment of Child Psychopathology 4.05 0.89 5% 35% 3.66 1.16 39% 23%
Sensory Integrative Treatment for Treatment of Autism 3.49 1.32 6% 18% 3.65 1.05 8% 13%
Scared Straight for Treating Adolescents With Conduct Disorders 4.19 0.90 3% 15% 3.64 0.88 2% 13%
Fast ForWord for Treatment of Children=Adolescents With
Memory or Attention
3.19 1.05 41% 38% 3.43 0.73 57% 43%
Floortime for Treatment of Autism 2.87 1.50 31% 28% 3.12 1.08 62% 23%
Neurofeedback for Treatment of ADHD 2.94 1.16 6% 16% 3.04 0.95 4% 15%
EMDR for Treatment of Trauma 3.08 1.11 6% 15% 3.02 1.28 4% 8%
EEG Biofeedback for Treatment of ADHD 2.98 1.13 5% 16% 3.00 0.96 0% 11%
Yoga for Treatment of Child Psychopathology 3.26 1.33 11% 33% 3.00 0.95 8% 26%
Picture Exchange Communication System for Treatment of Child
Psychopathology
2.41 1.52 34% 31% 2.38 1.45 30% 26%
Self-Control Training for Treatment of ADHD 2.29 1.05 13% 22% 2.34 0.74 9% 13%
Communication Cards to Improve Social Skills 2.61 1.29 20% 28% 2.25 0.86 13% 17%
TEACCH for Treatment of Autism 1.70 0.94 17% 17% 1.73 1.07 19% 15%
Note: 1¼not at all discredited,3¼possibly discredited,5¼certainly discredited. ADHD ¼attention deficit=hyperactivity disorder.
6KOOCHER ET AL.
Downloaded by [DePaul University] at 13:08 25 April 2014
Many of the treatment and assessment methods con-
demned as ‘‘discredited’ by the experts maintain current
adherents, as quick Internet searches will reveal. For
example, one can become ‘‘board certified’’ as a ‘‘past life
therapist’’ (http://www.ibrt.org/). At least one of the
techniques rated as likely discredited, rebirthing therapy,
has led to a documented patient death and a state law in
Colorado banning its use (Josefson, 2001).
When we urge fellow practitioners to refrain from using
or teaching the methods consensually judged as discre-
dited, we are frequently met with two immediate protests:
‘‘What do those experts know!’’ and ‘‘But it worked for
(me, my client, my aunt).’’ Regarding the former, we
remind protesters that dispassionate experts are imperfect,
but less imperfect than biased individuals, and that they
did excuse themselves from rating those methods with
which they were unfamiliar. Regarding the latter, we
acknowledge that all assessments and treatments will
indeed appear to work for some clients some of the time,
due to chance, time, accident, or placebo. The proper
comparison is to outperforming chance and placebo, not
whether a method happens to succeed on occasion.
The collective results remind us that ‘‘old (pro-
fessional) habits die hard.’’ Many practitioners adhere
to favored theories and treasured methods in which they
were originally trained in graduate school. Both histori-
cal analysis (Kuhn, 1962) and empirical research (e.g.,
Neimeyer, Taylor, & Rozensky, 2012) suggest that the
accelerating profusion of knowledge will probably
translate into shorter durability of current knowledge.
A recent Delphi poll indicated that the half-life of knowl-
edge in professional psychology is expected to decrease
within the next decade from nearly 9 years to just over
7 years (Neimeyer et al., 2012).
Readers should bear in mind both practical and con-
ceptual constraints when interpreting our results. On the
practical side, our panel consisted of psychotherapists
living and working in the United States; generalizations
regarding the perspectives of experts in other countries
are unwarranted. Second, our sample was largely
composed of seasoned, doctoral-level psychologists and
psychiatrists. Other professions or practitioners with dif-
ferent credentials may not share the same perspectives.
Third, the response rate to the first round was high
(139=150), but the response to the second round was less
so (67=139). We cannot rule out the potential of an
unknown response bias. Fourth, we acknowledge that
by not surveying experts in pseudoscientific interventions
per se, the conclusions reached in this study may
not reflect their particular consensus. It is possible that
some experts in child and adolescent psychology and
psychiatry know little about the ‘‘dark side’’ of their
profession. Finally, many of the items had an even lower
number of raters because the panelists could indicate
that they were unfamiliar with the method and thus
did not contribute to the mean rating. We cannot say
whether the experts’ lack of familiarity with some proce-
dures altered the final ratings, although one would expect
experts to possess familiarity with widely respected and
widely shunned practices as standard of care knowledge.
On the conceptual side, the experts’ ratings addressed
particular uses or purposes of the assessment or treat-
ment method. The validity is therefore conditional;
usefulness is purpose-and context-specific. A therapy
method considered discredited for youth might be con-
sidered more credible for another purpose or with a dif-
ferent population. The experts’ theoretical orientations,
which we did not assess, might also potentially impact
their ratings. One might reasonably suspect that, say, a
psychodynamic psychologist would respond more favor-
ably to the credibility of projective devices than, say, a
cognitive-behavioral psychologist. And these consensus
ratings may well change with the passage of time and
the publication of new research. Several of today’s
mainstream treatments and tests may be regarded as
discredited 30 years from now, and several of those
characterized as discredited in 2012 may emerge as
EBP within a decade. Psychological science should strive
to be vigilant and self-correcting.
Yet these results leave us feeling encouraged.
Psychology, in its scientific base, relies on evidence, and
the discipline is making progress in differentiating science
from pseudoscience, EBPs from discredited practices. We
ardently hope that our Delphi poll sparks a broader, over-
due discussion within the profession about discredited
practices in working with some of our most vulnerable
populations. The risk to patients and practitioners in
using discredited procedures is real; as Voltaire (1765)
wrote in Questions sur les miracles: ‘‘Those who make
you believe absurdities can make you commit atrocities.’’
It may prove as useful and easier to identify what does not
work for youth (as in this study) as it is to identify what
does work (as in the EBP compilations). In either case,
we can simultaneously avoid (consensually identified) dis-
credited practices to eradicate what does not work and use
(inclusively defined) EBPs to promote what does work.
ACKNOWLEDGMENTS
We express our gratitude to all the participants in
our Delphi poll and take pleasure in acknowledging
those who authorized us to share their names. They
include Thomas Achenbach, Anne Marie Albano, Cindy
Anderson, Barry Anton, Glen P. Aylward, Russell
Barkley, Jeffery E. Barnett, William Bernet, Steve Boggs,
Susan Campbell, Monit Cheung, Ann Davis, Andres De
Los Reyes, Dennis D. Drotar, Mina Dulcan, Sheila M.
Eyberg, Frank R. Ezzo, Kurt Freeman, Daniel Hiliker,
Yo Jackson, Daphne Keen, Kristin Kutash, John
DISCREDITED CHILD ASSESSMENT AND TREATMENT METHODS 7
Downloaded by [DePaul University] at 13:08 25 April 2014
Lavigne, Adam Lewin, Katherine A. Loveland,
Eric Mash, Elizabeth McQuaid, Thomas Ollendick,
Tonya Palermo, Brenda Payne, Mitch Prinstein, Cecil
Reynolds, Stephen Shirk, Jennifer Shroff Pendley,
Wendy Silverman, Douglas Tynan, Abby Wasserman,
Robert Weis, Linda Wilmhurst, and Keith W Yeates.
REFERENCES
American Psychiatric Association. (2006). Practice guidelines for the
treatment of psychiatric disorders: Compendium 2006. Retrieved from
http://psychiatryonline.org/content.aspx?bookid=28§ionid=
2021669
Ascher, W. (1978). Forecasting. Baltimore, MD: Johns Hopkins
University Press.
Carroll, R. T. (2003). The skeptic’s dictionary. New York, NY: Wiley.
Daubert v. Merrell-Dow Pharmaceuticals, Inc. 509 U.S. 579, 113 S. Ct.
2786 (1993).
Della Sala, S. (Ed.). (1999). Mind myths: Exploring popular assumptions
about the mind and brain. New York, NY: Wiley.
Eisner, D. A. (2000). The death of psychotherapy: From Freud to alien
abduction. Westport, CT: Praeger.
Gardner, M. (2000). Did Adam and Eve have navels? Debunking
pseudoscience. New York, NY: Norton.
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice
in clinical psychology: Child and adolescent disorders (Vol. 1).
New York, NY: Wiley.
Hunsley, J. J., Crabb, R., & Mash, E. J. (2004). Evidence-based clinical
assessment. The Clinical Psychologist,57(3), 25–32.
Hunsley, J. J., & Mash, E. J. (2005). Developing guidelines for evidence-
based assessment of adult disorders. Psychological Assessment,17,
251–323.
Jacobson, J. W., Foxx, R. M., & Mulick, J. A. (Eds.). (2005). Contro-
versial therapies for developmental disabilities: Fad, fashion, and
science in professional practice. Mahwah, NJ: Erlbaum.
Josefson, D. (2001). Rebirthing therapy banned after girl died in 70
minute struggle. British Medical Journal,322, 1014.
Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago, IL:
University of Chicago Press.
Kumho Tire Co. v. Carmichael. (97–1709), 526 U.S. 137 (1999).
131F.3d 1433, reversed.
LeCroy, C. W. (2008). Handbook of evidence-based treatment manuals
for children and adolescents (2nd ed.). New York, NY: Oxford.
Lilienfeld, S. O. (2007). Psychological treatments that cause harm.
Perspectives on Psychological Science,2, 57–70.
Lilienfeld, S. O., Lynn, S. J., & Lohr, J. M. (Eds.). (2003). Science
and pseudoscience in clinical psychology.NewYork,NY:
Guilford.
Martino, J. P. (1972). Technological forecasting for decision making.
New York, NY: American Elsevier.
Neimeyer, G. J., Taylor, J. M., & Rozensky, R. H. (2012). The dimin-
ishing durability of knowledge in professional psychology: A Delphi
Poll of specialties and proficiencies. Professional Psychology:
Research and Practice,43, 364–371.
Norcross, J. C., Koocher, G. P., Fala, N. C., & Wexler, H. K. (2010).
What doesn’t work? Expert consensus on discredited treatments
in the addictions. Journal of Addiction Medicine,4, 174–180.
doi:10.1097=ADM.0b013e3181c5f9db.
Norcross, J. C., Koocher, G. P., & Garofalo, G. P. (2006). Discredited
psychological treatments and tests: A Delphi poll. Professional
Psychology: Research and Practice,37, 515–522. doi:10.1037/0735-
7028.37.5.515.
Rowe, G., Wright, G., & McColl, A. (2005). Judgment change during
Delphi-like procedures: The role of majority influence, expertise, and
confidence. Technological Forecasting and Social Change,72, 377–399.
Rubin, A. (Ed.). (2011). Programs and interventions for maltreated
children and families at risk: Clinician’s guide to evidence-based
practice. New York, NY: Wiley.
Singer, M. T., & Lalich, J. (1996). ‘‘Crazy’’ therapies. San Francisco,
CA: Jossey-Bass.
Spirito, A., & Kazak, A. (2005). Effective and emerging treatments in
pediatric psychology. New York, NY: Oxford.
Voltaire, F. M. A. (1765). Questions sur les miracles. Retrieved from
http://www.quoteid.com/quotes/Voltaire/40982/
Weisz, J. R., & Kazdin, A. E. (Eds.). (2010). Evidence-based
psychotherapies for children and adolescents (2nd ed.). New York,
NY: Guilford.
8KOOCHER ET AL.
Downloaded by [DePaul University] at 13:08 25 April 2014
