Content uploaded by Donald Hatcher
Author content
All content in this area was uploaded by Donald Hatcher on Jul 27, 2018
Content may be subject to copyright.
INQUIRY
C
RITICAL
T
HINKING
A
CROSS
THE
D
ISCIPLINES
Summer 2010 Vol. XXV No. 2
The College of Arts and Sciences, the College of Business
Administration, the College of Humanities and Social Sciences,
the College of Education, and the College of Criminal Justice
Sam Houston State University
ISSN 1093-1082
977109 3 108003
2 2
gniknihT lacitirC gnihcaeT ni scigoL gnitcilfnoCzapraH maroY
?edutittA dna edutitpA fo noitseuQ A :gniknihT lacitirCdna oteiN .M anA
Carlos Saiz
Harikumar Sankaran and Implications for Critical Thinking Disposition:
ocixeM weN ni nemhserF morf ecnedivEcivejirtimiD aziraM
STCAT,riaF knarF TM : Developing a New Critical Thinking
tnemurtsnI tnemssessA,masheuM eirelaV
John Miller, and
Wendy McCoy Elliott
The College of Business Administration, the College of Criminal
Justice, the College of Humanities and Social Sciences,
the College of Education, the College of Fine Arts and Mass
Communication, College of Health Sciences, and
the College of Sciences
Sam Houston State University
FALL 2015 Vol. XXX No. 3
Sam Houston
State University
INQUIRY
CRITICAL
THINKING
ACROSS
THE
DISCIPLINES
Summer 2010 Vol. XXV No. 2
INQUIRY
C
RITICAL
T
HINKING
A
CROSS
THE
D
ISCIPLINES
Summer 2010 Vol. XXV No. 2
The College of Arts and Sciences, the College of Business
Administration, the College of Humanities and Social Sciences,
the College of Education, and the College of Criminal Justice
Sam Houston State University
ISSN 1093-1082
977109 3 108003
2 2
gniknihT lacitirC gnihcaeT ni scigoL gnitcilfnoCzapraH maroY
?edutittA dna edutitpA fo noitseuQ A :gniknihT lacitirCdna oteiN .M anA
Carlos Saiz
Harikumar Sankaran and Implications for Critical Thinking Disposition:
ocixeM weN ni nemhserF morf ecnedivEcivejirtimiD aziraM
STCAT,riaF knarF TM : Developing a New Critical Thinking
tnemurtsnI tnemssessA,masheuM eirelaV
John Miller, and
Wendy McCoy Elliott
INQUIRY
C
RITICAL
T
HINKING
A
CROSS
THE
D
ISCIPLINES
Summer 2010 Vol. XXV No. 2
The College of Arts and Sciences, the College of Business
Administration, the College of Humanities and Social Sciences,
the College of Education, and the College of Criminal Justice
Sam Houston State University
ISSN 1093-1082
9
77109 3 108003
2 2
gniknihT lacitirC gnihcaeT ni scigoL gnitcilfnoCzapraH maroY
?edutittA dna edutitpA fo noitseuQ A :gniknihT lacitirCdna oteiN .M anA
Carlos Saiz
Harikumar Sankaran and Implications for Critical Thinking Disposition:
ocixeM weN ni nemhserF morf ecnedivEcivejirtimiD aziraM
STCAT,riaF knarF TM : Developing a New Critical Thinking
tnemurtsnI tnemssessA,masheuM eirelaV
John Miller, and
Wendy McCoy Elliott
FALL 2015 Vol. XXX No. 3
From the Editor’s Desk ............................................................................................................. 3
Frank Fair
Critical Thinking Instruction: A Realistic Evaluation ........................................................... 4
Donald Hatcher
Are We Asking the Right Questions About Critical Thinking Assessment:
A Response to Hatcher ............................................................................................................. 20
David Wright
Effect Size and Critical Thinking Assessment: A Response to Wright .............................. 32
Donald Hatcher
Using the Critical Thinking Assessment Test (CAT) as a Model for Designing
Within-Course Assessments: Changing How Faculty Assess Student Learning ............... 38
Ada Haynes, Elizabeth Lisic,
Kevin Harris, Katie Lemming,
Kyle Shanks, and Barry Stein
Evidence-Based Critical Thinking Exercise:
Shaping Tomorrow’s Dental Choices ..................................................................................... 49
Pauletta G. Baughman,
Gustavo M. S. Oliveira,
Elizabeth M. Smigelski, and
Vida Vaughn
Critical thinking and the Pedagogy of Music Theory ........................................................... 59
Michael Lively
Instructions to Authors
INQUIRY: Critical Thinking Across the Disciplines is sponsored by the College of Educa-
tion, the College of Sciences, the College of Fine Arts and Mass Communication, the College
of Criminal Justice, the College of Humanities and Social Sciences, and the College of Business
Administration of Sam Houston State University. It is a forum for the discussion of issues related
to the theory, practice, and pedagogy of critical thinking across the disciplines, from precollege to
university settings.
INQUIRY: Critical Thinking Across the Disciplines invites submission of articles, responses,
reviews of the literature, notices, letters, and book reviews for consideration for publication. The
goal is to encourage an exchange of ideas about effective pedagogy in critical thinking instruction,
about methods of assessing critical thinking skills and dispositions, about systematic errors in our
thinking, about enhancing the quality of information on which we base decisions and inferences,
about common fallacies in argumentation, and about all other topics that are relevant to critical
thinking across the disciplines.
Given the diverse nature of the eld and of our readership, we ask that authors present early
in the manuscript the manner in which “critical thinking” is to be understood. Each submission
should have on a separate page the author’s name and complete contact information, including dis-
ciplinary specialty and institutional afliation. Each manuscript should be submitted in Microsoft
Word or as a pdf le as an email attachment. Send submissions to the Managing Editor, Frank
Fair, of the Department of Psychology and Philosophy, Box 2447, Sam Houston State University,
Huntsville, TX 77341-2447. The email address is criticalthinking@shsu.edu
Authors must follow the 6th edition of the Publication Manual of the American Psychologi-
cal Association. Manuscripts that fail to conform to APA style will not be considered for review.
By submitting a manuscript to INQUIRY an author is afrming that the manuscript has not been
published and is not under consideration for publication elsewhere. Authors are responsible for
obtaining and providing written permission from copyright owners for reprinting previously pub-
lished illustrations, tables, or lengthy quotes of 500 words or more. Authors are responsible for the
accuracy of the material in their manuscripts. Use of footnotes and endnotes should be limited as
much as possible.
All manuscripts must be accompanied by an abstract of 100-150 words on a separate page or
the manuscript will not be accepted for review. Include no more than three key words below the
abstract. All manuscript pages, including reference lists and tables, must be double-spaced. All
gures must be camera-ready.
INQUIRY: Critical Thinking Across the Disciplines is published three times a year in coop-
eration with the Philosophy Documentation Center. All correspondence regarding subscriptions,
renewals, change of address, back issues, sample copies, and related matters regarding the jour-
nal’s distribution should be addressed to: Philosophy Documentation Center, P.O. Box 7147,
Charlottesville, VA 22906-7147. Phone (800) 444-2419 (USA and Canada) or (804) 220-3300.
Fax: (804) 220-3301. URL: www.pdcnet.org.
For information about subscriptions, see the next to the last page of this issue.
Kevin Possin (Philosophy)
Winona State University
Maria Sanders (Philosophy and Law)
Plymouth State University
Michael Scriven (Psychology and Evaluation)
The Claremont Graduate School and
Western Michigan University
Harvey Siegel (Philosophy)
University of Miami
Keith Stanovich (Psychology)
University of Toronto
James Trel (Physics)
George Mason University
Mark Weinstein (Educational Foundations)
Montclair State University
Frank Zenker (Philosophy and Cognitive Science)
Lund University
INQUIRY: Critical Thinking Across the Disciplines ©
Sponsored by the
College of Business Administration, the College of Criminal Justice, the College of Humanities and
Social Sciences, the College of Education, the College of Fine Arts and Mass Communication, College of
Health Sciences, and the College of Sciences at Sam Houston State University
The Editorial Staff
Frank Fair, Managing Editor (Philosophy, SHSU)
Valerie Muehsam, Associate Editor (Business Analysis, SHSU)
Marcus Gillespie, Associate Editor (Geography, SHSU)
Jeff Anastasi, Associate Editor (Psychology, SHSU)
Mike Vaughn, Associate Editor (Criminal Justice, SHSU)
Daphne Johnson, Associate Editor (Curriculum and Instruction, SHSU)
Michael Henderson, Associate Editor (Art, SHSU)
Sheryl Murphy-Manley, Assosciate Editor (School of Music, SHSU)
Scott Miller, Associate Editor (Physics, SHSU)
Ben Hamby, Associate Editor (Philosophy, Coastal Carolina)
David Wright, Associate Editor (Philosophy, SHSU)
Kevin R. Anderson, Production Editor (Sam Houston Press & Copy Center, SHSU)
Janet Fair, Copy Editor (SAM Center, SHSU)
EDITORIAL ADVISORY BOARD
J. Anthony Blair (Philosophy)
University of Windsor
Linda Elder (Educational Psychology)
Foundation for Critical Thinking
Robert Ennis (Education and Philosophy)
University of Illinois
Daniel Fasko, Jr. (Educational Psychology)
Bowling Green State University
Alec Fisher (Philosophy)
University of East Anglia
Donald Hatcher (Philosophy)
Baker University
Ralph Johnson (Philosophy)
University of Windsor
Richard Mayer (Psychology)
UC Santa Barbara
Gerald Nosich (Philosophy)
Buffalo State College
Founding Editors and Layout and Cover Designers
Wendy Oxman, Robert Michael Esforme, Jeff Spradling, and Erin Back
Statements or opinions expressed in INQUIRY: Critical Thinking Across the Disciplines reect the views of the
author(s) and do not represent the policy of the College of Business Administration, the College of Criminal
Justice, the College of Humanities and Social Sciences, the College of Education, the College of Fine Arts and
Mass Communication, College of Health Sciences, or the College of Sciences at Sam Houston State University
unless so stated. INQUIRY: Critical Thinking Across the Disciplines has been indexed by the following services:
ERIC, Expanded Academic ASAP, Index Philosophicus, InfoTrac OneFile, International Bibliography of
Book Reviews of Scholarly Literature (IBR), International Bibliography of Periodical Literature (IBZ), MLA
International Bibliography, The Philosopher’s Index, PsycINFO.
©2012, INQUIRY: Critical Thinking Across the Disciplines. All rights reserved.
ISSN: 1093-1082 eISSN: 2153-9871 www.pdcnet.org/inquiryct
Printed in the United States of America
Sam Houston State University
Contents
From the Editor’s Desk ............................................................................................................. 3
Frank Fair
Critical Thinking Instruction: A Realistic Evaluation ........................................................... 4
Donald Hatcher
Are We Asking the Right Questions About Critical Thinking Assessment:
A Response to Hatcher ............................................................................................................. 20
David Wright
Effect Size and Critical Thinking Assessment: A Response to Wright .............................. 32
Donald Hatcher
Using the Critical Thinking Assessment Test (CAT) as a Model for Designing
Within-Course Assessments: Changing How Faculty Assess Student Learning ............... 38
Ada Haynes, Elizabeth Lisic,
Kevin Harris, Katie Lemming,
Kyle Shanks, and Barry Stein
Evidence-Based Critical Thinking Exercise:
Shaping Tomorrow’s Dental Choices ..................................................................................... 49
Pauletta G. Baughman,
Gustavo M. S. Oliveira,
Elizabeth M. Smigelski, and
Vida Vaughn
Critical thinking and the Pedagogy of Music Theory ........................................................... 59
Michael Lively
2INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Copyright Permission and Disclaimer
It is the policy of this journal to obtain and hold the copyrights for all articles published with the
exception of those articles or parts of articles in the public domain. Authors are allowed to make copies
of their articles for any scholarly purpose of their own. Other scholars are hereby given permission
to make single or multiple copies of single articles for personal scholarly use without explicit written
permission, providing that no such copies are sold or used for commercial purposes. It is the obligation
of any user to acknowledge sources explicitly and to obtain permission from the journal if extensive
use is to be made of materials or information contained in an article. Extensive use of copy privileges
in such a way would be unethical. These policies will be followed until further notice.
This journal is not responsible for the statements of any contributor. Statements or opinions
expressed in the journal reect the views of the author(s) and not the Editor, the Editorial Advisory
Board, or ad hoc reviewers. Publication should not be construed as endorsement.
Frank Fair, Managing Editor
FALL 2015, VOL. 30, NO. 3 3
From the Editor’s Desk
Frank Fair
In this issue we are fortunate to have a very thoughtful and thought-provoking
exchange about assessing the impact of critical thinking instruction. Many of us have been
pleased when whatever instrument we have used to do the assessment shows that the students
have made progress, but the question is seldom raised whether that progress is sufcient to
justify the resources it takes to achieve it. In the opening article Don Hatcher raises that issue
very pointedly, and, given the fact that Don had the experience of creating, managing, and
assessing the long-running program in critical thinking instruction at Baker University, his
concerns about this issue cannot be lightly dismissed. Hence, I invited my colleague, David
Wright, to pose some challenging comments to Don, with the understanding that Don would
have the last word. The result is one of those exchanges that—to my way of thinking—truly is a
model of those critical thinking skills and attitudes we try to instill in our students. But don’t take
my word for it, see for yourself.
The next article features an account of efforts to use one of the most widely
respected CT assessment instruments, the Critical thinking Assessment Test developed at
Tennessee Tech, as a tool to aid in “designing better course assessments to grade student work.”
The authors describe the development of “CAT Apps” and encourage institutions to use a
“community-based approach to CAT App development.” Thus the CAT becomes a tool that
promotes a faculty development focus on improving critical thinking skills.
The third piece in this issue takes us into the eld of evidence-based dentistry
(EBD). The authors describe an exercise that is to provide an occasion for rst-year dental
students to apply the principles of EBD and critical thinking to a simulated ofce sales call
by a dental pharmaceutical representative. Among other things, the students were expected to
use the PICO model (Patient, Intervention, Comparison, and Outcome) to formulate questions
that “facilitate the search for evidence.” The point of using the simulated ofce sales call as
a prompt for the exercise was, naturally, to expose the students to situations they will face in
their dental practices and to motivate them to apply the principles of EBD and critical thinking
that the American Dental Association’s Commission on Dental Accreditation (CODA) asserts
in its Standard 2-9 “graduates must be competent in the use of critical thinking and problem-
solving, including their use in the comprehensive care of patients, scientic inquiry and research
methodology.”
Finally, Michael Lively gives us an essay on “Critical Thinking and the Pedagogy
of Music Theory.” I am personally very glad that he chose to send us this essay because it makes
it clear one way that critical thinking is relevant to the arts and arts education. It helps the journal
live up to its claim to cover “critical thinking across the disciplines.
4INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Critical Thinking Instruction: A Realistic Evaluation
The Dream vs. Reality
Donald Hatcher
Baker University
Abstract
Since the 80s, educators have supported instruction in critical thinking (CT) as “an Educational
Ideal.” This should not be a surprise given some of the more common conceptions, e.g., Ennis’s
“reasonable reective thinking on what to believe or do,” or Siegel’s “being appropriately moved
by reasons,” as opposed to bias, emotion or wishful thinking. Who would want a doctor, lawyer,
or mechanic who could not skillfully evaluate arguments, causes, and cures? So, educators
endorsed the dream that, through proper CT instruction, students’ critical skills and “rational
passions” could be greatly improved. In spite of the dream’s appeal, the reality is, after 30+
years, there is little reason to think the dream resembles reality. After describing what I take to be
an adequate denition of CT, such a depressing conclusion will be supported by CT assessment
scores from across academe, the continued widespread disagreement among experts in nearly
all elds, including CT, and the abundant psychological research on rationality and decision
making. And nally, while the ideal extols the value of objectivity, I shall argue that bias may be
unavoidable because personal values play a vital role in the evaluation of many arguments.
Keywords: argument assessment and subjectivity, critical thinking, critical thinking assessment,
critical thinking denitions, critical thinking and human psychology
A little learning is a dangerous thing;
Drink deep or taste not the Pierian Spring:
–Alexander Pope
The Dream
In 1989, my colleague Anne Spencer
and I completed the rst edition of our critical
thinking text: Reasoning and Writing: From
Critical Thinking to Composition. For the
purposes of the text, with its focus on the
application of critical thinking (CT) skills to
writing clear, well- argued papers, we decided
on the following denition of CT: “Critical
thinking is thinking that attempts to arrive
at a judgment only after honestly evaluating
alternatives with respect to available
evidence and arguments” (p. 1). Our text was
consciously developed to prepare students
for being able to honestly evaluate alternative
beliefs, policies, programs, and theories, and
ultimately, to prepare them to write well-
argued position papers that included careful
consideration of alternative positions.
As we later surveyed the literature for
some of the more common denitions of CT,
we thought ours was consistent with many
of the other well-known conceptions and
could be easily defended. For example, we
thought “the honest evaluation of alternatives”
was consistent with Ennis’ (1987) “rational
reective thinking on what to believe and
do” because how could one decide what to
believe or do without honestly evaluating
the alternatives? It was also consistent with
Siegel’s conception of a critical thinker
as one who is “appropriately moved by
FALL 2015, VOL. 30, NO. 3 5
reasons” (1988, 1992), as well as McPeck’s
(1981) “reective skepticism,” and Lipman’s
(1991) “skillful, responsible thinking, that
facilitates good judgment.” We believed each
of these required “the honest evaluation of
alternatives with respect to available evidence
and arguments.” How could anyone claim to
be “appropriately moved by reasons” while
ignoring alternatives?
Furthermore, the denition seemed
to be consistent with the requirements
of a liberal education where students are
typically introduced to a variety of competing
perspectives from across disciplines and
asked to evaluate each and choose the most
reasonable one. Like Siegel’s view, it included
the essential notion that a critical thinker must
have a specic, albeit complex, character trait,
i.e., be an honest inquirer, and all that that
implies (Siegel, 1980, 1988). The denition
was also in line with Richard Paul’s important
distinction between “strong” and “weak sense
critical thinking” (1995), where the latter
does not involve an “honest evaluation of
alternatives.”
The denition avoided Scriven’s (1991)
criticism that, if a denition of CT is vague,
almost any course will satisfy a requirement. It
implies that only courses that involve students
in “the honest evaluation of alternatives”
will fulll a critical thinking requirement.
Hence, English composition courses or a
course in traditional formal logic will not do.
It also addressed Ralph Johnson’s concern
that CT be distinguished from other cognitive
activities like metacognition, creative thinking,
problem solving, or doing formal logic since
those activities need not involve “the honest
evaluation of alternatives” (Johnson, 1996).
The denition also gave teachers a
clear guideline to the sort of papers that would
count as critical papers. One could simply ask
“Do the assigned papers show that students
have honestly evaluated alternatives with
respect to available evidence and arguments?”
Finally, we thought that our denition was
consistent with the history of good thinking
as exemplied in the works of such thinkers
as Plato, Aristotle, Aquinas, Mill, and Russell.
For example, it seems pretty clear that many
of Plato’s dialogues are in fact “the honest
evaluation of alternatives,” often those held by
some sophist. Both Aristotle’s and Aquinas’
works often begin by surveying the alternative
answers given to a question, and then arguing
that one position is better than the others.
Mill’s (1859/1978) On Liberty is one long
argument for the social utility of the “honest
evaluation of alternatives,” and claimed that
education should focus on students learning
to defend their opinions against “the common
objections” (p. 34). One who “knows only his
side of the case knows little of that” (p. 35).
Russell’s (1959) commitment to the
scientic approach to philosophy is also a
good example of one who endorses the “honest
evaluation of alternative theories” with special
attention on empirical evidence. It is also
essentially the same as Popper’s (1965) and
others’ idea of “critical rationality” where each
alternative is rigorously tested in an attempt to
falsify the position or theory. In fact, a good
deal of the history of philosophy is attempts
by thoughtful philosophers to honestly
evaluate some alternative, e.g., Socrates on the
sophist’s ideas, Aristotle on Plato, Augustine
on the Manicheans, Aquinas on everyone but
Aristotle, Locke on Descartes, Kant on Hume,
Kierkegaard on Hegel, Marx on Hegel (and
everyone else), Ayer on Hegel, Popper on
Marx, and so on. Indeed, Richard Paul claimed
that it was reading Aquinas that inspired much
of his own thinking about CT (Paul, 1993, pp.
92-93). It may be for this reason that Siegel’s
and Paul’s seminal works on CT and the value
of centering education on a CT model was so
desirable to current philosophers. I know it was
to me.
In our text, we showed that the “honest
evaluation of alternatives’ required a few
skills and character traits. In line with Ralph
6INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Johnson’s work (2009), the skills included the
ability to recognize, clarify, understand and
evaluate arguments (both those for and against
a position), to envision the consequences of
a proposed position or policy, and the ability
to formulate and articulate judgments in clear
persuasive prose. The character traits focused
on what was required by an “honest evaluation
of alternatives.” They include a disposition
or willingness to consider all available
alternatives and their support, and present them
in a fair-minded way, a way that the persons
who hold the alternative position would accept.
Siegel called this the “critical spirit” (1980, p.
9). This means being as objective as possible
and not letting one’s personal preferences,
emotions or wishful thinking guide the inquiry.
Most importantly, people involved in honest
evaluations must be open to changing their
minds relative to new evidence and arguments.
To not be willing to do this would indicate
what Richard Paul called “weak sense critical
thinking” (1995, pp. 383-393).
With this denition as our guide and
with the help of two generous FIPSE Grants
($68,000 and $108,000), we began to work.
With great enthusiasm (perhaps “irrational
exuberance”), Anne Spencer and I recruited a
group of humanities faculty to help design a
two-semester course that integrated instruction
in critical thinking and written composition,
with the study of primary texts often taught
in humanities courses. The Baker faculty
agreed that the sequence would be required of
all freshmen. It ran from 1990 to 2009. (See
complete program descriptions in Hatcher
2006, 2009, 2011, 2013a, or 2013b.)
While our denition of CT still seems
eminently defensible, there is the troubling
question whether such a conception is
realizable as an educational ideal by most
students (Siegel, 1980, 1992). As Siegel
so nicely put it, “Critical thinkers must be
critical about critical thinking (1997, p.
73). So, if we are committed to honestly
evaluating claims with respect to available
evidence and arguments, what is the evidence
that most people, especially students given
CT instruction, are capable of the “honest
evaluation of alternatives”? What is the
evidence that students can identify, clarify,
and honestly evaluate arguments? I shall argue
that the evidence indicates that most students
cannot. The dream of CT as an educational
ideal, as worthy as it might be, is in reality
only a dream. This sad conclusion is based
on assessment scores on standardized tests
from a wide variety of scholarly sources,
the ongoing disagreements among experts,
and CT’s apparently faulty view of human
psychology, especially of the role that values
play in argument evaluation. Assuming my
arguments are sufcient to conclude that most
students are incapably of “honestly evaluating
alternatives,” what then should those who are
committed to teaching CT do? I will conclude
with a few suggestions.
The Reality
What the assessment data say
One reason to conclude that critical
thinking skills are not capable of being taught
in any signicant way is the pre to posttest
gains on standardized CT tests are minimal.
One would think that if instruction in the
CT skills needed for students to “honestly
evaluate alternatives” was truly effective, one
would see substantial pre to posttest gains
by students taking legitimate CT courses.
Analogously, at the beginning of the semester,
if one had a classroom full of students who did
not understand algebra taking two semesters
of algebra and gave them a standardized pre
and posttest, one would expect signicant
gains. The same is true of almost any subject.
However, if one looks at the reported effect-
size gains (i.e., percent of a standard deviation)
for students who either have taken CT courses
or over an entire college career, the gains are
anything but substantial. Hence, one might
conclude that most students are incapable
of mastering the skills and dispositions in
FALL 2015, VOL. 30, NO. 3 7
any signicant way needed for the honest
evaluations alternatives based on available
evidence and arguments. This, of course,
assumes that the standardized tests are
adequate measures of students’ CT abilities.
Each test’s manual makes a pretty good case
that each is reliable and valid.
Let’s look at the data from numerous
studies using standardized tests during a
college education (Arum and Roksa, 2011;
Bok, 2006; Hatcher, 2011; Pascarella and
Terenzini, 2005). If people think that CT is
being taught across the curriculum, Arum
and Roksa report, using the College Learning
Assessment (CLA) test, that students gained
only a pathetic gain of 0.18 of a standard
deviation in their rst two years of college.
(It should be allowed that the CLA test is not
without some serious criticism. See Possin
(2013) for a pointed critique.)
In their inuential How College
Affects Students, Vol.2 (2005), Pascarella
and Terenzini surveyed a wide variety
of assessment reports, using a variety of
standardized assessment instruments, with
most of their data coming from the California
Critical Thinking Skills Test (CCTST)
(Facione P. & Facione, N. C., 1994). According
to their research, the effect size gains from
freshman to senior year were estimated at 0.55
to 0.65, with freshman to sophomore gains of
only 0.34 (p. 157). To put this in perspective,
the CCTST consists of thirty-four multiple
choice questions and, according to the test
booklet, past students had a standard deviation
of 4.51 (Facione, P. & Facione, N. C., 1994).
An effect size gain of 0.50 means that after
either four years of college and perhaps a
course in CT, the average student score was
only 2.25 points better than what he or she
scored on the pre-test. This, I believe, is pretty
underwhelming.
My own data from eighteen years of
continuous assessment at Baker University
using pre and posttests of either the Ennis–
Weir Critical Thinking Essay Test (E-W), The
California Critical Thinking Skills Test, Form
A (CCTST), and the Cornel Level Z Critical
Thinking Test (CLZ) showed a variety of
effect size gains ranging from 0.97 to 0.57,
depending on the test used (Hatcher, 2013b,
2011, 2009, 2006). While these gains are better
than some of those reported in the literature,
one can hardly say they indicate great success
with respect to students learning CT skills. For
example, for the students at Baker University,
a 0.57 effect size gain on the CLZ means that
after a two-semester sequence integrating CT
and written composition with the study of
primary texts, the average student gained only
2.7 points on a 52-question test! Again, if this
sort of gain were in some other subject such
as English composition or algebra, we would
be very skeptical of any claims that students
were at a minimum level mastering writing or
mathematical skills in those areas. As I looked
more carefully at the pre and posttest scores,
it turned out that 30% of Baker students did
worse on the CLZ posttest than they did on the
pretest. Not a good sign for CT instruction in
a program that had weekly staff meetings to
go over the material for the following week,
summer faculty workshops, a teacher’s guide
covering each chapter of Reasoning and
Writing, and an additional $800,000 in grant
support to help fund the program (Hatcher,
2013b).
Another disturbing bit of data is
the actual scores on the assessment tests.
In the case of Baker University, after the
two-semester course integrating CT and
written composition, the average score on the
52-question CLZ test was 29.1. That is less
than 60%, a failing grade. Earlier, we used the
E-W. Again, after our two-semester CT and
composition course, the mean score was 12.8
out of a possible 29 or 44%—another failing
grade. For eight years, we used the CCTST,
and the posttest mean was 18 out of a possible
34 or 53%—again another failing grade.
To put this in perspective relative to other
8INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
standardized tests, from 1990 to 1998, we
gave pre and posttests of the Test of Standard
Written English (at one time the language
component of the SAT) to our freshmen in
the CT and composition sequence. The mean
posttest score for 1359 freshmen was 49.1 out
of 60, or 82%, much higher than the CT scores.
Solon reported posttest scores on the CLZ
of 30.28 out of a possible 52 or 58% —close
but still failing. The only place I have seen
posttest scores reported above 60% was Tim
van Gelder’s computer-assisted CT courses at
the University of Melbourne (van Gelder &
Bissett, 2004). The post-test mean scores on
the CCTST were 22.4 out of 34 or 66% — a
solid grade of D. Yet even this score is hardly
something to get excited about.
Someone might say that the tests are
intentionally difcult and students are not
intended to score 80 to 100%. However, one
year I gave the CCTST to my teaching staff,
people from elds across the humanities, and,
assuming the teachers were honest, no one
missed more than two questions. I should add
that this staff consisted, for the most part, of
the same humanities faculty who had worked
on designing the courses and editing Anne
Spencer’s and my text. So, they were very
familiar with the variety of skills the CCTST
was testing. Consequently, I do not think
the CCTST test is unreasonably difcult—
even though some of its questions have been
criticized (Groarke, 2009).
Others might say that the problem
is the departmentalization of CT instruction
(Bailin, 1999; Ennis, 2013; and Nosich,
2012, 2010). The problem is that attention to
understanding and evaluating arguments is not
taught across the curriculum, and CT skills are
taught in philosophy classes. However, in spite
of the appeal of this arrangement, Ortiz (2007)
has shown that most philosophy classes to not
enhance students’ CT skills. Hitchcock (2004)
also cited plenty of data the show normal
philosophy classes do not enhance students CT
skills. I have argued, based on my experience
with the Baker CT sequence, that most
teachers are incapable of teaching the typical
CT skills (2013a). In addition, in my research
with that program, student effect size gains
from freshman posttests to the senior year
were minimal. Using the CCTST, the freshman
posttests mean was 18 out of 34 (n=1447)
and the senior mean was 19.3 (n=396), a gain
of 1.3 points on a 34 question test (Hatcher,
2011). This is in a program where there was
signicant grant funding to pay large numbers
of the faculty to take summer workshops
where Anne Spencer and I worked through our
critical thinking text that all freshmen were
studying. We tried not only to educate the
majority of the faculty, but encourage them
to make assignments that asked students to
use the CT skills they were supposed to have
learned as freshmen-another failed attempt. As
Plato said somewhere, not every cloth dipped
takes the dye.
So, given the evidence from the
assessment data, I think it is fair to conclude
that either the various current approaches to
teaching CT are misguided and ineffective or
that most students are incapable of learning
the skills. The former seems unlikely because
the scores are from people using a wide
variety of approaches: focus on fallacies and
informal logic (Johnson & Blair), deductive
reconstruction (Hatcher and Cederblom),
computer-assisted approaches (Hitchcock and
van Gelder), and approaches that integrate CT
into standard courses like General Psychology
(Solon). In theory, CT instruction sounds
like a terric educational ideal. However, the
data from the standardized tests show that
the likelihood that most students are actually
learning these skills needed for effective CT in
any substantial way is minimal.
The argument from persistent disagreement
“Tis with our judgments as our watches; none
Go just alike, yet each believes his own”
–Alexander Pope
FALL 2015, VOL. 30, NO. 3 9
Another thing that should have made
those of us who were so optimistic over the
promise of CT at least somewhat skeptical
about the ability of students to become “honest
evaluators of alternative positions” is the fact
that, even when confronted with the same
evidence and arguments, experts in many
disciplines continued to disagree (Feldman and
Wareld, 2010). This is the case not only with
philosophers, who are renowned for not being
able to agree on much of anything (Goldstein,
2015, p. 48), but experts in general (Elgin,
2010). This suggests the obvious fact that
the dream of even well-educated, intelligent
people objectively evaluating evidence and
arguments and then agreeing on their merit
is just that: a dream. The argument is simple
enough:
If the ability to evaluate alternative
positions and arguments objectively
was possible, then experts (especially
experts) who looked at the same
evidence and arguments would agree.
However, we know that experts
can look at the same evidence and
arguments and not agree. Hence,
the honest or objective evaluations
of alternative positions may not be
possible.
When it comes to critical judgment, something
besides the impartial appeal to evidence and
logic is going on.
Unfortunately, the fact that experts
disagree also undermines one of the main
arguments given for the value of CT in higher
education (Hatcher & Spencer, 2006, p.
3). Typically, college students are required
to take courses in subjects where there are
many disagreements. The current debates
over the proper methods in economics or
psychology are examples. Confronted with
such disagreements, students can either
become skeptics and say what one believes
is only a matter of personal opinion or, so
the theory goes, they can develop the critical
skills necessary to choose the most reasonable
position among competing theories studied.
Of course, we do not want students to go
away thinking that what people believe is
only a matter of opinion or simply personal
commitment. We would like them to believe
that there are good reasons for forming beliefs.
Otherwise, why become educated? So, we
need to do our best to equip students with the
appropriate CT skills and dispositions. On its
face, this argument should support requiring a
legitimate CT course or two of all students.
It now seems obvious that if experts in
various disciplines continue to have genuine
disagreements, students with a one or two
semesters of instruction in critical thinking
will not be in a place to make an unbiased
rational choice among competing claims
and theories. The fact of deep disagreements
among experts is another indication that
critical thinking skills are not tools capable of
objectively settling most disagreements. I say
most disagreements because if one pointed
out that someone’s position was supported by
absolutely irrelevant reasons and the person
understood this, then it would seem likely that
such a critique might make the person at least
wonder about continuing to hold the belief. To
believe that CT skills will lead rational people
to agree implies that the experts who continue
to disagree either do not possess the needed CT
skills and or their dispositions prevent them
from honestly evaluating competing positions.
Those who disagree with us are written off, as
one friend likes to say, as “Just whackos!” But
these people are sometimes brilliant scholars
and thinkers. Why should one think that they
are not good critical thinkers? That seems
highly unlikely, and yet disagreement is the
persistent norm (Elgin, 2010).
To make things worse, this problem
of disagreement among experts is especially
present in the eld of critical thinking and
argumentation. If being procient in the skills
and dispositions normally associated with CT
was sufcient to settle disagreements, then
10 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
critical thinkers would, more than people in
any other eld, be in agreement. However, that
is hardly the case. To share just one example,
some twenty years ago, my wife accompanied
me to one of Richard Paul’s terric annual
CT Conferences at Sonoma State University.
By then, I was fortunate to be friends with
quite a few of the writers and scholars in
the “Critical Thinking Movement.” For four
days, she quietly observed heated discussions
on the nature of CT, on what constituted the
most desirable CT pedagogy, or on the nature
of human rationality. These discussions were
among some of the well-known scholars
and textbook writers: people like Harvey
Siegel, Ralph Johnson, Mark Weinstein, Jerry
Cederblom, Ed Damer, Bill Dorman, Sharon
Bailin, Mark Battersby, Ian Wright, Carol
LaBar, Connie Missimer, Richard Paul, and
others. (See the works cited pages for some
of the many books and articles by most of
these people.) At the end of the conference,
my wife said that the conference was very
interesting, but then pointed out the obvious:
“Here is a group of experts in your eld, most
who are authors of scholarly books, journal
articles, and popular CT textbooks. But, after
four days of discussing competing claims, not
one person changed his or her mind. Surely, if
critical thinking is what you have said it is, i.e.,
the honest evaluation of alternative positions,
such experts would change their minds after
understanding and carefully evaluating the
competing arguments for the various positions.
But they do not. What conclusion should I
draw?” So much the worse for CT and the
ability of those who possess those skills to
honestly evaluate competing positions.
I have never been able to answer
my wife’s challenge, and now I take it very
seriously. If the “honest,” that is objective,
evaluation of alternative positions were
possible, surely people who teach CT should
be able to change their minds easily relative
to new arguments and evidence. But, they do
not. (I should point out as an exception to this
general rule that Bob Ennis claims that Gerald
Nosich convinced him to add “and do” to his
denition of critical thinking.) More on an
explanation of this persistent disagreement
later.)
Where did CT theorists go wrong? The
Relevance of Psychological Research
In spite of what now appears to be
our “irrational exuberance,” one problem we
ignored was a lot of psychological research
on how people actually respond to arguments
or on the relative force of rational argument
in decision-making for most people. When
Harvey Siegel says a critical thinker is a person
“appropriately moved by reason” (1988, p. 2),
he is assuming, like Plato, that there are other
forces in the human psyche and that reason
should and can over-ride and control them.
That is the heart of the Platonic education of
the rulers in Books IV to VII in The Republic.
The critical thinking theorists are to that extent
Platonists. We believed that, given a proper
training in the skills and dispositions needed
for effective CT, reason would ultimately win.
However, no matter how appealing
this vision of the human condition is, there
is a good deal of evidence by psychologists,
neurologists, and historians of science against
it (Ariely, 2008; Bishop & Trout, 2005; Burton,
2008; Damasio, 1994; Elgin, 2010; Haidt,
2012; Kuhn, 1962; Marcus, 2008 ; Tavris &
Aronson, 2007). The list could go on and
on. What is compelling about these books is
that most cite scores and scores of scientic
studies in well-regarded, refereed journals by
neurologists and psychologists indicating that
reason is not the powerful objective tool that
philosophers in the Platonic to Kant tradition
had typically thought.
The conclusion by most of these is
that even if students are taught to identify all
of the logical fallacies and how to avoid them
or how to construct valid arguments with
reasonable premises, that does not mean they
FALL 2015, VOL. 30, NO. 3 11
will use these skills when it is appropriate
to do so. According to these researchers,
passion, instinct, and intuition are the primary
motivators. Haidt, for example, uses the
metaphor of the elephant and the rider, where
emotion, intuition, etc. are the elephant, while
reason is the rider and “the rider evolved to
serve the elephant” (p. 49). So “The elephant
is far more powerful than the rider” (p. 68).
According to Haidt’s research, the most
common time that people change their minds
in response to an argument is “…when there is
affection, admiration, or a desire to please the
other person, then the elephant leans toward
that person and the rider tries to nd the truth
in the other person’s arguments” (Haidt, 2012,
p. 68). Hence, change in belief is a function of
the social setting and the afnity for the person
who disagrees, not the objective evaluation
of the opposing arguments. That is not to say
the rider cannot at times override the elephant,
but that is rare (Haidt, 2012, p. 68). However,
for the most part, “Automatic processes
run the human mind, just as they have been
running animal minds for 500 million years…
When human beings evolved the capacity for
language and reasoning at some point in the
last million years, the brain did not rewire
itself to hand over the reins to a new and
inexperienced charioteer” (Haidt, 2012, pp.
45-46).
To give another example, in Mistakes
Were Made, But Not By Me (2007), Tavris and
Aronson spend chapter after chapter showing
how people deal with cognitive dissonance.
They do so not by thinking rationally about
what might be the most reasonable response,
but by creating scenarios or causes of the
discomfort that make them feel better. For
example, college students who have done well
in high school believe they are pretty bright.
However, they have just done badly on a test.
How is that possible? Rather than concluding
that they may not be as bright as their high
school grades indicate or that they should have
studied more, they conclude that the teacher is
not a good teacher. Reason is not the objective
evaluator or seeker of truth we would like
to think. Of course, this makes “the honest
evaluation of alternatives” nearly impossible.
Some studies even show that emotion
is a necessary condition for decision-making.
For example, after studying the case of Phineas
Gage and others who lost the part of the brain
responsible for emotion, Damasio (1994) and
others point out that without the emotional
part of the brain, people cannot even make a
decision (pp. 3-33). What should we conclude
about the possibility of an “honest evaluation
of alternatives”? Where does that leave the
goal of objective judgment?
To say that CT theorists have
overestimated the power of objective
rationality and human reason as a guide to
making intelligent decisions not only means
that humans are prone to making faulty
inferences. The problem, according to the
research by numerous psychologists, is more
fundamental than that. In fact, it may be
entirely possible to teach students to recognize
and perhaps even avoid faulty logic. Hasty
generalizations, circular reasoning, false cause
or slippery slope are not difcult to recognize.
But the problem is, from Plato to Kant,
philosophers have correctly recognized that
there are indeed competing forces in the human
psyche, e.g., reason vs. emotion, but concluded
incorrectly that reason is most powerful one.
However, the research by the neurologists
and psychologists indicates that the strongest
and primary inclination is to follow instincts,
intuitions, and passions that lie in the animal or
pre-conscious parts of our brains. These may
or may not align with critical reason. Hume
may well have been right, and reason may
primarily be the slave of the passions.
Recognition of this explains a lot in
human behavior and why pre to posttest CT
test gains are so paltry. Again, at least ideally,
critical thinking involves the honest evaluation
of alternatives, and honesty requires the
12 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
unbiased, objective evaluation of competing
ideas and arguments. However, given the
research, it is easy to see that subjective
elements or values will be present in most
argument evaluation. Consider Nosich’s
prescription for argument evaluation in his
Reasons and Arguments (1982). According
to Nosich, we should evaluate the reasons
for a position by treating the conclusion and
reasons as an enthymeme, and then adding
the major premise to turn the argument into
a valid deductive argument, i.e. we apply the
technique of Deductive Reconstruction. (Please
note that Nosich has since modied this
strategy (2010; 2012) to be more inclusive of
disciplines beyond philosophy. However, this
move may bring up what Ralph Johnson calls
“the network problem” (2012). That is, it does
not distinguish CT from other kinds of thinking
like problem solving or creative thinking.)
The method of Deductive
Reconstruction is summarized by Nosich as
follows:
Step 1. Paraphrase the argument so that
you are sure you understand it.
Step 2. Break the argument down into
premises and conclusion.
Step 3. Arrange the premises and
conclusion in their logical order.
Step 4. Fill in the missing premises
needed to make the argument valid.
Step 5. Criticize the argument for
validity and the premises for truth.
(Nosich, 1982, p. 142)
Notice that it is possible (perhaps common)
to follow these steps and for two people to
disagree on the merit of the argument. Why
is this? The answer does not need to be
anything as complicated as Kuhn’s problematic
treatment of paradigms determining what
counts as a good reason (Kuhn, 1962). (See
Siegel (1987) for an excellent discussion of
the problems with Kuhn’s position.) When the
reasons in support of some contested position
are listed, the preference for one position over
another can be understood as a function of the
value or weight the persons in the dispute place
on the reasons they claim support their position
(Hatcher, 2003).
There is plenty of research to show
that the ranking or weighing is a matter of
emotion rather than reason (Burton, 2008;
Damasio, 1994; Haidt, 2012). For example,
in the debate in the U.S. over gun ownership,
for some people, their handgun collection
is one of the most important things in their
lives. “There should be a right to bear arms
because I really like my handgun collection!”
So, while the fact that there are thousands
of handgun deaths each year in the U.S. is
recognized as a relevant reason for banning
handguns, it does not carry as much weight
relative to other reasons with those who dearly
love their handgun collections. In addition,
when trying to understand current deep
disagreements on important social issues,
it is helpful to recognize that people might
share the same values, e.g. freedom, security,
equality before the law, etc., but rank or weight
them differently. The real issue is not so much
having different values, but their weighting.
For some, a troubling consequence
of this is there can be strong arguments with
reasonable —at least to each of the participants
—premises both for and against a position.
That is, the model seems to allow that both A
and not-A can be reasonable positions. This
is because of the different weights different
people give to the reasons, which in the end
may be more a product of emotion than reason.
However, rather than be seen as a problem,
this consequence may be the key to why
there are “deep disagreements” in so many
areas and why even the “honest evaluation of
alternatives” does not settle the disagreement.
Implications: What We Should Not Do
FALL 2015, VOL. 30, NO. 3 13
Given these arguments for the relative
ineffectiveness of critical thinking instruction,
what should we, as people who once believed
in the great promise of CT instruction, do?
First, there are a couple of things we should
not do.
I have to not claimed that the poor
performance on standardized CT tests is a
function of poor teaching, with professors
not really understanding what CT is or how
to teach it. While there is evidence that there
are plenty of professors who may lack such
understanding (Paul, Elder, Bartell, 1995), the
data reported in this paper are from some of the
best-known researchers, scholars, and teachers
in the eld (e.g., Arum and Roska, 2011;
Facione and Facione, 1994; Hitchcock, 2004;
Pascarella & Terenzini, 2005; Solon, 2003; Van
Gelder & Bissett, 2004). It would indeed be
different if the effect size scores were recorded
in classes that were pseudo-CT courses: those
courses that Michael Scriven once described as
“The Prostitution of Critical Thinking” (1991;
1992), courses that purport to teach CT, but
in fact include none of the skills needed to be
a critical thinker or to do well on any of the
standardized tests (Hatcher, 2013b). But here
the effect-size numbers being cited are from
legitimate attempts to teach the logical skills
needed for effective CT. While each of these
teachers might employ different methods, they
are all trying to teach the necessary skills.
Second, it would be wrong to blame the
paltry effect-size gains on professors simply
using the wrong techniques in their attempts
to teach CT. Again, the scores come from a
wide variety of approaches: computer-assisted
instruction (Hitchcock, 2004; Van Gelder
and Bissett (2004), deductive reconstruction
(Cederblom & Paulsen, 2006; Hatcher, 1999a,
1999b, 2001, 2006, 2009, 2011, 2013a,
2013b; Solon, 2003); and informal approaches
(Johnson & Blair, 1994a; Damer, 2001). While
there is some difference in gains, no one
approach has proved to be signicantly better
than others (Hatcher, 2013b). This in itself
is important because there is a tendency for
teachers to be unduly critical of approaches
to CT that are dissimilar to their own. For
example, even if Kahane’s famous account
of how teaching quantication rules was not
helpful to students in reasoning about practical
issues (Kahane, 1971, p. vii) is correct, it does
not follow, as some have claimed (Govier,
1999; Johnson; 2012), that those who use some
formal logic in their CT classes are wrong in
doing so (Hatcher, 1999a, 2013b). My survey
of the assessment literature indicates that
no legitimate approach to teaching students
how to evaluate arguments and evidence is
signicantly better or worse than any other.
Implications: So, what should we do?
If the test scores are low, if
disagreement among CT experts is often the
rule rather than the exception (Feldman &
Wareld, 2010), and if the psychological
research indicates that reason does not play the
dominant role in decision making that those in
the CT Movement thought possible, then what
should we do? Should we abandon CT as an
educational ideal?
Even if it is highly unlikely that most
students will be able to honestly evaluate
alternatives that does not entail that nothing
positive comes from a course in CT. Even
if the only thing students learn is how to
recognize really bad reasoning, e.g., circular
argumentation, slippery slopes, ad hominems,
or false cause arguments, that is a positive
outcome. Even if the odds of changing
one’s mind by considering alternatives and
evaluating their support are slim to none, there
is value in recognizing that, if one’s belief is
not universally shared, then there are almost
always relevant reasons and strong arguments
in opposition to one’s favored positions. That
is, those who disagree with us need not be
seen as close-minded bigots or “irrational
whackos.” A certain respect for the beliefs of
others, or what Siegel calls fallibilism (1988,
p.144; 1997), should come from recognizing
14 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
there are good reasons for disagreements.
For example, if a believer understands the
arguments that atheists have against religious
beliefs, the believer may not agree with those
arguments, but the atheist will not be simply
dismissed as in “indel.”
Siegel has also pointed out that CT as
an educational ideal can be justied on both
practical and moral grounds (1992, pp.100-
101). So, even if students’ CT skills and
dispositions continue to be weak, students
who are given CT instruction are still better
able to protect themselves from fallacious
reasoning than students who have no idea
how to distinguish good from bad arguments.
On an even more practical level, in 2012,
Forbes magazine listed CT as the number one
skill employers were looking for (Casserly).
Second, education that aims at creating
critical thinkers is one with the goal of helping
students become free, autonomous thinkers.
That is an easily defensible moral ideal. The
contrary to an education that focuses on CT is
indoctrination which, if rational freedom is the
goal, is the opposite of that goal.
Also, there is nothing wrong with
forcing students in their papers to state clearly
and simply their positions and the supporting
reasons. They will soon see that not all reasons
are relevant to their positions. Even if the odds
of changing a mind or belief are slim, clarity,
like Kant’s good will, is unqualiedly good,
and like Spinoza’s things of excellence, is
“as difcult as it is rare.” The reason clarity
is absolutely essential is that one cannot even
begin the process of critically evaluating
alternatives if one does not rst understand
them, and one cannot understand what is not
clear.
I would also suggest that we teachers
of CT be more modest in our expectations.
Often inspired by the work of Richard Paul
and Harvey Siegel, we could not help being
overly enthusiastic about the prospects of
teaching CT to as many students as possible.
Scriven even describes Paul as “one of the
most inuential evangelists of the Critical
Thinking Movement” (Paul, 2011, p. 22).
Some might have imagined we were laborers
working hard for converts in “The Church of
Reason.” But, in spite of the evidence, there
is nothing wrong with admitting that people’s
emotions or values play an important role in
human decision making. As Bacon pointed
out in his “Four Idols” (Hatcher & Spencer,
pp. 318-324), if we are more acutely aware of
tendencies that can interfere with objectivity
and honest inquiry, then we should be inclined
to be more reective before forming a belief or
committing to a decision. We and our students
might be more inclined to be “fallibilists”
rather than “true believers,” and that would
be a good thing. As W.K. Clifford (2005) has
pointed out, once a rm belief is formed, it
inuences our future willingness to inquire,
limiting our inclination to honestly evaluate
alternatives.
Finally, even though 30% of students
do worse on the post-tests, 70% do better,
and some students do much better. In fact,
in examining three years of pre and posttest
data for the CLZ, 58 out of 534 students, that
is 11%, improved by 9 or more points. That
is an improvement of two or more standard
deviations, an indisputably signicant effect
size of 2.0. So, the effects of CT instruction
on some students are profound. For example,
while these are anecdotal, here is an email
from one of my former students who had taken
the GRE prior to applying for graduate school,
one of the 11%: “I just found out that I got
a 6.0 (on a six-point scale) on the analytical
section of the GRE. Which is pretty exciting,
and says good things about philosophy majors
and the LA program, I think. I mean, we don’t
spend very much time on critical thinking,
argument analysis, logical fallacies, etc.
(which is basically what the analytical section
was about) in any of my other classes” (M.
Ireland, personal communication December
FALL 2015, VOL. 30, NO. 3 15
17, 2005. The LA Program is the BU Critical
Thinking and Composition sequence.). The
student is now an Assistant Professor of
Psychology at a large university. Here is
another from a graduate who is a practicing
attorney: “There’s absolutely no way I’d be
where I am without the LA series and my
philosophy degree…. I’m not exaggerating
when I say it changed the whole trajectory of
my life: it improved my LSAT score, helped
me get a full scholarship to KU law, made me
a better appellate attorney,and helped me get
a prestigious clerkship that (I hope) will open
even more doors for me” (L. Krebs, personal
communication, June 2, 2016).
So, rather than thinking we are going
to alter the course of Western Civilization,
perhaps we CT teachers should be more
modest and compare ourselves to those
who teach music. If we were music teachers
and had beginning students for one or two
semesters, we would expect that some students
simply could not learn to play. Others would
learn to play —but not well. But, there may
be a few who would learn to play very well.
Some of these might even become virtuosos.
And, likewise for the teaching of CT, there
will be a few students who make teaching
how to “honestly evaluate alternatives” a
very rewarding job. And, the fact is, we have
no idea what the long-term effects of CT
instruction on our students might be. As far
as I know, no one gives posttests ten years
after graduation. Perhaps some other curious
professor will take that on.
References
Ariely, D. (2008). Predictably irrational: The
hidden forces that shape our decisions.
New York: Harper Collins.
Arum, R. and Roksa, J. (2011). Academically
adrift: Limited learning on college
campuses. Chicago, IL: University of
Chicago Press.
Bailin, S. (1999). The problem with Percy:
Epistemology, understanding, and
critical thinking. Informal Logic, 19(2
& 3), 161-170.
Bailin, S. & Battersby, M. (2010). Reason in
the balance. Whitby, ON: McGraw-Hill
Ryerson Higher Education.
Battersby, M. (2013). Is that a fact? (rev. ed.)
Petersborough, ON.: Broadview Press.
Bishop, M. and Trout, J. (2005) Epistemology
and the psychology of human judgment.
New York: Oxford University Press.
Bok, D. (2006). Our underachieving colleges:
A candid look at how much students
learn and why they should be learning
more. Princeton, NJ: Princeton
University Press.
Brafman, O. & Rom Brafman. (2008). Sway:
The irresistible pull of irrational
behavior. New York: Broadway Books.
Burton, R. (2008). On being certain: Believing
you are right even when you aren’t.
New York: St. Martins.
Casserly, M. (2012, December 12). The
10 skills that will get you hired
in 2013. Forbes. Retrieved from
http://www.Forbes.com/sites/
megancasserly/2012/12/10/the-10-
skills-that-will-get-you-a-job-in-2013/
Cederblom, J. and Paulsen, D. (2006). Critical
reasoning. 6th Ed. Belmont, CA:
Wadsworth Publishing.
Clifford, W.K. (2005). The ethics of belief. In.
S. M. Cahn (Ed.), Ten essential texts in
the philosophy of religion (pp. 367-
372). New York: Oxford University
Press.
Damer, T. E. (2001). Attacking faulty
reasoning. (4th ed.) Belmont, CA:
Wadsworth Publishing.
16 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Damasio, A. R. (1994). Descartes’ error. New
York:Quill.
Elgin, C. (2010). Persistent disagreement. In
Richard Feldman & T. Wareld (Eds.)
Disagreement (pp. 53-68). New York:
Oxford University Press.
Ennis, R.H. (1987). A conception of critical
thinking—with some curriculum
suggestions. APA Newsletter on
Teaching Philosophy. Summer, 1-5.
Ennis, R. H. (2011a). Critical thinking:
Reection and perspectives. Part I.
INQUIRY: Critical Thinking Across the
Disciplines. 26(1), 4-18.
Ennis, R. H. (2011b). Critical thinking:
Reection and perspectives. Part II.
INQUIRY: Critical Thinking Across the
Disciplines. 26(2), 5-19.
Ennis, R. H. (2013). Critical thinking across
the curriculum: The wisdom CTAC
program. INQUIRY: Critical Thinking
Across the Disciplines. 28(2), 25-45.
Ennis, R. H., Millman, J., & Tomko, T. N.
(2004). Cornell critical thinking
tests, level X and level Z manual,(4th
ed.) Seaside, CA.: Critical Thinking
Company.
Ennis, R.H. & Weir, E. (1985). The Ennis-
Weir critical thinking essay test. Pacic
Grove, CA: Midwest Publications.
Facione, P. & Facione, N. C. (1994). The
California critical thinking skills test:
Form A and B test manual. Milbrae,
CA: The California Academic Press.
Facione, P. (1986). Testing college level
critical thinking. Liberal Education,
Fall.
Feldman, R. & Wareld, T. (Eds.). (2010).
Disagreement. New York: Oxford
University Press.
Fisher, A. & Scriven, M. (1997). Critical
thinking: Its denition and assessment.
Point Reyes, CA: Edgepress.
Fumerton, R. (2010). You can’t trust a
philosopher. In Richard Feldman & T.
Wareld (Eds.) Disagreement (pp.91
-110). New York: Oxford University
Press.
Goldstein, R. (2015). What philosophers
really know. (Review of Philosophy of
Language: The Classics Explained, by
Colin McGinn) New York Review of
Books. 62(15). 48-50.
Govier, T. (1999). The philosophy of argument.
Newport News, VA: Vale Press.
Groarke, L. (2009). What’s wrong with the
California critical thinking skills test?
In Sobocan, J. & Groarke, L. (Eds.)
(2009). Critical thinking education and
assessment: Can higher order thinking
be tested? London, ON: The Althouse
Press, 35 -54.
Haidt, J. (2012). The righteous mind: Why
good people are divided by politics and
religion. New York: Pantheon Books.
Halpern, D. F. (1996). Thought and knowledge:
An introduction to critical thinking (3rd
ed.).Mahwah, NJ: Lawrence Erlbaum
Associates Publishers.
Hatcher, D. L. (1985). Critique of Critical
Thinking.” (1985). Teaching Thinking
and Problem Solving, October, 1-2, 5.
Reprint: Thinking. (1986), Fall, 14-16.
Hatcher, D. L. (1999a). Why formal logic is
essential for critical thinking. Informal
Logic, (19)1, 77-89.
Hatcher, D. L. (1999b). Why we should
combine critical thinking and written
instruction. Informal Logic, 19(2 & 3),
171-183.
FALL 2015, VOL. 30, NO. 3 17
Hatcher, D. L. (2000). Arguments for
another denition of critical thinking.
INQUIRY: Critical Thinking Across the
Disciplines, 20(1), 3-8.
Hatcher, D. L. (2001). Why Percy can’t think:
The departmentalization of critical
thinking. Informal Logic, 21( 2), 173-
181.
Hatcher, D.L. (2003). The role of personal
values in argument evaluation. J.A.
Blair, D. Farr, H.V. Hansen, R.H.
Johnson, & C.W. Tindale (Eds.).
Informal Logic @25: Proceedings
of the Windsor Conference. Windsor,
Ontario: OSSA.
Hatcher, D. L. (2006). Stand-alone versus
integrated critical thinking courses.
Journal of General Education. 55 (3-
4), 247-272.
Hatcher, D. L. (2009). Assessing critical
thinking skills: A fteen-year
perspective. Critical thinking,
education, and assessment. Althouse
Press, University of Western Ontario,
London, Ontario.
Hatcher, D. L. (2011) Which test? Whose
scores? Comparing standardized
critical thinking tests. New Directions
in Institutional Research. Ed. Jeremy
Penn. San Francisco: Jossey-Bass.
Hatcher, D. L. (2013a). Is critical thinking
across the curriculum a plausible
goal? Mohammed, D., & Lewiński,
M. (Eds.). Virtues of Argumentation.
Proceedings of the 10th International
Conference of the Ontario Society for
the Study of Argumentation (OSSA), 22-
26 May 2013. Windsor, ON: OSSA, pp.
1-16.
Hatcher, D. L. (2013b). Reections on
critical thinking: Theory, practice
and assessment, INQUIRY: Critical
Thinking Across the Disciplines, 28(1)
4-24.
Hatcher, D. L. & Spencer. L. A. (2006).
Reasoning and writing: From critical
thinking to composition. (3rd ed.).
Boston: American Press. (Original
work published by Copley Publishing
, 1992 and Rowman and Littleeld,
1993.)
Hitchcock, D. (2004). The effectiveness of
computer-assisted instruction in critical
thinking. Informal Logic, 24(3), 183-
217.
Johnson, R. H. (2009). The implications of
the dialectical tier for critical thinking.
In J. Sobocan and L. A. Groarke
(Eds.) Critical thinking education and
assessment. London Ontario: Althouse
Press, 55-74.
Johnson, R. H. (1996). The rise of informal
logic. Newport News, VA.: Vale Press.
Johnson, R. H. (2000). Manifest rationality:
A pragmatic theory of argumentation.
Mahwah, N.J.: Lawrence Erlbaum
Associates.
Johnson, R. H. (2012). When informal logic
met critical thinking. INQUIRY:
Critical Thinking Across the
Disciplines, 27(3), 5-14.
Johnson, R. H. & Blair, J. A. (1994a). Logical
self-defense. New York: McGraw-Hill.
Johnson, R. H. & Blair, J. A. (1994b). New
essays in informal logic. Windsor:
Informal Logic.
Kahane, Howard. (1971). Logic and
contemporary rhetoric. Belmont, CA:
Wadsworth Publishing.
Kuhn, T. S. (1962). The structure of scientic
revolutions. Chicago: University of
Chicago Press.
18 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Lipman, M. S. (2003). Thinking in education
(2nd ed.) New York: Cambridge
University
Marcus, G. (2008). Kluge: The haphazard
construction of the human mind.
Boston: Houghton Mifin.
McPeck, J. E. (1981). Critical thinking and
education. New York: St. Martin’s
Press.
Mill, J. S. (1978). On liberty. Elizabeth
Rapaport(Ed.). Indianapolis: Hackett
Publishing.
Missimer, C. (1986). Good arguments: An
introduction to critical thinking.
Englewood Cliffs, NJ: Prentice Hall.
Nosich, G. M. (1982). Reasons and arguments.
Belmont, CA: Wadsworth Publishing.
Nosich, G. M. (2010). From argument and
philosophy to critical thinking across
the curriculum. INQUIRY: Critical
Thinking Across the Disciplines, 25(3),
4-13.
Nosich, G. M. (2012). Learning to think things
through: A guide to critical thinking
across the curriculum. (4th ed.) Boston:
Pearson.
Ortiz, C.M.A. (2007). Does philosophy
improve critical thinking skills?
(Unpublished Master’s Thesis)
University of Melbourne, Melbourne,
Australia.
Pascaralla, E.T. & Terenzini, P. T. (2005). How
college affects students (Vol. 2): San
Francisco: Jossey-Bass.
Paul, R. W. (2011). Reections on the nature
of critical thinking, its history, politics,
and barriers, and its status across the
college/university curriculum. Part I.
INQUIRY: Critical Thinking Across the
Disciplines, 26(3), 5-24.
Paul, R. W. (1993). Interview. In
Conversations with critical thinkers.
J. Esterle and D. Clurman, (Eds.) San
Francisco, CA: The Whitman Institute:
91-111.
Paul, R., Elder, L., Bartell, T. (1995). “Study
of 38 Public Universities and 28
Private Universities to Determine
Faculty Emphasis on Critical Thinking
in Instruction: Executive Summary.”
Accessed March 15, 2013. http://www.
criticalthinking.org/pages/study-of-
38-public-universities-and-28-private-
universities-/598.
Plato. (1973). The collected dialogues of Plato.
E. Hamilton and H. Cairns, (Eds.).
Princeton: Princeton University Press.
Popper, K. (1963). Science: conjectures
and refutations. In Conjectures and
refutations: The growth of scientic
knowledge. New York: Harper and
Row, 33-65.
Possin, K. (2013). A fatal aw in the collegiate
learning assessment test. Assessment
Updates, 25(1). 8-12.
Russell, B. (1959). The scientic outlook. New
Youk: W.W. Norton.
Scriven, M. (1991). The prostitution of the
critical thinking requirement, Part I. CT
News, 10(2), 1-5.
Scriven, M. (1992). The prostitution of the
critical thinking requirement, Part II.
CT News, 10(3-4), 6-12.
Siegel, H. (1980), Critical thinking as an
educational ideal. Educational Forum,
45(1), 7-23.
Siegel, H. (1987). Relativism refuted. Boston:
D. Riedel Publishing.
Siegel, H. (1988). Educating reason. New
York: Routledge, Inc.
FALL 2015, VOL. 30, NO. 3 19
Siegel, H. (1992). Education and the fostering
of rationality. In R. Talaska (Ed.)
Critical reasoning in contemporary
culture (pp. 89–112). Albany, NY: State
University of New York Press.
Siegel, H. (1997). Rationality redeemed. New
York: Routledge, Inc.
Sobocan, J. & Groarke, L. (Eds.). (2009).
Critical thinking education and
assessment: Can higher
order thinking be tested? London, ON.:
The Althouse Press.
Solon, T. (2003). Teaching critical thinking:
The more, the better! The Community
College Enterprise, 9, (2), 25-38.
Tavris, C. and Elliot Aronson. (2007). Mistakes
were made (but not by me): Why we
justify foolish beliefs, bad decisions,
and hurtful acts. Orlando, FL:
Harcourt.
Van Gelder, T. & Bissett. M. (2004).
Cultivating expertise in informal
reasoning. Canadian Journal of
Experimental Psychology, 58(2), 142-
152.
Author Information
Donald Hatcher is Professor of Philosophy
in the College of Arts and Sciences at Baker
University in Baldwin City, KS, 66006.
From 1989 to 2009, he was Director of
Baker’s Liberal Arts Program, an experiment
in combining critical thinking and written
composition with the study of primary texts.
From 1988 to the present, he serves as the
Treasurer for the Association for Informal
Logic and Critical Thinking (AILACT). He
can be contacted at dhatcher@bakeru.edu
20 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Are We Asking the Right Questions about Critical Thinking Assessment?
A Response to Hatcher
David Wright
Sam Houston State University
Abstract
This is a response essay to Donald Hatcher’s (2015), “Critical Thinking Instruction: A Realistic
Evaluation: The Dream vs. Reality.” Hatcher argues that critical thinking (CT) instruction
seriously falls short of the ideal of honestly evaluating alternative evidence and arguments.
This failure is apparent, he argues, when one surveys student performance on a variety of CT
assessment tests. Hatcher reviews the current CT assessment data, which includes an extensive
pool of results collected from Baker University where Hatcher oversaw a sophisticated and
well-funded CT program for about two decades. Hatcher also argues that evidence from the
philosophical and psychological literatures on disagreement and judgment suggests even CT
experts fail to model the ideal and that CT has suffered from an unrealistic conception of
rationality and human decision-making. I reply by arguing that, by putting the CT assessment
data in a different context and asking an alternative set of questions, one can justiably derive
a more positive evaluation of the future of CT instruction in light of the CT ideal. Instead of
focusing on whether students are achieving the CT ideal by the time of the post-test, instructors
should ask whether they are making the kind of progress that there is good reason to expect. I
close by challenging the soundness of the proposed implications of Hatcher’s arguments.
Keywords: critical thinking, critical thinking assessment, critical thinking and human
psychology
I. Introduction
In his provocative and thoughtful
essay, Donald Hatcher provides a striking
challenge to critical thinking (CT) theorists
and instructors. Under one guise or another,
those in the academic CT community
commonly believe that most students, if
taught and motivated in the right way, can be
“appropriately moved by reasons,” (Siegel,
1992 as cited in Hatcher, 2015, p.2), engage
in “rational reective thinking about what
to believe or do” (Ennis, 1987 as cited in
Hatcher 2015, p.2), or, in Hatcher’s preferred
formulation, arrive at a judgment “only after
honestly evaluating alternatives with respect
to available evidence and arguments” (Hatcher
and Spencer, 2006 as cited in Hatcher, 2015,
p.4). As denitions, the academic community
will quibble over just how to express the
precise nature of CT, but few will reject it as
an ideal to strive for alongside their students.
However, Hatcher (2015) has some news for
this community:
The dream of CT as an educational ideal,
as worthy as it might be, is in reality only
a dream. This sad conclusion is based
on assessment scores on standardized
tests from a wide variety of scholarly
sources, the ongoing disagreement among
experts, and CT’s apparently faulty view
of psychology, especially of the role that
values play in argument evaluation (p.6).
After articulating his reasons to
substantiate each of these points, Hatcher
draws out the implications of his ndings.
These include suggestions for why the CT
ideal might not be abandoned altogether. In
FALL 2015, VOL. 30, NO. 3 21
this response essay, I challenge Hatcher’s
dream deation argument as well as a selection
of his proposed implications. First, I suggest
that CT instruction may not be in quite as bad
a shape as he suggests once one puts his CT
assessment ndings into proper context. Rather
than disputing Hatcher’s numbers, I propose
a different set of questions that CT instructors
might ask than the ones prompted by Hatcher’s
results and say why these are arguably more
appropriate. Second, I question the reasoning
behind some of his suggestions. For example,
I question whether CT instructors should
take solace in the idea that, at minimum, they
are probably encouraging fallibilism in their
students or that in teaching their students to
state clearly the reasons for their views they
are not doing anything wrong. Finally, I inquire
as to whether more research might be done on
some of the especially high and low achievers
in Hatcher’s study and how these outliers
could point to a way forward in CT assessment
research. Before turning to these objections,
however, it is rst necessary to review the key
elements of Hatcher’s arguments.
II. The Mere Dream
If one is going to argue that the dreams
of CT as an educational ideal are merely
dreams, one needs a plausible target for a
CT ideal. To this end, Hatcher begins with
an explanation of how his conception meets
the relevant challenges and expectations of a
CT ideal. By identifying CT with the honest
evaluation of alternatives concerning evidence
and arguments, Hatcher’s proposed denition
becomes broadly consistent with the most
prominent scholarly denitions of CT, avoids
prominent objections to CT denitions, serves
as a practical guideline to CT instructors, and
maintains consistency with the aims of those
who have been widely regarded as practicing
“good thinking” in the history of philosophy.
Hatcher and his colleagues put this ideal
to the test by designing a well-organized
critical thinking program at Baker University
over the course of twenty years. As Hatcher
(2013b) says in an earlier paper, “The Baker
program was developed and sustained in ideal
conditions,” amassing grants totaling over
one million dollars, allowing his faculty to
attend prominent CT conferences, receive
course releases, organize summer instructional
workshops, and other course enhancement
opportunities (p. 9).
Hatcher nds the results of the program
that he created (following a textbook that
he co-authored), supervised, and evaluated
(see Hatcher 2013b for an overview), as
well as student CT assessment scores more
broadly, discouraging to say the least. One
of the key sources for discouragement is the
effect sizes of the gains (represented as a
percent of a standard deviation) that Hatcher
found in the studies of his university and for
CT higher education more generally. As an
example illustrating the latter point, Pascarella
and Terenzini’s (2005) research using the
California Critical Thinking Skills Test
(CCTST) (Facione P. & Facione, N. C., 1994)
found that the effect size gains for critical
thinking performance from students in their
freshman to senior year in college was between
.55-.65 (as cited in Hatcher, 2015, p. 7). To put
that effect size in context, an effect size of .5
on the CCTST (which has 34 questions) would
mean that an average student taking the test
as a senior would see an improvement of only
2.25 questions correct in comparison to what
the student scored as a freshman.
At Baker, there were a variety of
results with pre- and post-test effect size
gains ranging from .97-.57, depending on the
particular assessment test in use. Using the
Cornell Z (CLZ) (Ennis, Millman, & Tomko,
2004), which is a 52-question test, the mean
student effect size gain was .57. This means
that students improved just 2.7 questions after
taking the two-semester sequences overseen
by Hatcher and his colleagues (Hatcher,
2015, 7). And it is not as if students had little
room for improvement: the average Baker
student taking the test after the two-semester
22 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
CT course scored at approximately 56%, a
failing grade. Changing the test did not alter
that outcome either—post-test mean scores
were at failing levels on both the Ennis &
Weir Critical Thinking Essay Test (E-W)
(Ennis & Weir, 1985) and the CCTST. Hatcher
compares these outcomes to the post-tests
found in the Test of Standard Written English
at Baker, where the mean post-test score
was about 80%. This suggests that students’
poor performance is linked more to CT than
comparable subjects in the curriculum. Even
more discouraging, while the average student
score at Baker showed some improvement on
the CT post-test, 30% of Baker students did
worse on the CLZ post-test than they did on
the pre-test. One could arguably summarize
these ndings by saying that, regardless of
whether students participated in a well-funded
and carefully planned curriculum at Baker or
elsewhere, most college students come into
their universities with failing CT skills and
leave with failing CT skills.
To this gloomy news, one might have a
number of objections. One might say that the
problem is that Baker students are especially
poor at CT, but Hatcher replies that essentially
everyone that reports on students’ performance
on CT post-tests nds students getting failing
grades, with one not-too-inspiring exception
being Tim van Gelder’s (van Gelder & Bissett,
2004) CT courses achieving post-test mean
scores of 66% (as cited in Hatcher, 2015, p.8).
To the charge that the tests are too difcult and
that no one should expect someone to get a
passing grade on these tests, Hatcher says that
he gave the CCTST test to his colleagues at
Baker and no one reported missing more than
two questions, suggesting that it is far from
impossible for groups to achieve high scores.
Alternatively, the issue may be that CT skills
are only taught in philosophy classes and not
across the curriculum. Perhaps what is needed
is just more philosophy courses to bolster CT
performance. But, alas, there is little evidence
that philosophy courses substantially improve
CT skills (Ortiz, 2007, Hitchcock, 2004; as
cited in Hatcher, 2015, p.8). This assessment
data, then, suggests a dilemma: either standard
approaches to CT are misguided or especially
ineffective or most students are incapable
of learning the skills. The problem with the
former is that low assessment scores result
from every known CT theory and teaching
method—no one is doing especially better than
any other. The more likely option, Hatcher
argues, is that the problem is not the ideal but
the degree to which most students can achieve
that ideal.
Hatcher’s second type of argument for
why his ideal of CT is problematic concerns
the problem of persistent disagreement.
Hatcher’s denition of CT requires that
individuals form judgments resulting from an
honest evaluation of evidence and arguments.
Yet, when Hatcher looks at the literature on
the epistemology of disagreement, he notices
that there is a widespread lack of consensus
among experts (Elgin, 2010, as cited in
Hatcher 2015, p.9). This is distressing given
that one would think that at least experts would
be able to objectively settle disagreements.
After all, if two experts are epistemic peers
(roughly, each person is equally familiar with
the evidence and arguments and is equal in
their epistemic virtues such as intelligence
and thoughtfulness), then what could
possibly account for ongoing disagreement
except that even experts do not live up to
Hatcher’s CT ideal? To put the argument more
directly, Hatcher (2015) gives the following
formulation:
If the ability to evaluate alternative
positions and arguments objectively was
possible, then experts (especially experts)
who looked at the same evidence and
arguments would agree. However, we
know that experts can look at the same
evidence and arguments and not agree.
Hence, the honest or objective evaluations
of alternative positions may not be
possible. (p.9, emphasis removed)
FALL 2015, VOL. 30, NO. 3 23
Persistent disagreement among experts
extends even to the CT scholarly community.
Hatcher relates a story where he and his wife
attended a renowned CT conference with
many of the top scholars in the CT world.
At the close of the four-day conference, his
wife pointed out that, despite all the vigorous
arguments she had witnessed concerning CT
and CT pedagogy, she did not observe one
person claiming to have changed his or her
mind. What conclusion can she draw except to
say that even experts at CT are not living up
to the ideal? As Hatcher puts it, “I have never
been able to answer my wife’s challenge, and
now I take it very seriously. If the ‘honest,’ that
is objective, evaluation of alternative positions
were possible, surely people who teach CT
should be able to change their minds easily
relative to new arguments and evidence. But,
they do not” (Hatcher, 2015, p.13).
The remainder of Hatcher’s paper
serves as a diagnostic exercise of where
CT went wrong in its conception of human
judgment and decision-making. Since the
critical responses to follow are not especially
concerned with this section, this discussion
will be especially brief. To put Hatcher’s
point too crudely, CT theorists have been
overly Platonistic in their conception of how
human decision-making works, emphasizing
the power of reason to reliably guide humans
in overcoming their spirited and passionate
psyches. More recent ndings in cognitive
psychology suggest that intuition and emotion
play a dominant role in how humans reason
and choose (Damasio, 1994; Haidt, 2012, as
cited in Hatcher, 2015, p.11). In addition, it
needs to be recognized that, in some cases,
two interlocutors may both make reasonable
and mutually respected points on some
controversial issue, but still continue to
disagree because of different values. In some
cases, the interlocutors may even have the
same values (e.g. “liberty” and “equality”),
but weigh or rank the importance of the two
values differently, which can account for their
respective divergent judgments. If weighing
or ranking is more a matter of emotion than
reason as some respected researches have
thought (Burton, 2008, Damasio 1994, Haidt
2012, as cited in Hatcher, 2015, p.12), then it
seems plausible and even predictable that two
interlocutors with admirable CT skills could
reasonably disagree on an issue, a state of
affairs that appears to be pervasive.
Hatcher provides at least ve positive
suggestions that follow from the implications
of his argument. First, even if most students
cannot reliably reach the ideal of honestly
evaluating alternatives, it is not as if nothing
has been gained by moving them through
familiar points in the CT curriculum. One
positive outcome of CT instruction should be
some measure of “fallibilism,” (Siegel, 1997
as cited in Hatcher, 2015, p. 14), which is a
certain kind of respect for the beliefs of others
in light of the recognition that there can be
good reasons for positions with which one
disagrees. This is a notable outcome since
this attitude could promote a less dismissive
discourse among interlocutors. Second, there
are practical and moral grounds for teaching
CT. This is because CT instruction gives
students at least some way of distinguishing
good and bad arguments, even if they do not
reach an ideal level of competence in being
fair-minded. Moreover, CT skills are skills
especially coveted by employers (Casserly,
2012 as cited in Hatcher, 2015, p. 14), which
provides a clear practical reason to keep
teaching the subject. From a moral point of
view, CT instruction aims at an eminently
defensible moral ideal—creating more free
and autonomous thinkers. Third, Hatcher notes
that requiring students to be clear in stating
their reasons for their views is hardly doing
something wrong, and can even claim to be
contributing to clarity, which is arguably an
unqualied good. By compelling students
to be clear, CT instructors, at minimum, are
requiring that students establish the very
conditions for the proper evaluation of ideas.
24 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Fourth, CT instructors should scale back their
ambitions in a CT class. They should not
expect to transform students into unemotional
rational evaluators, but rather take a more
limited aim of encouraging their tendencies
toward being “fallibilists” rather than “true
believers” who refuse to alter their beliefs
regardless of the evidence presented to them.
Finally, in keeping with modest
expectations theme, Hatcher recommends a
frame shift. No longer the self-proclaimed
evangelists of the church of reason, CT
instructors should borrow expectations from
elsewhere in the educational profession:
introductory music instruction. After one or
two semesters of coursework, these teachers
expect that some students will achieve basic
competence (but not much more than that),
some will not learn to play at all, and a few
will become virtuosos. Hatcher is led to this
suggestion, in part, from some of the outlier
students at Baker. When reviewing the data,
he noticed that 11% of the students in a three-
year sample improved their scores by 9 points
or more on the CLZ—which represents an
effect size of 2.0. This, plus other anecdotal
cases of students experiencing great personal
growth and benets from their CT experience,
suggests that CT instruction can be quite
inuential for some. Instructors can stop
wringing their hands at the paltry effect sizes
of CT instruction and accept, as so many music
teachers do, that mastery is rare and worth
pursuing and incompetence is the norm and,
well, predictable.
III. Questions about Assessment Data
Hatcher does make a strong case that,
even under optimal conditions, mastering
CT during a college career is unlikely for the
average student. Given the data he provides, it
is difcult to say that students, both at Baker
and elsewhere, who perform at failing levels
on tests like the CLZ have attained mastery
of CT in the sense that they will be, say,
appropriately moved by reasons or prone to
honestly evaluate alternatives. Yet one wonders
to what extent Hatcher could justiably expect
something dramatically different. Hatcher
(2015) reports that at Baker students achieved
post-test effect size gains from .97-.57 and
notes that while this is better than some gains
reported in the assessment literature, “one can
hardly say they indicate great success with
respect to students learning CT skills” (p. 7).
That may be true, but it would be helpful to
hear precisely what kinds of numbers Hatcher
was expecting to see students achieve on these
tests if one were to count their performance as
a “great success.” In this paper and elsewhere,
Hatcher (2013a, 2013b, 2015) disparages effect
size increases by contextualizing these with
the number of problems that students improve
on the test (2013a, p. 3; 2013b, p. 19). For
example, when the effect size gain for Baker
students on the CLZ was .57, this amounted to
the average student improving his or her score
by a mere 2.7 questions (Hatcher, 2015, p. 7).
Such results hardly suggest that most students
are well on their way to transformative
experiences as critical thinkers. The question,
however, is whether this sort of framing is
the proper context to evaluate successful CT
instruction.
In his landmark work on effect sizes,
Jacob Cohen (1988) famously suggested,
with cautionary warnings, that in the social
sciences researchers consider effect size
according to the standard of .20 (small effect),
.50 (medium effect), and .80 (large effect). By
that rough standard, such as it is, the students
at Baker responded at least moderately well
to Hatcher’s program. But, as many have
recognized, Cohen’s suggested measurements
are imperfect for broad characterizations of
educational performance gains. Part of what
is needed for evaluating educational gains is
a more specied context concerning, among
other factors, the difculty of altering student
performance and how Hatcher’s scores
compare to relevant alternatives. For example,
Lipsey et al (2012) report that educational
FALL 2015, VOL. 30, NO. 3 25
interventions concerning standardized math
and reading achievement tests rarely achieve
effect size gains of .30. That fact establishes
the benchmark from which other claims of
successful educational intervention should
be evaluated. “By appropriate norms—that
is, norms based on empirical distributions
of effect sizes from comparable studies—an
effect size of .25 on such outcome measures
is large and an effect size of .50, which would
only be ‘medium’ on Cohen’s all encompassing
distribution, would be more like ‘huge’”
(Lipsey et al., 2012, p. 4).
One prominent source of context for
CT instruction comes from studies reported
by Pascarella and Terenzini (2005), who say
that one can expect freshman to sophomore
effect size gains on the CCTST at about .34
and a freshman to senior gain at .54 (p. 157;
cf. Hatcher, 2006, p. 256; Hatcher, 2011, p. 33;
Hatcher, 2013a, 2; Hatcher, 2013b, p. 15). For
context concerning particular interventions,
Abrami et al’s (2008) meta-analysis found
mean effect size gains of .34. Huber and
Kuncel’s (2016) more recent meta-analysis on
CT interventions and college instruction was
consistent with Pascarella and Terenzini’s,
nding mean effect size gains from freshman
to senior years of .59 (p.455). All of these
numbers, of course, come with cautionary
notes. For example, in Hitchcock’s (2004)
study, he identies a variety of reasons why
students’ growth in CT might not show up on
tests like the CCTST. These include issues
related to motivation, actual course content,
or even students’ familiarity with answering
CT-oriented multiple-choice questions.
Students who enrolled in computer-enhanced
courses that use multiple-choice formats would
arguably be more comfortable answering
CT questions in that format than Hatcher’s
students whose CT training was more closely
tied to essay writing (see Hatcher, 2013b, esp.
pp. 19-20 for a related discussion of this).
Similarly, Hatcher (2011) reports on the need
to examine features like individual teacher
skill in considering how effective a particular
intervention might be (p. 36-37; cf. Solon,
2003), as well as how much variance there
might be from one test to another. When, for
various reasons, the program switched from
giving Baker students the E-W test to the
CCTST or the CLZ tests, the effect size gains
from pre- and post-tests did go down from
.97 to .58 (mean of both CCTST and CLZ)
(Hatcher, 2013b, pp. 18-19). In the same
paper, Hatcher (2013b) also suggests that
in his judgment, “the E-W is in fact a better
gauge of students’ abilities to think critically
in real situations than either the CCTST or the
CLZ,” which gives one at least some reason to
take those scores as relevant to Hatcher’s CT
ideal (p. 20). Moreover, using the E-W, Baker
students achieved a 1.47 mean effect-size gain
from their freshman to senior years (2013b, p.
19). This is nearly three times what Pascarella
and Terenzini (2005) said that one should
expect in freshman to senior improvement (p.
574).
Now the above cited remarks and data
are hardly news to Hatcher. The question, then,
is whether Hatcher is using the appropriate
standard in declaring the CT ideal merely
a dream. By familiar grading standards,
available evidence does show that CT students
are typically “failing” CT post-tests, but
did Hatcher expect that the average student
would move from failing to mediocre (say,
by getting a ‘C’) or even generally competent
(with a ‘B’)? If so, what is the basis for that
expectation? Given what is known about
educational interventions at the college
level, in order to get the average student
from a failing pre-test to ‘B’ level post-test
competence would require atly astonishing
results. In a paper reecting on his experiences,
Hatcher (2013b) notes how he recognized that
Baker students were doing well comparable
to what relevant statistics said that one could
expect. Thanks in part to Hatcher’s planning
and the resources available, students were,
by some measures, signicantly exceeding
26 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
expectations. However, this did not make him
feel better about students’ failing performance.
As he said, “My intuition is that it is reasonable
to expect a better gain” (Hatcher, 2013b, 18). I
confess that I, too, would hope for higher gains
in Hatcher’s circumstances, but given what is
known from the CT assessment literature, my
intuition is that it is not reasonable to expect a
signicantly better gain. A relevant question,
then, is what precisely, beyond intuitions,
should one use to evaluate student achievement
on CT assessment tests.
Consider Hatcher’s (2015) statement
that, “the data from the standardized tests
show that the likelihood that most students
are actually learning these skills needed
for effective CT in any substantial way is
minimal” (p. 10). I contest this claim. The
statement’s contestability turns on what
one means by “substantial.” If the term is
referenced to the standard of whether most
students will become even “C” level critical
thinkers by post-test time, then, no, average
student CT achievement is not substantial.
This should not be especially surprising. As
Tim van Gelder (2005) has said, “Humans
are not naturally critical. Indeed, like ballet,
critical thinking is a highly contrived activity.
Running is natural; nightclub dancing is less
so; but ballet is something people can only do
well with many years of painful, expensive,
dedicated training” (p.42). Notice that the
“only” qualication in this context species
a necessary condition for doing ballet well.
Painful training is required but signicant
results are not guaranteed. Like Hatcher’s
music teacher analogy, one should not be
surprised that many humans begin and, even
after costly and rigorous training, remain
inadequate ballet dancers. Here, Hatcher and I
are in apparent agreement.
If, however, the term “substantial”
is referenced to comparisons of standard
teaching techniques among comparable student
populations, then Hatcher is mistaken: his
program at Baker did achieve substantial
results among a range of student cohorts.
This is also true of others like those using van
Gelder’s argument mapping methods (Ortiz,
2007; see also van Gelder, 2001), which
reliably achieved results of .80 or Twardy
(2004), who reported effect size gains of .72
on the CCTST post-tests. This is signicant
given that Ortiz’s (2007) meta-analysis on
CT instruction found that university students
using traditional CT methods (lectures and
discussion but no argument mapping) achieved
effect size gains of .34 (p.73). Recall Huber
and Kuncel’s (2016) ndings about freshman
to senior improvement in CT general ability.
They had a quite favorable response to that
kind of gain saying that, “For a specic group
of individuals who already possess above-
average cognitive abilities, a gain of 0.59
SDs on a purportedly general ability is quite
impressive (comparable to going from the 50th
percentile to the 72nd percentile)” (p. 457).
Even after Baker students switched to the
CCTST, the freshman to senior effect size gain
was .95 (Hatcher 2013b, p. 19), which sounds
as if it might impress Huber and Kuncel if
not Hatcher himself. That said, Huber and
Kuncel (2016) are generally unimpressed by
the results of CT educational interventions in
the data set they considered, though they too
think that more and better research is needed
since, “The central limitation of the literature
we synthesize is the inability to make clear
causal conclusions” (p. 458). This is especially
because current research does not sufciently
account for general maturation (which might
increase CT of college-age populations apart
from what they experience at college) and
because nursing majors (who constitute a
signicant portion of their samples) might
not be representative of college students more
generally.
Now, of course, far from every CT
instructor is achieving the kinds of results
that Hatcher has. Suppose, though, that one is
achieving above average results if not quite the
numbers that Hatcher achieved in his enriched
FALL 2015, VOL. 30, NO. 3 27
environment. Whether one views this in a
positive or negative light depends greatly on
the questions an instructor uses for evaluation.
Suppose the questions are “Are my students
capable critical thinkers?” or “Are my students
now prepared to rigorously and fearlessly
confront their biases and avoid sloppy
thinking?” The answer to these questions is
almost surely, “no.” The same might be said
of many CT instructors and experts. However,
guided by the ndings cited above, instructors
might ask themselves, “Are the majority of
my students making real progress toward
being critical thinkers?” or “Are my methods
achieving what I should reasonably expect
given the available evidence?” Then the
answers may be “yes.” If the evidence points
to “no,” then that will likely prompt further
investigation.
Admittedly, these latter questions are
less grand and inspiring than the former, but
I take it that they are meaningfully relevant
to the ideals mentioned at the start of this
essay. This is true, admittedly, insofar as
the content of the tests go some way toward
evaluating relevant CT skills. With some
qualications (like those mentioned about
Hatcher’s preference for the E-W), I take it
that this is so. If the content of the tests is
not relevant to Hatcher’s CT ideal, then he
should not be concerned if students fail to
improve signicantly. Given that Hatcher does
indeed seem concerned about the students’
performance on the tests, I take it that he does
think that the tests’ content is relevant to his
CT ideal. On the other hand, if the content
of the tests is relevant to the CT ideal, then it
presumably matters that students scored higher
on the tests. The available evidence suggests
that many, if far from all, students did improve
their respective abilities to honestly evaluate
alternative evidence and arguments or else one
would not have seen results like those achieved
at Baker.
To better assess what questions
instructors should be asking (and how to
answer them), one suggestion likely to be
welcomed in all corners is that more precise
measurements and experimental controls are
in order. In one widely cited paper on effect
size interpretation in educational contexts, Hill,
Bloom, Black, & Lipsey (2008) argue that
effect size gains should be evaluated according
to three empirical benchmarks: normative
expectations for growth over time (i.e. “How
does the effect of an intervention compare
to a typical year of growth for a given target
population of students?”), policy relevant
performance gaps (“How do the effects
of an intervention compare with existing
differences among subgroups of students
or schools?”), and observed effect sizes for
similar interventions (“How do the effects of
an intervention compare to those from previous
studies for similar grade levels, interventions,
and outcomes?”) (p. 173, p. 174, p. 175). On
this scheme, it is only when one contextualizes
one’s ndings within the answers to these
questions that effect size gains, whatever
they are, become truly meaningful as guides
for what interventions are successful. One
can hope that as sophisticated demographic
information is incorporated into CT assessment
research, there will be greater consensus on
reasonable expectations.
IV. Questioning Implications
There is much to agree with in
Hatcher’s discussions of the implications of his
argument. From what has been said already it
should be evident that Hatcher is correct that
many CT instructors would be wise to adopt
a more humble approach to CT instruction.
There remain, however, some unanswered
questions concerning some suggestions.
The rst concerns the reasoning behind the
claim that fallibilism “should” result from
the recognition that others have good reasons
for their disagreements. It is not hard to work
out how this rationally should follow—if I
can see that other people have good reasons
for disagreeing with my beliefs, then I should
at least be open to revising my knowledge
28 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
claims. Indeed, Siegel (2006) calls fallibilism
a “cornerstone of contemporary epistemology”
(p. 145). However, this connection might be
missed by precisely those who are otherwise
not skilled at CT. It may well be that those
most in need of a fallibilistic attitude are the
least aware that they need to be open to that
very possibility. This would be in sympathy
with the famous ndings from Kruger and
Dunning (1999) that those who are the most
incompetent in, say, logical reasoning are
the least aware of their incompetence. If,
as Hatcher and others have shown, most
students are at a failing level in CT skills, one
wonders if there is any reason to think that
this particular cognitive attribute is one that
CT classes are likely to inculcate. If Hatcher
has any knowledge of relevant studies on this
question it would be helpful for him to point
to these out, especially since this is supposed
to be one of the things that CT instructors
can count on within his proposed recalibrated
expectations.
There are also some reasons to question
just how much solace CT instructors should
take in the practical or moral goals that Hatcher
identies. CT instruction, Hatcher claims,
can go at least some way toward helping
students distinguish good from bad arguments,
promotes freedom and autonomy, is something
that employers clearly seek in who they hire,
and forces students to do be clear, which is a
pre-condition for more serious critical thought.
Regarding employer desires, one small point
is that employers presumably want students
who actually have robust CT skills, not just
students who took a class or two in the subject
(though it is difcult to disentangle the role of
signaling and human capital in hiring practices
and human productivity, Page, 2010). If
some universities can justiably say that their
methods produce better CT instruction than is
otherwise available (as Ortiz, 2007 arguably
does), then that would be worth promoting
to employers. This provides further reason
to better specify the standards by which CT
instruction should be evaluated (something
one imagines that Hatcher would hardly
discourage).
Regarding the other practical and
moral goals like promoting autonomy
and forcing students to be clear in their
thinking, there remain questions here as to
whether CT instructors should take these
on as encouragements given that these
goods are at least equally available in many
other areas of academic instruction. For
instance, it seems to me that students in my
introduction to philosophy, ethical theory,
and political philosophy courses are required
or actively encouraged to identify good and
bad arguments, be clear in evaluating and
putting forward arguments, and grow as
autonomous thinkers. One guesses that many
in philosophy, the humanities, and the academy
more generally would say similar things with
at least some justication. The question is
whether CT instruction does as well or better
promoting these various goods than competing
alternatives. If CT instructors want to look
to these goods as reasons for encouragement
in their craft, it would be welcomed if they
could be claimed as a matter of comparative
advantage. It is not enough to argue that it
is false that, “nothing positive comes from a
class in CT,” (Hatcher, 2015, p. 20) because
there are some positive outcomes that can be
gained. Opportunity costs for departments,
administrations, faculty, and students are
decidedly real. While taking CT courses,
students are not taking other classes that may
offer these same practical and moral benets
and, perhaps, more.
The nal question regarding the
implications of Hatcher’s argument concerns
one of the brighter spots in Hatcher’s statistical
report: the 11% of students who improved by
two or more standard deviations on the CLZ.
One question is whether any further statistical
analysis could be done to determine who these
students were and what accounted for the
success that they experienced when so many
FALL 2015, VOL. 30, NO. 3 29
others faltered. Were most of these students
the pupils of particular professors? Did their
improvement usually raise them to ‘C’ or ‘A’
levels on the CLZ? Many instructors have
the experience of seeing a student’s mind
captivated and transformed by a course (like
those in the anecdotes from high achievers
that Hatcher includes), but comparatively
few systematically study the precise causal
contributions to these effects. Almost surely,
the causes are mixed and interactive, ranging
from demographics, class time, student
interest, instructor, course content, or even
class composition. Going any way toward
isolating variables could be helpful for future
research. In my experience, my inability to
predict just which CT students will rocket off
to signicant improvements is quite motivating
since any of them, from what little I know,
could be the ones that improve dramatically
as the course goes along. But if Hatcher or
others have done relevant research on how
to identify such students, then their number
could be expanded. Conversely, something
similar might be said for those 30% who did
worse on the CLZ post-test. It is undoubtedly
discouraging that such a signicant fraction of
students would fail to improve after Hatcher
and his colleagues’ assiduous efforts. It would
be interesting and helpful to know, after all,
just what are the proles of students who get
worse at CT after experiencing ideal conditions
for CT growth. If Hatcher has insight on these
questions, there are many who would welcome
his answers.
V. Conclusion
Disputes over how to interpret CT
assessment data will not be ending any time
soon. What Hatcher’s (2015) argument shows
is that if the interpretation of that data turns on
whether students are achieving the CT ideal,
then that ideal remains merely a dream. If the
interpretation turns on whether CT instructors
can lead the average student to make
meaningful progress toward that ideal, then
what is currently known suggests that some
CT programs are making substantive progress.
The dream of CT is arguably still in play, then,
if not quite in the form that Hatcher and many
others labored so long and hard to bring about.
References
Abrami, P. C., Bernard, R. M., Borokhovski,
E., Wade, A., Surkes, M. A., Tamim,
R., & Zhang, D. (2008). Instructional
interventions affecting critical thinking
skills and dispositions: A stage 1
meta-analysis. Review of Educational
Research, 78(4), 1102-1134
Burton, R. (2008). On being certain: believing
you are right even when you aren’t.
New York: St. Martins.
Casserly, M. (2012, December 12). The
10 skills that will get you hired
in 2013. Forbes. Retrieved from
http://www.Forbes.com/sites/
megancasserly/2012/12/10/the-10-
skills- that-will-get-you-a-job-in-2013/
Cohen, J. (1988). Statistical power analysis
for the behavioral sciences (Second
ed.). Hillsdale, NJ: Lawrence Earlbaum
Associates.
Damasio, A. R. (1994). Descartes’ error. New
York: Quill.
Elgin, C. (2010). Persistent disagreement. In
Richard Feldman & T. Wareld (Eds.),
Disagreement (pp. 53-68). New York:
Oxford University Press.
Ennis, R.H. (1987). A conception of critical
thinking—with some curriculum
suggestions. APA Newsletter on
Teaching Philosophy, Summer, 1-5.
Ennis, R. H., Millman, J., & Tomko, T. N.
(2004). Cornell critical thinking
tests, level X and level Z manual (4th
ed.) Seaside, CA.: Critical Thinking
Company.
Ennis, R.H. & Weir, E. (1985). The Ennis-
30 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Weir critical thinking essay test. Pacic
Grove, CA: Midwest Publications.
Facione, P. & Facione, N. C. (1994). The
California critical thinking skills test:
Form A and B test manual. Milbrae,
CA: The California Academic Press.
Haidt, J. (2012). The righteous mind: Why
good people are divided by politics and
religion. New York: Pantheon Books.
Hatcher, D. L. (2006). Stand-alone versus
integrated critical thinking courses.
Journal of General Education. 55(3-4),
247-272.
Hatcher, D. L. (2011) Which test? Whose
scores? Comparing standardized
critical thinking tests (pp. 29-39). New
Directions in Institutional Research.
Jeremy Penn (Ed.) San Francisco:
Jossey-Bass.
Hatcher, D. L. (2013a). Is critical thinking
across the curriculum a plausible
goal? Mohammed, D., & Lewiński,
M. (Eds.). Virtues of Argumentation.
Proceedings of the 10th International
Conference of the Ontario Society for
the Study of Argumentation (OSSA),
22-26 May 2013. Windsor, ON: OSSA,
pp. 1-16.
Hatcher, D. L. (2013b). Reections on
critical thinking: Theory, practice
and assessment. INQUIRY: Critical
Thinking Across the Disciplines, 28(1),
4-24.
Hatcher, D. L. (2015). Critical thinking
instruction: a realistic evaluation: the
dream vs. reality. INQUIRY: Critical
Thinking Across the Disciplines, 30(3),
4-19.
Hatcher, D. L. & Spencer. L. A. (2006).
Reasoning and writing: From critical
thinking to composition. (3rd Ed.).
Boston: American Press. (Original
work published by Copley Publishing,
1992 and Rowman and Littleeld,
1993.)
Hitchcock, D. (2004). The effectiveness of
computer-assisted instruction in critical
thinking. Informal Logic, 24(3), 183-
217.
Hill, C. J., Bloom, H. S., Black, A. R., &
Lipsey, M. W. (2008). Empirical
benchmarks for interpreting effect
sizes in research. Child Development
Perspectives, 2(3), 172-177.
Huber, C. R., & Kuncel, N. R. (2016). Does
college teach critical thinking? A
meta-analysis. Review of Educational
Research, 86(2), 431-468.
Kruger, J., & Dunning, D. (1999). Unskilled
and unware of it: How difculties in
recognizing one’s own incompetence
lead to inated self-assessments.
Journal of Personality and Social
Psychology, 77(6), 1121-1134.
Lipsey, M.W., Puzio, K., Yun, C., Hebert,
M.A., Steinka-Fry, K., Cole, M.W.,
Roberts, M., Anthony, K.S. and Busick,
M.D. (2012). Translating the Statistical
Representation of the Effects of
Education Interventions into More
Readily Interpretable Forms. National
Center for Special Education Research
(2013-3000).
Ortiz, C.M.A. (2007). Does philosophy
improve critical thinking skills?
(Unpublished Master’s Thesis)
University of Melbourne, Melbourne,
Australia.
Page, M. (2010) Signaling in the Labor
Market. In D. J. Brewer and P. J.
McEwan (Eds.), Economics of
Education (pp. 33-36). Oxford:
Elsevier.
Pascarella, E. T., & Terenzini, P. T.
FALL 2015, VOL. 30, NO. 3 31
(2005). How college affects students
(Vol. 2). San Francisco, CA: Jossey-
Bass.
Siegel, H. (1992). Education and the fostering
of rationality. In Richard Talaska (Ed.),
Critical reasoning in contemporary
culture (pp. 89–112). Albany, NY: State
University of New York Press.
Siegel, H. (1997). Rationality redeemed?
further dialogues on an educational
ideal. New York: Routledge.
Sigel, H. (2006). Educating reason: rationality,
critical thinking, and education. New
York: Routledge. (Original work
published 1988)
Solon, T. (2003). Teaching critical thinking:
The more, the better! The Community
College Enterprise, 9(2), 25-38.
Twardy, C. (2004). Argument maps improve
critical thinking. Teaching Philosophy,
27(2), 95-116.
van Gelder, T. J. (2001). How to improve
critical thinking using educational
technology. In G. Kennedy, M. Keppell,
C. McNaught, & T. Petrovic (Eds.),
Meeting at the Crossroads. Proceedings
of the 18th Annual Conference of the
Australian Society for Computers
In Learning In Tertiary Education
(ASCILITE 2001) (pp. 539–548).
Melbourne: Biomedical Multimedia
Unit, The University of Melbourne.
(http://www.medfac.unimelb.edu.au/
ascilite2001/ pdf/papers/vang%eldert.
pdf)
van Gelder, T. (2005) Teaching critical
thinking: Some lessons from cognitive
science. College Teaching, 53(1), 41-
46.
van Gelder, T., & Bissett. M. (2004).
Cultivating expertise in informal
reasoning. Canadian Journal of
Experimental Psychology, 58(2), 142-
152.
Author Information
David Wright is a Visiting Assistant Professor
at Sam Houston State University. He regularly
teaches sections of critical thinking and
has research interests in moral and political
philosophy, moral psychology, and critical
thinking. Correspondence should be addressed
to David Wright, Department of Psychology
& Philosophy, Sam Houston State University,
Campus Box 2447, Huntsville, Texas 77341-
2447 or dew019@shsu.edu.
32 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Effect Size and Critical Thinking Assessment: A Response to Wright
Donald Hatcher
Baker University
Abstract
This is a brief response to David Wright’s commentary on my paper, “Critical Thinking
Instruction: A Realistic Evaluation: The Dream vs. Reality.” Wright claims that if one looks more
closely at the literature on critical thinking (CT) assessment that the reported effect sizes for CT
instruction are quite respectable and my standards are too high. My comments will focus is on
whether effect size is both problematic and an adequate measure for assessment.
Keywords: critical thinking, critical thinking assessment, effect size
First, I commend David Wright for an
excellent summary of the major arguments
in my paper, for the lucidity of his critical
comments, and for making me think more
carefully about my prior beliefs in my paper
(2015). His presentation of my attempts at
“dream deation“ is absolutely accurate
and is focused on those points that are most
important to my argument, i.e., my concern,
in spite of the appeal of CT as an educational
ideal (Siegel, 1980. 1988, 1992, 1997),
about the effectiveness of critical thinking
(CT) instruction based on effect size gains
on standardized CT tests. I do, however,
disagree with his conclusion that my negative
conclusions are not warranted. In what
follows, I will try to elaborate more fully why
my concerns about the effectiveness of CT
instruction based on effect size may be more
warranted than Wright suggests.
It is pretty clear that Wright and I
interpret the studies to assess CT instruction
that use effect size, both mine and those in the
CT literature, differently. Wright points out
correctly that if I compare the effect size gains
of the students at Baker University (BU) who
went through our CT program to those in the
norms reported in the literature, BU students,
as well as others, do quite well (p.7). Looking
at the norms proposed by Cohen (1988) and
Lipsey, et.al. (2012), Wright concludes that my
standards are too high. For example, Cohen
claims that in the social sciences, any effect
size gain ever 0.50 is a moderate gain, and
Lipsey et al. say that research on math and
reading scores show that “an effect size gain
of 0.25…. is large.” By that standard, Cohen’s
0.50 would be a huge gain. So, according to
these studies, I am indeed applying an unduly
high standard to BU and other students. Any
effect size gains of 0.57 and above are quite
respectable.
Admittedly, for years, whenever I
reported our students’ scores to my faculty
colleagues at BU, I espoused the same
conclusion. And, as Wright does, I too would
point out that as far as effect size gains, the
students in our CT program were doing about
as well or sometimes better than any others
reported in the literature (Hitchcock, Pascarella
& Terenzini, Solon, van Gelder). But, for all of
those years that I was directing the program,
I never asked “What does an effect size gain
of approximately 0.50 mean in terms of the
number of questions answered correctly on the
posttests relative to the pretest performance?
Is that gain something of which we can really
be proud?” Or, if I were looking at the data
FALL 2015, VOL. 30, NO. 3 33
from Lipsey et al., “What does a pre to posttest
effect size gain of 0.25 mean on a test with
50 questions with standard deviation of 5?”
In fact, what an effect size of 0.25 means is
that, after a semester of instruction in math
or reading, the average student increased his
or her score by 1.25 points! Yes, it is true that
a 0.50 or less of a standard deviation gain
seems to be typical across much of academe
(Norman et al., 2003), but when I considered
the post-test scores on the tests as percentages,
the scores were all failing grades, i.e., below
60% (2015). Or, if I looked at the actual pre to
posttest numerical gains, in my judgment, they
were and are minimal. Take, for example, the
average scores on the Cornell Level Z Critical
Thinking Test (CLZ). After a year-long course
focusing on CT skills and their application to
critical papers, the average student increase
was 2.7 questions on a 52 question test. To me,
that is worrisome.
So, in both my and Wright’s papers,
because much of the evidence both justifying
and critiquing CT instruction is based on us
looking at the same effect size gains or losses,
more needs to be said about the problems I
see with using effect size to evaluate student
performance. This seems to be just another
example where two scholars in a eld can look
at the same data and arguments and disagree
(Feldman & Wareld, 2010).
First, on the positive side, the use
of effect size, the percentage of a standard
deviation in a specic testing situation,
tells us much more than simply saying
that the gains or losses were “statistically
signicant.” Effect size, as it has been used
in my paper and Wright’s response, is used
as a way of comparing outcomes, and that
is very important when assessing alternative
approaches to teaching CT. However, one
should notice that any calculated effect size
is relative to and varies with the specic
standard deviations used in the calculation.
So, interesting questions arise. When standard
deviations vary, are we always making a fair
comparison? Or are we comparing apples and
oranges? For example, as specic standard
deviations vary for two groups taking the same
test tests and getting the same results or groups
taking different tests with varying standard
deviations, so will the effect sizes. Consider,
Group A taking a standard 50-question CT
exam and the group had a standard deviation
of 3, while Group B takes the same test but has
a standard deviation of 5. However, if in both
cases the average pre to posttest gain was the
same, say 2 pts., then the effect size for Group
A would be 0.66, while the effect size for
Group B would be only 0.40. Is it fair to claim
the students in Group A developed stronger
CT skills than students in Group B or that the
methods used in Group A were better because
of the difference in effect size when the only
difference was standard deviation of the two
groups while performance was overall the
same?
In the case of the three different
tests that were used to assess BU’s Program
(the California Critical Thinking Skills Test
(CSTST), the CLZ, and the E-W), there was
signicant variance in the standard deviations
for each: CCTST = 4.2; CLZ = 5.0; and E-W
= 5.3. As a result, a 2 point gain on each test
would result in different effect sizes, e.g., 0.38
for the E-W; 0.40 for the CLZ, and 0.48 for
the CCTST. That means that students taking
the E-W would need a mean gain of 2.65 pts.
to have an effect size gain of 0.50, while those
taking the CLZ would only need to gain 2.0
pts.
Another concern when groups being
compared are using different tests is that
different tests have different numbers of
questions and this should affect the level of
posttest gains that will then affect the effect
size. For example, imagine that School A uses
the Ennis-Weir Critical Thinking Essay Test
(E-W), which (on my interpretation of the test
booklet) has a top score of 29, while School
B uses the CLZ, which has a top score of 52.
Let’s assume that the scores of each group
34 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
have a standard deviation of 5.0 and a pre to
posttest gain of 2.5 points. So, both have an
effect size of 0.50. However, is it fair to say
both made equal progress? It would seem that
it would be easier to improve 2.5 points on
a 52 question test than on one with only 29
possible points. The 2.5 pt. gain on the 29 pt.
E-W is a 9% gain, while it is only a 5% gain
for the person taking the 52 pt. CLZ. These
are the sorts of questions I should have been
asking in numerous prior papers (2006, 2009,
2011, 2013a, 2013b).
Finally, there is the issue of low versus
high pre-test scores. Imagine School A takes
the CLZ and has a pre-test average of 20, while
School B has a mean pre-test score of 30. My
experience at BU tells me that it will be harder
to increase their pre to posttest scores for
those who start at 30 relative to those whose
pretest is 20. I saw this over the years with
my philosophy majors who tended to have
relatively high pre-test scores but then, when
their effect size was compared with those
who began with lower pre-test scores, the
philosophy majors were a bit lower.
Given all of these concerns, concerns
that people more statistically sophisticated than
I may be able to address, it seemed reasonable
for me to look beyond what is considered an
acceptable effect size and question whether
students were learning the desired CT skills
at an acceptable level (Hatcher, 2015). Were
we realizing Siegel’s “educational ideal” at
an acceptable level, even under the best of
circumstances (2013a)?
Wright cites Cohen’s and Lipsey’s
data to show that my standards using effect
size are too high. In response, assuming the
studies are accurate and the issues I have just
brought up have been adequately addressed,
I would be equally suspect of the success of
instruction in math and reading as I have been
for CT instruction. If an effect size of 0.50 is
unacceptable for CT, then so is the effect size
of 0.25 they cite for reading and math.
Of course, the situation with math and
reading is very different from CT instruction.
Unlike CT, I do not think that most humans
are inherently inclined to be illiterate or
unable to do math. However, as I argued in
my paper (and van Gelder agrees), there is a
lot of evidence that humans are not rational
when it comes to deciding what to do and
believe (Ennis, 1987) or “honestly evaluating
alternatives” (Hatcher and Spencer, 2006). This
alone makes teaching critical thinking a unique
challenge, to say the least.
On a different note, at the end of my
paper, when I drew the analogy between
teaching music and critical thinking instruction
and, along the same lines, Wright cites van
Gelder for drawing the analogy between
teaching ballet and CT instruction, the point
is, just as very few students will become
virtuosos with respect to music and dancing
(especially ballet), we should expect few to
master the CT skills we try to teach. However,
even here, I think the analogy overlooks a
relevant difference that should have been
obvious. Music and dance are natural human
dispositions. We like to sing and dance, and,
in most cultures, we regularly experience
both. Music is ubiquitous in our students’
environment. We have to tell them daily to
take their ear buds out when they are in our
classes. But CT is not something with which
most students have much, if any, experience.
The culture, as put forth in the media and in
political discourse, is primarily lled with
supercial, fallacious thinking. What are billed
as arguments are merely shouting matches
rather than reasoned debate in the tradition of
Socrates. Rhetoric (or worse) is the norm rather
than reasoned judgment based on reasons
that are the fruit of honest, open-minded
inquiry. So, not only have cognitive and
social psychologists shown that humans are
innately irrational in their formation of beliefs,
not surprisingly, rationality is the exception
rather than the norm in the culture. This
makes teaching critical thinking successfully
FALL 2015, VOL. 30, NO. 3 35
extremely difcult and rare. Indeed, it is a
special student who truly excels. When one
does, I am reminded of the glowing description
Theodorus gives of Theaetetus at the opening
of that dialogue and how rare Socrates claims
such students are (Plato, 1973,144a).
So, given these complexities with using
effect-size, Wright is correct when he says
that we need “to better specify the standards
by which CT instruction should be evaluated”
(28). If effect size is used to determine the
relative effectiveness of CT instruction, we
should make sure that the groups that are
compared had similar pre-test scores, are
given the same standardized CT test, and the
standard deviation that is used is the same for
all groups—which in reality it seldom is.
Under such ideal conditions, what
would an acceptable effect size gain in just
one CT class look like? That is hard to say. It
would be great if we had charts for CT like
those the ACT has correlating ACT scores in
math and English to certain percentiles. If we
did, one might even consider saying no student
passes the CT class without at least scoring
in the 60th percentile of the standardized
test. Unfortunately, as far as I know, such
correlations do not exist.
Finally, Wright is also not so sanguine
with the positive implications I claim
accompany students who take CT courses,
e.g., the CT courses promote the attitude of
fallibilism, “can go at least some way toward
helping students distinguish good from bad
arguments, promotes freedom and autonomy,
is something that employers clearly seek
in who they hire, and forces students to do
be clear, which is a pre-condition for more
serious critical thought” (28). I suppose
whether these implications actually occur is
an empirical matter and evidence tends to
be lacking. However, clarity is a natural by-
product of argument evaluation because one
cannot evaluate fairly what one does not rst
understand. But, from my own experience (see
Hatcher 2013a and 2013b), there are very few
college courses that focus in any systematic
way on argument evaluation or ask students
to “honestly evaluate alternatives.” The idea
of requiring all papers to show that students
have “honestly evaluated alternatives” and
include objections and replies is foreign to
many teachers and so to students. So, in my
judgment, CT courses, for all their problems,
still have a special place in the curriculum.
So, to conclude, I think David Wright is
correct when he points out that, if we use effect
size as the measure, when compared to others
(Hitchcock, Solon, van Gelder) , the BU CT
program was not all that bad. However, I think
the effect size data he shares points out that
the efforts at CT instruction (as well as math
and reading) are getting minimal results. So,
in reality, the evangelical fervor for changing
the educational culture through teaching CT, a
fervor many of us shared in the 80s, needs to
be tempered. We did not realize what we were
up against. As students of Plato, we should
have remembered how Socrates was received.
Nonetheless, I believe we need to keep up the
ght, but realize the successes may be few and
the student response less than welcoming. I am
often reminded of a quote by Don Marquis that
Ralph Johnson shared with my BU colleagues
years ago: ‘If you make students think they are
thinking, they will love you. If you actually
make them think, they will hate you.”
What would ultimately help move
things forward, in spite of the problems I have
described, is, if more CT teachers carefully
described their courses and teaching methods,
did pre and post testing using the same
standardized and professionally-validated CT
tests, and carefully recorded their successes
and failures. We might then nd ways to
improve student performance, as well as
establish the needed norms to judge success.
However, this may never happen. I remember a
conversation with a now retired math professor
who said he had done pre and post testing
with math students and gave it up because the
36 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
results were too depressing. The same may
also be true for much CT instruction.
References
Cohen, J. (1988). Statistical power analysis
for the behavioral sciences (Second
ed.). Hillsdale, NJ: Lawrence Erlbaum
Associates.
Ennis, R.H. (1987). A conception of critical
thinking—with some curriculum
suggestions. APA Newsletter on
Teaching Philosophy. Summer, 1-5.
Ennis, R. H., Millman, J., & Tomko, T. N.
(2004). Cornell critical thinking
tests, level X and level Z manual,( 4th
ed.) Seaside, CA.: Critical Thinking
Company.
Ennis, R.H. & Weir, E. (1985). The Ennis-
Weir critical thinking essay test. Pacic
Grove, CA: Midwest Publications.
Facione, P. & Facione, N. C. (1994). The
California critical thinking skills test:
Form A and B test manual. Milbrae,
CA: The California Academic Press.
Feldman, R. & Wareld, T. (Eds.). (2010).
Disagreement. New York: Oxford
University Press.
Hatcher, D. L. (2006). Stand-alone versus
integrated critical thinking courses.
Journal of General Education, 55(3-4),
247-272.
Hatcher, D. L. (2010). Assessing critical
thinking skills: A fteen-year
perspective. Critical thinking education
and assessment. Eds. Jan Sobcan
and Leo Groarke. Althouse Press,
University of Western Ontario, London,
Ontario.
Hatcher, D. L. (2011) Which test? Whose
scores? Comparing standardized
critical thinking tests. New Directions
in Institutional Research. Ed. Jeremy
Penn. San Francisco: Jossey-Bass.
Hatcher, D. L. (2013a). Is critical thinking
across the curriculum a plausible
goal?” Mohammed, D., & Lewiński,
M. (Eds.). Virtues of Argumentation.
Proceedings of the 10th International
Conference of the Ontario Society for
the Study of Argumentation (OSSA),
22-26 May 2013. Windsor, ON: OSSA,
pp. 1-16.
Hatcher, D. L. (2013b). Reections on
critical thinking: Theory, practice
and assessment. INQUIRY: Critical
Thinking Across the Disciplines, 28(1),
4-24.
Hatcher, D. L. (2015). Critical thinking
instruction: a realistic evaluation: the
dream vs. reality. INQUIRY: Critical
Thinking Across the Disciplines, 30,
(3), 4-19
Hatcher, D. L. & Spencer. L. A. (2006).
Reasoning and writing: From critical
thinking to composition. 3rd ed.
Boston: American Press. (Original
work published by Copley Publishing,
1992 and Rowman and Littleeld,
1993.)
Hitchcock, D. (2004). The effectiveness of
computer-assisted instruction in critical
thinking. Informal Logic, 24(3), 183-
217.
Lipsey, M., Puzio, K., Yun, C., Hebert, M.,
Steinka-Fry, K., Cole, M., Roberts, M.,
Anthony, K., and Busick, M. (2012).
Translating the statistical representation
of the effects of education interventions
into more readily interpretable forms.
National Center for Special Education
Research (NCSER 2013-3000).
Norman, G.R., Sloan, J.A., and Wyrwich,
K. (2003). Interpretation of changes
in health-related quality of life: The
FALL 2015, VOL. 30, NO. 3 37
remarkable universality of half a
standard deviation. Medical Care, 41,
582-92.
Ortiz, C.M.A. (2007). Does philosophy
improve critical thinking skills?
(Unpublished Master’s Thesis)
University of Melbourne, Melbourne,
Australia.
Pascaralla, E.T. & Terenzini, P. T. (2005). How
college affects students (Vol. 2). San
Francisco: Jossey-Bass.
Plato. (1973). The collected dialogues of Plato.
E. Hamilton and H. Cairns, (Eds.).
Princeton: Princeton University Press.
Siegel, H. (1980), Critical thinking as an
educational ideal. Educational Forum,
45(1), 7 -23.
Siegel, H. (1988). Educating reason. New
York: Routledge, Inc.
Siegel, H. (1992). Education and the fostering
of rationality. In Richard Talaska (Ed)
Critical reasoning in contemporary
culture (pp. 89–112). Albany, NY: State
University of New York Press.
Siegel, H. (1997). Rationality redeemed. New
York: Routledge, Inc.
Sobocan, J. & Groarke, L. (Eds.) (2010).
Critical thinking education and
assessment: Can higher
order thinking be tested? London, ON:
The Althouse Press.
Solon, T. (2003). Teaching critical thinking:
The more, the better! The Community
College Enterprise, 9(2), 25-38.
Van Gelder, T. & Bissett. M. (2004).
Cultivating expertise in informal
reasoning. Canadian Journal of
Experimental Psychology, 58(2), 142-
152.
Wright, D. (2015) Are we asking the right
questions about critical thinking
assessment? A response to Hatcher.
INQUIRY: Critical Thinking Across the
Disciplines, 30(3), 20-31.
Author Information
Donald Hatcher is Professor of Philosophy
in the College of Arts and Sciences at Baker
University in Baldwin City, KS, 66006.
From 1989 to 2009, he was Director of
Baker’s Liberal Arts Program, an experiment
in combining critical thinking and written
composition with the study of primary texts.
From 1988 to the present, he serves as the
Treasurer for the Association for Informal
Logic and Critical Thinking (AILACT). He
can be contacted at dhatcher@bakeru.edu
38 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Using the Critical Thinking Assessment Test (CAT) as a Model for Designing Within-
Course Assessments: Changing How Faculty Assess Student Learning
Ada Haynes, Elizabeth Lisic, Kevin Harris, Katie Leming, Kyle Shanks, &
Barry Stein
Tennessee Technological University
Abstract
This article provides a brief overview of the efforts to develop and rene the Critical thinking
Assessment Test (CAT) and its potential for improving the design of classroom assessments. The
CAT instrument was designed to help faculty understand their students’ strengths and weaknesses
using a short answer essay format. The instrument assesses a broad collection of critical thinking
skills that transcend most disciplines. The questions were deliberately designed around real-
world scenarios that did not require specialized knowledge from any particular discipline.
Various faculty who collaborated in the national dissemination of the CAT instrument found that
it was a helpful model for designing better course assessments to grade student work. Classroom
assessments modeled on the CAT emphasize more critical thinking within the discipline and less
rote retention of factual information. We describe the ongoing work to help faculty successfully
adapt the CAT to applications that can be used in each discipline’s courses to evaluate and
encourage students’ critical thinking.
Keywords: argument assessment and subjectivity, critical thinking, critical thinking assessment,
critical thinking denitions, critical thinking and human psychology
Background
The Critical thinking Assessment
Test (CAT) began as a faculty driven effort to
assess and improve critical thinking skills at
Tennessee Technological University (TTU) in
the year 2000. Subsequent funding from the
National Science Foundation (NSF) supported
the renement and national dissemination
of the instrument. The underlying goal was
to identify a subset of critical thinking skills
relevant to most disciplines that could be
evaluated using short answer essay questions.
The methods incorporated a combination of
practices in learning sciences and dynamic
assessment to measure these critical thinking
skills.
Initially, faculty from diverse
disciplines (e.g., STEM, Humanities, Social
Sciences) were asked to identify the most
important critical thinking skills that would
lead to students’ success in their discipline.
These skills were then targeted with various
questions that could reveal students’ thinking
skills using a series of deeper and deeper
question prompts. An institution’s own faculty
members would evaluate their students’
responses to these questions to provide a
better understanding of students’ strengths
and weaknesses. Providing faculty members
the opportunity to witness, rst-hand, their
students’ strengths and weaknesses was
essential if the assessment was to drive
changes in educational practices.
There were numerous challenging
problems that had to be overcome to
accomplish the goals of both assessing and
FALL 2015, VOL. 30, NO. 3 39
improving students’ critical thinking skills.
These are listed below:
• Identifying a set of skills that faculty in
most disciplines and institutions would
agree are important components of
critical thinking
• Developing questions that assess these
skills and that have high face validity
for faculty across disciplines
• Developing questions that did not
have a oor or ceiling effect so that
they could be used by a wide range of
institutions from community colleges to
more selective Ivy League universities
• Developing a consistent and reliable
way of scoring these questions that
would yield reliable scoring across
institutions
• Establishing validity and ensuring that
the instrument is not biased (i.e., by
culture, race, or gender)
• Developing a method for training/
dissemination that would ensure
reliable and consistent scoring across
institutions
• Developing an assessment that would
drive change in educational practices
• Developing an assessment that could
be used as a model for better course
assessments
The work conducted over the last 16
years has met these challenges with varying
degrees of success. The 12 underlying skills
(see Table 1) targeted by the CAT instrument
represent a cross-section of skills from which
faculty can usually nd 8 or more skills that
are relevant to their discipline. In addition,
the face validity of the questions designed to
assess these skills is high for faculty across
disciplines (Stein, et al., 2007).
The scoring process and rubric have
been rened to yield reliability between scorers
of .92. The reliability and accuracy of faculty
scoring sessions is the result of continuous
CHANGINGHOWFACULTYASSESSSTUDENTLEARNING
5
Table 1
Skill Areas Assessed by the CAT Instrument
Evaluating Information
Separate factual information from inferences.
Interpret numerical relationships in graphs.
Understand the limitations of correlational data.
Evaluate evidence and identify inappropriate conclusions.
Creative Thinking
Identify alternative interpretations for data or observations.
Identify new information that might support or contradict a hypothesis.
Explain how new information can change a problem.
Learning and Problem Solving
Separate relevant from irrelevant information.
Integrate information to solve problems.
Learn and apply new information.
Use mathematical skills to solve real-world problems.
Communication
Communicate ideas effectively.
The scoring process and rubric have been refined to yield reliability between
scorers of .92. The reliability and accuracy of faculty scoring sessions is the result of
continuous feedback from faculty, detailed training and scoring procedures, detailed
training and scoring guides, and continual monitoring of scoring accuracy across
institutional training sessions to address any discrepancies in scoring.
The questions on the CAT have been carefully evaluated for cultural fairness and
bias across the United States. When controlling for entering ACT and SAT scores, no
evidence of racial or gender bias is found (Harris et al., 2014). In addition, various
methods have been used to help evaluate validity. Converging evidence from correlations
with other measures of intellectual ability, other types of critical thinking tests, and
observations of the sensitivity of the test to the effects of high impact practices in
education designed to impact these skills have been used to support the validity of the
40 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
feedback from faculty, detailed training and
scoring procedures, detailed training and
scoring guides, and continual monitoring of
scoring accuracy across institutional training
sessions to address any discrepancies in
scoring.
The questions on the CAT have been
carefully evaluated for cultural fairness and
bias across the United States. When controlling
for entering ACT and SAT scores, no evidence
of racial or gender bias is found (Harris et al.,
2014). In addition, various methods have been
used to help evaluate validity. Converging
evidence from correlations with other
measures of intellectual ability, other types
of critical thinking tests, and observations
of the sensitivity of the test to the effects of
high impact practices in education designed
to impact these skills have been used to
support the validity of the instrument (Stein
et al., 2007). The test has been sensitive to the
effects of high impact educational practices in
both formal and informal settings that span a
semester or less (Alvarez, Taylor, & Rauseo,
2015; Carson, 2015; Gasper, Minchella,
Weaver, Csonka, & Gardner, 2012; Gottesman
& Hoskins, 2013; Rowe et al., 2015).
Using the CAT to Assess High Impact
Practices
There is increasing interest in the
scholarship of teaching and learning to
improve educational outcomes (McKinney,
2007; National Research Council, 2015).
Educators have sought to identify and assess
the impact of various educational practices
upon students’ learning of skills such as critical
thinking, problem solving, and communication.
Many of the efforts to identify educational
practices that improve students’ critical
thinking have relied on students’ self-reports
of learning gains (Anaya, 1999; Pike, 1995).
Unfortunately, there is considerable evidence
that students cannot accurately report gains in
critical thinking and other higher order skills
(Bowman, 2009; Harris, 2015; Porter, 2012).
The CAT instrument has provided
researchers with an alternative performance
measure that is sensitive to the effects of both
formal and informal educational experiences
in a single course (Alvarez, Taylor, & Rauseo,
2015; Carson, 2015; Gasper et al., 2012;
Gottesman & Hoskins, 2013). The CAT
instrument has also been useful to many
researchers who are studying the impact of
educational practices on a broader range
of critical thinking skills (e.g., identifying
alternative explanations, identifying additional
information that may be needed to evaluate
an idea or hypothesis, explaining how the best
solution to a real-world problem would change
if the problem constraints were altered) than
tests designed to evaluate a more limited set of
critical thinking skills such as formal reasoning
(Frisch, Jackson, & Murray, 2013; Rowe et al.,
2015).
Some individual courses have produced
gains in CAT scores equivalent to or greater
than an entire four-year college experience
(Harris et al., 2014). Unfortunately, most
college courses, that do not specically target
critical thinking skills, do not show any
signicant gains on the CAT. An important
question for future research is whether the
positive effects observed for high impact
practices on the CAT are additive across
multiple courses in a program of study. If the
effects are additive, then it may be possible to
dramatically improve students’ gains in critical
thinking during four years of higher education
compared to what is typically observed (Arum,
Roska, & Cho, 2011).
Need For Faculty Development in
Assessment Critical Thinking
Even though most faculty believe that
critical thinking is one of the most important
goals of higher education (Bok, 2006;
DeAngelo et al., 2009), critical thinking does
not appear to be emphasized in the methods
most courses use to assess student learning.
For example, most faculty agree that critical
FALL 2015, VOL. 30, NO. 3 41
thinking involves higher order thinking
skills and not the rote retention of factual
knowledge, however, recent responses to
the National Survey of Student Engagement
(NSSE) indicate that, throughout the country,
over 70 percent of rst year undergraduate
students and over 60 percent of senior
undergraduate students think their courses
emphasize rote retention either quite a bit
or very much (NSSE, 2015). This data is
consistent with ndings that objective tests of
factual information appear to be the primary
type of assessment used in higher education
classrooms (Kvale, 2007). This disconnect
between educational goals and assessments
indicates that even though faculty recognize
the need for improvement in students’ critical
thinking skills, they are encountering barriers
which prevent them from implementing
changes in their instructional and assessment
practices.
Figure 1. How much do courses emphasize
memorizing material.
One explanation for the widespread
emphasis on the rote retention of factual
information is that constructing a factual
knowledge test is much easier than designing
an assessment that evaluates critical thinking
skills. Most faculty have little or no training
in developing classroom assessments that
promote the development of critical thinking
skills (Fox & Hackerman, 2003; Hutchings,
2010; Petress, 2007). Assessment is frequently
an afterthought to faculty due to their lack of
training, as well as time constraints (Gayton,
2007). Faculty that receive more training are
more likely to use active learning strategies
in their courses and are more likely to use
critical thinking and real-world problem
solving assessments to determine their
students’ nal grades (Haynes et al., 2016).
Furthermore, faculty are often unaware of the
impact that their course assessments have on
student learning (Hutchings, 2010), which is
problematic since the classroom assessments
faculty use to assign grades ultimately
determine how students will invest their efforts
to learn (Gibbs & Simpson, 2004). The way
faculty assess student learning can have a
greater impact on learning than the particular
teaching pedagogy that is used (Boud &
Falchikov, 2007). Assessment guides the
students’ prioritization of what is important for
success in a particular course.
Various educators have been very
CHANGINGHOWFACULTYASSESSSTUDENTLEARNING
8
students’ critical thinking skills, they are encountering barriers which prevent them from
implementing changes in their instructional and assessment practices.
Figure 1. How much do courses emphasize memorizing material.
One explanation for the widespread emphasis on the rote retention of factual
information is that constructing a factual knowledge test is much easier than designing an
assessment that evaluates critical thinking skills. Most faculty have little or no training in
developing classroom assessments that promote the development of critical thinking
skills (Fox & Hackerman, 2003; Hutchings, 2010; Petress, 2007). Assessment is
frequently an afterthought to faculty due to their lack of training, as well as time
constraints (Gayton, 2007). Faculty that receive more training are more likely to use
active learning strategies in their courses and are more likely to use critical thinking and
real-world problem solving assessments to determine their students’ final grades (Haynes
et al., 2016). Furthermore, faculty are often unaware of the impact that their course
0
5
10
15
20
25
30
35
40
45
50
Very Little Some Quite a bit Very much
How Much do Courses Emphasize Memorizing Material
(2015 NSSE)
First Year Students Seniors
42 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
critical of higher education for doing little to
improve students’ critical thinking skills (Arum
et al., 2011; Bok, 2006; Pascarella et al., 2011).
Unfortunately, when faculty use rote retention
tests to assess student performance, they are
inadvertently encouraging students to devote
most of their time and energy to memorizing
information. Excessive reliance on factual
knowledge assessments can sabotage the
impact of active learning pedagogies and may
lead to dissatisfaction with programs of study
and negatively impact retention.
The CAT as a Tool for Faculty Development
The CAT instrument has led to
progress on overcoming what Bissell and
Lemons (2006) note as two major obstacles
to incorporating more critical thinking into
courses: (1) dening critical thinking and
(2) developing critical thinking assessments.
A variety of experts in higher education
assessment regard the CAT instrument as a
valuable assessment tool and consider it a
“creative” and “authentic” assessment that
promotes faculty involvement in assessment
(Ewell, 2010; Ewell, 2014). Many faculty
consider the CAT to be an excellent model for
how to think about assessing critical thinking
skills in a wide variety of disciplines (Lisic,
2015).
A recent national survey of faculty
around the country who had participated in
scoring the CAT indicated that seeing their
students’ responses on the CAT motivated
faculty to change their teaching pedagogy and
their course assessments. Lisic (2015) found
that participating in a single CAT scoring
session led faculty to focus more on active
learning in their classes. Lisic (2015) also
found that the more exposure that faculty had
to scoring the CAT, the more likely they were
to change their course assessments to focus
more on critical thinking. Lisic’s ndings
indicate it is more difcult for faculty to
change their course assessments than it is to
change their teaching pedagogy. These ndings
are consistent with the observation that faculty
often lack training in developing effective
classroom assessments (Gayton, 2007;
Hutchings, 2010; Petress, 2004).
During the early phases of national
dissemination, we noticed that some faculty
members who participated in the CAT training
workshops immediately recognized how
they could adapt questions from the CAT
instrument to assess student learning within
their own course content. Some examples
of these adaptations include having students
identify alternative explanations for patterns
of information such as symptoms associated
with a medical problem, historical events,
or correlations observed in the natural and
social sciences. Students would then be
asked to identify additional information that
might be needed to evaluate these alternative
explanations. These course assessments would
encourage students in their courses to practice
the critical thinking skills assessed by the CAT
while learning their discipline’s content. These
alternative assessment activities derived from
the CAT instrument provided an alternative
way of evaluating students’ mastery without
emphasizing the rote retention of factual
information (e.g., Shannon and Bennett, 2011).
As more faculty were involved in
national dissemination it became clear that
faculty needed more guidance in how to adapt
the CAT instrument as a model to develop
better course assessments that emphasized
critical thinking skills. This observation was
also supported by survey ndings from faculty
who participated in CAT scoring indicating
that they needed more training to develop
course assessments that integrated discipline
content with critical thinking (Lisic, 2015).
In fact, only 10% of faculty in one case
study were able to spontaneously develop
course assessments that incorporated skills
targeted by the CAT instrument (Haynes et al.,
2016). Faculty noted two major difculties in
modifying their course assessments, lack of
training and lack of time.
FALL 2015, VOL. 30, NO. 3 43
CAT Application Training
CAT Applications (CAT Apps) are
course assessments that allow faculty to
simultaneously assess mastery of discipline
content and critical thinking skills by providing
students with opportunities to think critically
about issues and problems in the discipline.
These assessments provide an alternative to
course assessments that focus primarily on
the rote retention of information. Faculty
from a variety of institutions thought that
incorporating CAT Apps in their courses
and providing students feedback throughout
the semester would help students improve
their critical thinking skills while mastering
discipline specic content. Initial efforts to
help faculty develop CAT Apps revealed the
need to develop a more systematic approach
that would help guide faculty through the
process (Stein et al., 2013).
The approach taken to assist faculty
in CAT App training was based, in part, on
the seminal work of Bransford, Brown, &
Cocking (1999), How People Learn, and on a
general model for problem solving (Bransford
& Stein, 1993). The resulting framework
assisted faculty in walking through the steps
of developing a CAT App for their courses by
presenting the development of effective course
assessment as a form of problem solving.
This system helps faculty tie their course
assessments with their student learning goals, a
process known as construct alignment (Biggs,
1996). The general adaptation of the IDEAL
Framework for Developing CAT Apps is
provided below.
• Identify the critical thinking
skills on the CAT that would
be relevant for the discipline
content in your course.
• Dene the learning goal(s)
– how will the discipline
information be used to think
critically about a situation or
help solve a problem.
• Explore strategies for assessing
students’ critical thinking skills
using the discipline information
by building tasks that require
students to use discipline
information to evaluate ideas or
solve problems.
• Anticipate student responses
to help develop the rubric
to evaluate the quality of
responses.
• Look back and Learn from
using the CAT App. Does the
application engage students?
Does it align with the learning
goal(s)? Does the scoring rubric
differentiate different levels of
quality?
We noticed early on that faculty who
participated in at least one CAT scoring session
found it much easier to develop CAT Apps and,
consequently, exposure to scoring the CAT
instrument became an essential element in the
overall efforts to help faculty develop CAT
Apps for their courses. We also noticed that
faculty found it especially difcult to develop
scoring rubrics that they felt comfortable and
condent in using to evaluate student responses
and to assign grades. The development of
effective scoring rubrics would ultimately
play an important role in whether CAT Apps
were deployed as meaningful assessments in
courses. Consequently, we needed to provide
clear guidance in how to develop reasonable
scoring rubrics for CAT Apps.
Initially, our efforts focused on
outlining the steps and processes that faculty
could follow to develop a CAT App. Later,
we developed a web-based interface to guide
participants through the CAT App development
process and used participant input to rene
that system. The web interface has undergone
considerable modication and is being used
44 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
in training workshops to help faculty work
through the process of developing content-
based assessments that emphasize critical
thinking skills like those evaluated in the CAT
instrument. Research has been conducted on
the impact of these CAT App training sessions.
Thus far, faculty who attended these CAT App
training sessions reported that they intended
to spend more time on active learning and
that they intended to base a larger percentage
of students’ grades on critical thinking
assessments in their courses (Haynes et al.,
2016).
The survey data discussed above is
preliminary and is based upon what changes
faculty intend to do after returning to their
home institutions. It is too early to know
whether faculty will actually follow through
and make all of these changes in their courses.
However, there is some evidence that at least
some faculty are making intended changes
after CAT App training. For example, one
faculty member modied her biology research
class to more explicitly emphasize critical
thinking and creative thinking after attending
a CAT App training. She developed CAT Apps
with her discipline specic content using real-
world scenarios that were incorporated into the
course. Students in the class demonstrated a
signicant gain on the CAT instrument and on
a rubric designed to assess course objectives
in other work and these gains were superior
to those in a control class the previous year
(Carson, 2015). We have also heard from other
faculty who are in various stages of modifying
their course assessments based on CAT App
training.
The Importance of Community
Some of the more successful efforts
to develop CAT Apps have occurred at
institutions that have sought to create and
support teaching communities with a focus
on CAT App development. Professional
collaboration is often an integral part of any
faculty development activity (Hutchings,
2010). The development and renement of
new instructional strategies is often most
effectively achieved as part of a professional
community (Eddy & Garza Mitchell, 2012).
These teaching and learning communities
of practice encourage participants to work
together toward a common goal. Collaborative
teams allow faculty to step back and view
these initiatives through the lens of multiple
perspectives and often multiple disciplines
(D’Avanzo et al., 2012).
The community-based approach to
CAT App development has been utilized at
multiple institutions as part of an effort to
broadly impact course-based assessment
strategies across departments and programs
of study. The University of Wyoming and
Northwestern University were two of the
rst institutions to bring faculty together to
collaboratively develop CAT Apps (Stein,
Haynes, & Drane, 2011; Stein, Haynes,
Skinner, & Prager, 2012). Other institutions
including Iowa State University, Florida State
University, the University of Delaware, and
Keene State College have sought to involve
faculty in similar activities. One of the biggest
obstacles to CAT App development and
deployment is the faculty time commitment
required to develop these assessment activities.
The involvement in a teaching community
not only provides additional support to help
with the development process, but it also
provides a support group with a common goal
that serves to increase motivation to complete
these assessment activities. Furthermore, these
community groups allow successes in one area
to be more easily replicated by others.
Future Directions
We continue to explore the most
effective methods for helping faculty develop
CAT Apps in their disciplines. Improving the
efciency and effectiveness of these methods
will make it easier for faculty to use course
assessments that emphasize more critical
thinking and less rote retention of factual
FALL 2015, VOL. 30, NO. 3 45
information in each discipline.
Future research should also focus
on clarifying the impact of CAT Apps on
improving students’ critical thinking skills.
We would expect that the contextual variety of
CAT Apps students are exposed to in a course
will impact the likelihood that those skills are
transferred to an interdisciplinary tool like the
CAT instrument (Bransford, 1979; Bransford
et al., 1999). As previously stated, it is also
important to see if the effects of improving
students’ critical thinking skills are additive
across successful high impact courses that
emphasize critical thinking. If the effects are
additive, we could expect that it is possible to
make much greater gains in critical thinking
over a four-year college experience than is
currently achieved.
The growing interest in developing
better discipline assessments (AAC&U, 2016)
can have a meaningful impact on student
learning. However, it is important to determine
(1) whether the positive impact of using these
new course assessments can be corroborated by
other performance measures and (2) whether
the critical thinking skills transfer beyond the
connes of the material covered in the course.
In order to make these determinations, many of
the faculty who are developing their own CAT
Apps are using the CAT instrument to validate
the gains their students are making in critical
thinking and to show that those skills can be
transferred beyond the discipline content.
References
Alvarez, C. M., Taylor, K. A., & Rauseo,
N. A. (2015). Creating thoughtful
salespeople: Experiential learning
to improve critical thinking skills
in traditional and online sales
education. Marketing Education
Review, 25(3), 233–243. doi:
10.1080/10528008.2015.1044859
Anaya, G. (1999). College impact on student
learning: Comparing the use of self-
reported gains, standardized test
scores, and college grades. Research
in Higher Education, 40(5), 499–526.
doi:10.1023/a:1018744326915
Arum, R., Roksa, J., & Cho, E. (2011)
Improving undergraduate learning:
Findings and policy recommendations
from the SSRC-CLA Longitudinal
Project. Retrieved from http://www.
ssrc.org
Biggs, J. (1996). Enhancing teaching through
constructive alignment. Higher
Education, 32, 347-364.
Bissell, A. N., & Lemons, P. P. (2006). A new
method for assessing critical thinking
in the classroom. BioScience, 56(1),
66-72.
Bok, D. (2006). Our underachieving colleges:
A candid look at how much students
learn and why they should be learning
more. Princeton, NJ: Princeton
University Press.
Boud, D., & Falchikov, N. (2007.)
Introduction: Assessment for
the longer term. In D. Boud &
N. Falchikov (Eds.), Rethinking
Assessment in Higher Education:
Learning for the Longer Term (pp.
3-13). New York, NY: Routledge.
Bowman, N. A. (2009). Can 1st-year
college students accurately report
their learning and development?
American Educational Research
Journal, 47(2), 466–496.
doi:10.3102/0002831209353595
Bransford, J. D. (1979). Human cognition:
Learning, understanding and
remembering (p. 181). Belmont, CA:
Wadsworth.
Bransford, J. Brown, A. & Cocking (2000).
How people learn. Washington, D.C.:
National Academy Press.
46 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Bransford, J. & Stein, B. (1993). The IDEAL
problem solver. New York: Freeman.
Carson, S. (2015). Targeting critical thinking
skills in a rst-year undergraduate
research course. Journal of
Microbiology & Biology Education,
16(2), 148. doi:10.1128/jmbe.v16i2.935
D’Avanzo, C., Anderson, C. W., Harley, L.
M., & Pelaez, N. (2012). A faculty-
development model for transforming
introductory biology and ecology
courses. BioScience, 62(4), 416–427.
doi:10.1525/bio.2012.62.4.12
DeAngelo, L., Hurtado, S., Pryor, J. H., Kelly,
K. R., Santos, J. L., & Korn, W. S.
(2009). The American college teacher:
National norms for the 2007–2008
HERI faculty survey. Retrieved from
http://www.heri.ucla.edu
Eddy, P. L., & Garza Mitchell, R. L. (2012).
Faculty as learners: Developing
thinking communities. Innovative
Higher Education, 37(4), 283–296.
doi:10.1007/s10755-011-9202-z
Ewell, P. (2010). Foreword to opening
doors to faculty involvement in
assessment. National Institute for
Learning Outcomes Assessment.
Champaign, IL: University of
Illinois. Retrieved from: http://www.
learningoutcomeassessment.org
Ewell, P. (2014). The “quality agenda:” An
overview of current efforts to examine
quality higher education. A discussion
paper prepared for the American
Council on Education (ACE). Retrieved
from http://www.acenet.edu/
Fox, M. A., & Hackerman, N. (2003).
Committee on Recognizing,
Evaluating, Rewarding, and
Developing Excellence in Teaching of
Undergraduate Science, Mathematics,
Engineering, and Technology.
Frisch, J. K., Jackson, P. C., & Murray, M.
(2013). WikiED: Using Web 2.0
Tools to Teach Content and Critical
Thinking. Journal of College Science
Teaching 43(1), 70–80. doi: 10.2505/4/
jcst13_043_01_70
Gasper, B. J., Minchella, D. J., Weaver, G.
C., Csonka, L. N., & Gardner, S. M.
(2012). Adapting to osmotic stress
and the process of science. Science,
335(6076), 1590–1591. doi:10.1126/
science.1215582
Gayton, E. (2007.) “Introduction: Why these
assessment opportunities make sense
in a world where assessment of factual
knowledge has taken hold.” In R. J.
Mazeske & B. A. Mazeske (Eds.).
Beyond tests and quizzes: Creative
assessments in the college classroom
(pp. xvi-xxiv). San Francisco, CA:
Jossey-Bass.
Gibbs, G., & Simpson, C. (2004). Conditions
under which assessment supports
students’ learning. Learning and
teaching in higher education, 1(1),
3–31.
Gottesman, A. J., & Hoskins, S. G. (2013).
CREATE cornerstone: Introduction to
scientic thinking, a new course for
STEM-interested freshmen, demysties
scientic thinking through analysis of
scientic literature. CBE-Life Sciences
Education, 12(1), 59–72. doi:10.1187/
cbe.12-11-0201.
Harris, K. (2015). Assessing course impacts
on critical thinking: The relationships
between self-reported learning
gains and performance (Doctoral
dissertation). Retrieved from Proquest
Digital Dissertations. (3705421).
Harris, K., Stein, B., Haynes, A., Lisic, E., &
FALL 2015, VOL. 30, NO. 3 47
Leming, K. (2014). Identifying courses
that improve students’ critical thinking
skills using the CAT instrument: A
case study. Proceedings of the Tenth
Annual International Joint Conference
on Computer, Information, Systems,
Sciences, & Engineering. Retrieved
from www.reserachgate.net.
Haynes, A., Lisic, E., Goltz, M, Stein, B., &
Harris, K. (2014). CAT-Apps: Moving
beyond just assessing to improving
critical thinking skills. In press.
Hutchings, P. (2010). Opening doors to faculty
involvement in assessment. National
Institute for Learning Outcomes
Assessment. Champaign, IL: University
of Illinois.
Kvale, S. (2007). Contradictions of assessment
for learning in institutions of higher
learning. In D. Boud & N. Falchikov
(Eds.), Rethinking assessment in higher
education: Learning for the longer
term, (pp. 3–13). New York, NY:
Routledge.
Lisic, E. S. (2015). Creating change:
Implementing the critical thinking
assessment test (CAT) as faculty
development to improve instruction
(Doctoral dissertation). Retrieved
from Proquest Digital Dissertations.
(3722968).
McKinney, K. (2007). Enhancing learning
through the scholarship of teaching and
learning: The challenges and joys of
juggling. Boston: Anker Publishing.
National Research Council (2015). Reaching
Students: What Research Says About
Effective Instruction in Undergraduate
Science and Engineering. Washington,
DC: The National Academies Press.
National Survey of Student Engagement.
(2015). Annual Results. Retrieved from
http://nsse.iub.edu/html/annual_results.
cfm
Pascarella, E.T., Blaich, C., Martin, G.L., &
Hanson, J.M. (2011). How robust are
the ndings of Academically Adrift?
Change: The Magazine of Higher
Learning, 43(3), 20–24. Retrieved from
http://www.changemag.org
Petress, K. (2004). Critical thinking: An
extended denition. Education, 124(3),
461–467. Retrieved from http://www.
projectinnovation.biz/education
Pike, G. R. (1995). The relationship between
self-reports of college experiences
and achievement test scores. Research
in Higher Education, 36(1), 1–21.
doi:10.1007/bf02207764
Porter, S. R. (2012). Self-Reported Learning
Gains: A Theory and Test of College
Student Survey Response. Research
in Higher Education, 54(2), 201–226.
doi:10.1007/s11162-012-9277-0
Rowe, M. P., Gillespie, B. M., Harris, K. R.,
Koether, S. D., Shannon, L. Y., & Rose,
L. A. (2015). Redesigning a general
education science course to promote
critical thinking. CBE Life Sciences
Education, 14(3). doi:10.1187/cbe.15-
02-0032
Shannon, L. J. Y., & Bennett, J. (2012). A case
study: Applying critical thinking skills
to computer science and technology.
Information Systems Education
Journal, 10(4), 41–48. doi: 10.1109/
icemt.2010.5657674
Stein, B., Haynes, A., & Drane, D. (2011).
Getting faculty involved in assessing
and improving critical thinking.
Proceedings of the Higher Learning
Commission, Chicago, USA, 29-33.
Stein, B., Haynes, A., Light, G., Drane, D.,
Skinner, M., & Prager, E. (2013).
48 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Engaging faculty in assessing and
improving students’ critical thinking.
Panel presentation presented at the
meeting of the International Society
for the Scholarship of Teaching and
Learning, Raleigh, NC.
Stein, B., Haynes, A., Skinner, M., & Prager
E. (2012). What the CAT offers for
assessment: Engaging faculty in
assessing and improving students’
critical thinking skills. Presented at
the meeting of the Reinvention Center,
Arlington, VA.
Stein, B., Haynes, A., Redding, M., Ennis,
T., & Cecil, M. (2007). Assessing
critical thinking in STEM and beyond.
Innovations in E-learning, Instruction
Technology, Assessment, and
Engineering Education, 79–82.
doi:10.1007/978-1-4020-6262-9_14
Author Information
Dr. Ada Haynes is Professor of Sociology and
Co-director of The Center for Assessment
and Improvement of Learning at Tennessee
Tech University, and she may be contacted
at ahaynes@tntech.edu. Dr. Elizabeth Lisic
is Assistant Director of The Center for
Assessment and Improvement of Learning
at Tennessee Tech University, and she may
be contacted at elisic@tntech.edu. Dr. Kevin
Harris is Associate Director of The Center for
Assessment and Improvement of Learning
at Tennessee Tech University, and he may
be contacted at kharris@tntech.edu. Katie
Leming is a Ph.D. student at Tennessee
Tech University, and she may be contacted
at kpierce@tntech.edu. Kyle Shanks is a
Ph.D. student at Tennessee Tech University,
and he may be contacted at kshanks@tntech.
edu. Dr. Barry Stein is Professor and Chair of
Counseling and Psychology and Co-director of
The Center for Assessment and Improvement
of Learning at Tennessee Tech University, and
he may be contacted at bstein@tntech.edu.
FALL 2015, VOL. 30, NO. 3 49
Evidence-Based Critical Thinking Exercise: Shaping Tomorrow’s Dental Choices
Pauletta G. Baughman, University of Louisville School of Dentistry,
Gustavo M.S. Oliveira, Department of Operative Dentistry, University of North Carolina,
Chapel Hill, USA
Elizabeth M. Smigielski, Kornhauser Health Science Library, University of Louisville,
Vida M. Vaughn, Kornhauser Health Science Library, University of Louisville
Abstract
The objective was to educate rst-year dental students on how to appraise new dental treatments
by applying critical thinking (CT) and evidence-based dentistry (EBD) skills. To facilitate this
task, we utilized a learning exercise involving a simulated ofce visit by a dental pharmaceutical
representative. The simulated ofce sales call was conducted after instruction by dental school
faculty and clinical librarians on EBD and CT principles. Students’ critical thinking and evi-
dence-based practice skills were tested using a validated critical thinking assessment tool and a
rubric-based written assignment. Results showed that ninety-one percent of students demonstrat-
ed a high/positive response. Students agreed that the exercise helped them to consider multiple
perspectives in subject matter. The majority of students also scored high/positive in understand-
ing the components of a clinical question employing the PICO format. Students agreed that the
instruction received supported their ability to demonstrate critical thinking skills. Eighty percent
indicated instruction as having a high/positive impact on navigating complex clinical questions.
The authors concluded that simulated ofce visit plus explicit instruction in EBD principles im-
proved rst year dental students’ CT skills.
Keywords: dental education, critical thinking, evidence based dentistry, EBD
I. Introduction
The American Dental Association
Commission on Dental Accreditation (CODA)
has established Accreditation Standards for
Dental Education Programs (http://www.
ada.org/~/media/CODA/Files/2016_predoc.
pdf?la=en). Standard 2-9 requires that
“graduates must be competent in the use
of critical thinking and problem-solving,
including their use in the comprehensive care
of patients, scientic inquiry and research
methodology” (Association, 2010). Since this
standard was only recently introduced, there
is not yet a consensus regarding how to teach
and assess critical thinking. The commission
also established Standard 5-2, which requires
the use of evidence-based practice in clinical
decision making (Association, 2010). This
standard requires an approach to oral health
care that integrates (1) the best available
evidence, (2) the dentist’s individual clinical
experience, and (3) the values and expectations
of the patient (D. L. Sackett, Rosenberg, Gray,
Haynes, & Richardson, 2007). This triad of
elements comprises evidence-based dentistry
(EBD). The process of evidence-based practice
involves ve key skills that we present
as the ve A’s: asking a clinical question,
acquiring evidence, appraising the evidence,
applying the evidence to the patient’s care,
and assessing the outcomes (David L Sackett,
Straus, Richardson, Rosenberg, & Haynes,
50 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
2000).
The development of critical thinking
skills is necessary for successful evidence-
based practice. In fact, critical thinking is a
foundational competency of evidence-based
practice. It sets standards on the process of
thinking and decision-making to “reduce bias
and distortion and increase the completeness
of available information (Rousseau & Gunia,
2016). The rst step in attempting to nd
an evidence-based answer is to formulate
an answerable clinical question, taking into
consideration the specic factors of the case.
The Miniature Guide to Critical Thinking:
Concepts and Tools describes the ideal
traits possessed by a well-cultivated critical
thinker. Three of these are closely related to
the process of forming a clinical question:
formulating questions clearly and precisely,
gathering and assessing relevant information,
and coming to well-reasoned conclusions and
solutions (Paul & Elder, 2001). University
of Louisville denes critical thinking as “the
intellectually disciplined process of actively
and skillfully conceptualizing, applying,
analyzing, synthesizing, and/or evaluating
information gathered from, or generated by,
observation, experience, reection, reasoning,
or communication, as a guide to belief and
action” (Center, 2015). This denition was
developed to cultivate an environment in
which dialog could occur across all disciplines
within a consistent framework.
The application of critical thinking
and EBD in the dental education curriculum
is relatively recent. Examples of effective
and interactive teaching methods are still
scarce in the dental literature. At University
of Louisville School of Dentistry, critical
thinking and EBD are introduced to students
in their rst year of study in two courses:
Introduction to Clinical Dentistry 1 (ICD1)
and Evidence-based Dentistry. Students are
exposed to a critical thinking framework
developed from Paul and Elder that has three
sorts of components. There are ten intellectual
standards—items such as clarity, accuracy,
depth, relevance, signicance etc.--then there
are eight elements of reasoning—purposes,
questions, inferences, assumptions, concepts,
etc.—and, nally, eight intellectual traits—
personal qualities such as intellectual humility,
intellectual courage, and condence in
reason (see the Critical Thinking Community
website at http://www.criticalthinking.org/
pages/critical-thinking-where-to-begin/796).
Evidence-based dentistry principles are then
introduced with the Five A’s: asking a focused
clinical question, acquiring the evidence,
appraising the evidence, applying the evidence
to their circumstance/practice, and assessing
the outcome.
In a collaboration between the
courses, an active-learning lesson based on a
simulation of a routine clinical experience was
developed. The objective of the exercise was
the practical application of critical thinking and
EBD concepts to a real-life scenario during
which a dental industry sales representative
presented tooth-whitening products. The aim
of this study was to assess the student’s ability
to apply critical thinking skills through an
evidence-based dentistry exercise, based on
an active learning method developed at our
university.
II. Methods
Exemption for this study was sought
and received from the university’s Institutional
Review Board on Human Subjects, study
number 14.0046. Study participants included
rst-year dental (D1) students enrolled in
Introduction to Clinical Dentistry. Dental
faculty introduced students to a curriculum
emphasizing principles from the Paul-Elder
framework of critical thinking and EBD
concepts. As previously stated, the rst stage
of any evidence-based practice process is the
creation of an answerable question. A well-
formulated question will facilitate the search for
evidence by incorporating the PICO (patient,
intervention, comparison, and outcome) model.
FALL 2015, VOL. 30, NO. 3 51
For example, do plastic coatings protect pediatric
teeth from decay? The clinical components of
this question are as follows: (P) – children, (I)
– Plastic coatings or sealants, (C) – Standard
Practice, and (O) – prevention of decay.
Additional EBD concepts concerned
the types of research studies, the hierarchy of
evidence, and instruction on acquiring the best
available evidence (Paul & Elder, 2001). The
evidence hierarchy reects the relative authority
of different research methods of biomedical
research. The outcomes of tightly structured
intervention studies, such as randomized control
7
Table 1
Learning Critical Thinking Inventory-A (LCTI-A) for students.
Note: Data adapted from Assessing teaching critical thinking with validated critical
thinking inventories: the learning critical thinking inventory (LCTI) and the
teaching of the critical thinking Inventory (LCTI) (van Zyl et al., 2013)
52 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
trials, possess higher empirical strength than the
narrative design of patient case reports.
A week following the overview of
evidence-based and critical thinking principles
in both courses, a dental sales representative
delivered a presentation on an over-the-counter
whitening product to the ICD1 class. This
simulated a typical ofce visit from a sales
representative. After the presentation, using
a clinical scenario involving an over-the-
counter tooth whitening product, librarians
reviewed the elements of a clinical question
based on the PICO method and demonstrated
the American Dental Association® EBD
webpage (Association, 2014), TRIPdatabase™
(Database, 2015), and PubMed Clinical
Queries (Information, 2015), an evidence
based search lter component of PubMed.
Students then practiced using these resources
and located evidence-based literature on over-
the-counter tooth whitening products. The
results of these searches were used in a written
assignment in which students were required
to summarize their ndings of the literature,
to refute or support the sales representatives’
claims, and guide their clinical practice
regarding over-the-counter whitening products.
Students’ self-reported subjective
measurement of critical thinking skills was
assessed through a validated critical thinking
inventory, while their application of EBD
principles and skills was assessed through the
written assignment.. The validated Learning
Critical Thinking Inventory (LCTI) is a Likert
scale questionnaire consisting of two parts:
questions that assess the students’ opinions of
their own critical thinking skills, and questions
that assess their opinions of the instructor’s
ability to aid students in developing critical
thinking skills (See Table 1) (van Zyl, Bays,
& Gilchrist, 2013). Permission was obtained
by the authors to use the student section of the
inventory. Upon completion of the exercise,
students had the option to complete the
inventory. Students were given the opportunity
to obtain extra credit points upon completion
of the survey and written assignment.
To measure the students understanding
of how to apply EBD principles, participants
were required to write a paper that indicated
the use of said principles as they related to the
clinical question presented in the workshop.
Students were scored on three factors: (1)
acquisition and assessment of the evidence
(2) formation of a clearly stated and logical
recommendation for or against application of
the product in dental practice (3) assessment
on the usefulness of the exercise in their
practice. The writing assignment was done
independently, outside of class.
Survey results were calculated using
the Blackboard software (Blackboard Inc.,
Washington, D.C., USA), and analyzed by
grouping responses in the following categories:
Likert points 1 and 2, were considered a
low/negative response; Likert point 3 was
considered neutral, Likert points 4 and 5
were considered a high/positive response.
One course instructor graded all written
assignments according to a previously
developed rubric (Table 2).
III. Results
Of 120 students, 108, or 90%
completed the survey (n=108). Responses to
all questions were rated high/positive, with
percentages ranging from 79.63% to 92.59%
across all eleven questions (Table 3). Ninety
percent of students rated four questions - Q4,
Q8, Q10, and Q11 - as high/positive. The
lowest scoring questions - Q2 and Q7 - were
still rated high/positive, with percentages of
79.63% and 80.56%, respectively
Of the 119 students who completed the
written assignment, 118 (99.12%) scored 90-
100%. One student scored 88%.
IV. Discussion
As the dental education literature is
fundamentally devoid of research related to
FALL 2015, VOL. 30, NO. 3 53
the cognitive components of critical decision
making (Hendricson et al., 2006), the goal of
this study was to assess the students’ abilities
to comprehend and apply the principles
of critical thinking introduced through an
active-learning method, embedded within an
evidence-based dentistry curriculum.
Survey results were positive. Students
rated all eleven questions in the high range,
Likert scale 4 or 5, with scores ranging from
80% to 93%, across the questions. Questions
4 (91%), 8 (93%), 10 (91%), and 11 (91%)
were the highest scoring questions. Ninety
percent or more of the students rated these
questions in the high range, Likert scale 4 or
5. The high scores likely reect the degree
of varied, interactive, and hands-on activities
9
Table 2
Scoring rubric for the Evidence-based Dentistry written assig
54 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
used throughout the four-hour class. Questions
2 (80%) and 7 (81%) were the lowest scoring
questions. These questions are the primary
focus of this analysis.
Question 4 (91%) measured students’
opinions about their ability to consider
multiple perspectives. The students’ responses
may represent several aspects of their thinking:
to accept or refute the sales representative’s
claims, the multiple perspectives of the
evidence-based practice triad (clinical
expertise, the best-available evidence, and the
values of the patient), or the use of multiple
sources from which to acquire evidence.
Critically examining the sales claim forms
the crux of the exercise: nding evidence to
support or refute those claims as they apply to
clinical care. The high/positive results obtained
11
Table 3
Results of the LCTI-A for students expressed in percentages (n=119).
FALL 2015, VOL. 30, NO. 3 55
supports the idea that students are willing
to translate science, i.e., the best available
evidence, into practice. Either to reinforce it,
or to change it. This mindset is positive toward
EBD, and we should consider that EBD has
been negatively regarded by some (Chambers,
2010).
The emphasis in dental
education is on learning content and
developing skills. It is necessary for students
to understand early in their training that
critical thinking expertise has a valuable
role in evidence-based practice since that
practice is dependent on presenting a well-
formed clinical question. They also need to
understand early in their education that their
expertise has a valuable role in evidence-based
practice, depending on that clinical question,
since their expertise may allow them to accept
or reject relevant evidence. However, their
expertise with the evidence is only one-third
of the triad, with the other two parts being
the dentist’s individual clinical experience,
and, importantly, the values and expectations
of the patient. Evidence-based dentistry is
not “cookbook dentistry” if practitioners are
mindful of the other two parts of the triad
when applying the evidence (D. L. Sackett
et al., 2007). Similarly, students will be
stronger critical thinkers and evidence-based
practitioners if they rely on a variety of sources
for evidence, rather than falling into the habit
of drawing their information from only one
resource.
Question 10 (91%), assessed the
instructors’ efforts to encourage the student
in applying their insights to new or other
situations. This is related to question 4,
“Consider multiple perspectives in my subject
matter,” which emphasizes the students’ ability
to assess complex concepts. The critical
thinking and evidence-based practice curricula
are designed to introduce and encourage
students to apply what they have learned to
new situations, namely, their clinical practice.
Using a sales representative’s pitch as the basis
for the exercise exposes students to a novel
aspect of dental practice and forces them to
apply their newly acquired evidence-based
practice skills to an area with which they likely
have little experience.
The interactive nature of the exercise
may justify the high/positive score for
question 11, which assessed how often the
instructors made the students feel engaged in
class. The complement of this is in Question
8 (93%) which assessed the degree to which
the instructors supported the students in
their efforts to show good thinking. This
suggests that students recognized that the
encouragement and reinforcement offered
by the instructors helped them succeed.
The complexity of teaching methods and
signicant classroom time - a total of 5 hours
devoted to this curriculum - reects the value
the instructors place on the course content.
The combined results of Questions 8 and 11
indicate a strong positive correlation between
the instructors’ investment and the students’
self-assessed understanding of the critical
thinking process.
Although scores for Questions 2 (80%)
and 7 (81%) were still in the high/positive
range, they were comparatively the lowest
scoring responses. Question 2 measures the
degree to which the instructor or course helped
students know what a clear question is. The
ability to identify an issue and create a clinical
question using the PICO framework is a skill
that may be more difcult than is apparent
(Lai, 2009). The lower scores for question
may reect some degree of confusion with the
PICO format or the process of developing a
PICO question. This may suggest the need for
more in-depth instruction on using the PICO
method.
The second part of the survey, the
Teaching Critical Thinking Inventory-A
(TCTI-A) (van Zyl et al., 2013), was not used
since the focus of this study was on student
performance. One line of thought argues that
56 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
instructors report teaching critical thinking but,
in actuality, they are teaching the students their
opinions (Paul & Elder, 2001). Another line
of thought argues that instructors believe they
are teaching higher level reasoning and critical
thinking skills but, in fact, they are teaching
facts and concepts, content that requires the
lowest form of cognition (Gardiner, 1994).
Thus, the students are told what to think and
not taught how to think. Or, at minimum,
they are given no consistent tools with which
to think. Perhaps the use of this part of the
survey and continued classroom clarity and
identication of critical thinking would reveal
more opportunities for faculty improvement
(Behar-Horenstein, Mitchell, & Dolan, 2005).
The writing assignment was both a tool
to assess the application and understanding
of core EBD concepts, and, in itself, an
exercise in critical thinking. Implementing
the critical thinking skills used to assess the
clinical question, students had the opportunity
to expand on the process by demonstrating
in essay form how those principles would,
in turn, effect the decision-making process
in their practice as dentists. The scores of
the writing assignment were also high, with
only one student scoring less than 90%.
This suggests that the majority of students
successfully applied critical thinking skills and
EBD concepts, as required by the assignment.
The assignment encompasses the EBD steps
of asking, acquiring and appraising. Despite
the possible confusion regarding the PICO
method, as suggested by one of the questions,
students were able to formulate a PICO
question based on the clinical scenario in class.
This may be due to the instructors leading the
students through the process. The exercise
did not test if students could develop a PICO
question unaided. Based on the requirement
to submit citations for their sources, students
were able to use the demonstrated sources
to acquire information. Students’ ability to
logically and clearly argue in favor of or
against the sales person’s claim indicates
that they were able to critically appraise their
evidence for this assignment.
The authors would also like to
acknowledge the limitations of the present
study. The writing assignment was graded
subjectively by one person, which exposes it to
bias. Blinded grading by several people would
improve the inter-rater reliability of the study.
As discussed, the second part of the survey,
Teaching Critical Thinking Inventory-A, which
focuses on teaching performance, was not
used.
Further research is needed to
determine the effectiveness of this active-
learning exercise. The survey applied in
our study could be used throughout the four
years, in a variety of classes, to demonstrate
progression of critical thinking from novice
to expert (Johnsen, Finkelstein, Marshall, &
Chalkley, 2009). Moreover, a longitudinal
study would be benecial to determine if the
skills developed through dental school will
be successfully applied throughout the dental
careers of the students, if there is continued
self-directed learning. If we genuinely want
our students to apply their critical thinking
skills as they progress through dental school,
so that they become instinctive users of EBD,
then more work is needed to ensure that we -
as educators - are modeling instinctive critical
thinking through EBD, and giving students the
skills they need eventually to do likewise.
V. Conclusion
Within the limitations of the present
study, our ndings showed overall high/
positive scores in regard to the student’s
ability to apply critical thinking skills through
an evidence-based dentistry exercise. This
positive response likely reects the degree
of interactive activities used throughout the
active-learning exercise, based on a simulation
of a routine clinical experience, developed at
the University of Louisville. It also emphasizes
the importance of a curriculum embedded
FALL 2015, VOL. 30, NO. 3 57
with principles of critical thinking and EBD
concepts, and the benet of using a clinical
scenario and written assignment, which are
practical applications of critical thinking and
EBD skills. New information has been gained
from this work, but it is clear that further
research on the cognitive components of
clinical decision-making is needed.
References
Association, A. D. (2010). Commission on
Dental Accreditation. Accreditation
standards for dental education
programs. Chicago, ADA, December
l985,
Association, A. D. (2014). ADA Center for
Evidence Based Dentistry. Retrieved
from http://ebd.ada.org/en
Behar-Horenstein, L. S., Mitchell, G. S., &
Dolan, T. A. (2005). A case study
examining classroom instructional
practices at a U.S. dental school. J Dent
Educ, 69(6), 639-648.
Center, U. o. L. D. (2015). What is Critical
Thinking? Retrieved from http://
louisville.edu/ideastoaction/about/
criticalthinking/what
Chambers, D. W. (2010). Evidence-based
dentistry. J Am Coll Dent, 77(4), 68-80.
Database, T. (2015). TRIP.
Gardiner, L. F. (1994). Redesigning Higher
Education: Producing Dramatic Gains
in Student Learning. ASHE-ERIC
Higher Education Report No. 7: ERIC.
Hendricson, W. D., Andrieu, S. C., Chadwick,
D. G., Chmar, J. E., Cole, J. R., George,
M. C., . . . Kalkwarf, K. L. (2006).
Educational strategies associated with
development of problem-solving,
critical thinking, and self-directed
learning. J Dent Educ, 70(9), 925-936.
Information, N. C. f. B. (2015). PubMed
Clinical Queiries. Bethesda, MD: U.S.
National Library of Medicine.
Johnsen, D. C., Finkelstein, M. W., Marshall,
T. A., & Chalkley, Y. M. (2009).
A model for critical thinking
measurement of dental student
performance. J Dent Educ, 73(2), 177-
183.
Lai, N. M. (2009). Dissecting students’ bedside
clinical questions using the ‘PICO’
framework. Med Educ, 43(5), 479-480.
doi:10.1111/j.1365-2923.2009.03320.x
Paul, R., & Elder, L. (2001). The miniature
guide to critical thinking: Concepts
& tools (Vol. 2): Foundation Critical
Thinking.
Rousseau, D. M., & Gunia, B. C. (2016).
Evidence-Based Practice: The
Psychology of EBP Implementation.
Annu Rev Psychol, 67, 667-
692. doi:10.1146/annurev-
psych-122414-033336
Sackett, D. L., Rosenberg, W. M., Gray, J.
A., Haynes, R. B., & Richardson, W.
S. (2007). Evidence based medicine:
what it is and what it isn’t. 1996. Clin
Orthop Relat Res, 455, 3-5.
Sackett, D. L., Straus, S. E., Richardson, W.
S., Rosenberg, W., & Haynes, R. B.
(2000). Evidence-based Medicine: How
to Teach and Practice EBM (2nd ed.).
New York: Churchill Livingston.
van Zyl, M. A., Bays, C. L., & Gilchrist, C.
(2013). Assessing Teaching Critical
Thinking with Validated Critical
Thinking Inventories: The Learning
Critical Thinking Inventory (LCTI)
and the Teaching Critical Thinking
Inventory (TCTI). Inquiry: Critical
Thinking Across the Disciplines, 28(3),
40-50.
58 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Author Information
Pauletta G. Baughman is an Associate
Professor, Department of General
Dentistry and Oral Medicine, University of
Louisville School of Dentistry. Please direct
correspondence and requests for reprints to her
at University of Louisville School of Dentistry,
Department of General Dentistry and Oral
Medicine, 501 S. Preston St., Room 42,
Louisville, KY 40202.
Phone 502-852-1186; Fax 502-852-1220;
Email: pgbaug01@louisville.edu
Gustavo M.S. Oliveira is an Assistant
Professor, Department of Operative Dentistry,
University of North Carolina, Chapel Hill,
USA. oliveirg@email.unc.edu.
Elizabeth M. Smigielski is a Professor,
Kornhauser Health Science Library, University
of Louisville, Kentucky, USA. emsmig01@
louisville.edu
Vida M. Vaughn is an Assistant Professor,
Kornhauser Health Science Library, University
of Louisville, Kentucky, USA. vmvaug01@
louisville.edu
FALL 2015, VOL. 30, NO. 3 59
Critical Thinking and the Pedagogy of Music Theory
Michael Lively
Southern Methodist University
Abstract
Students of music theory are often challenged by both the complexity of the concepts that they
are expected to learn and by the abstract nature of these ideas. For many students, their direct
experience with music, acquired during the study of skills associated with musical performance,
does not directly translate into the intellectual environment of traditional music theory classes.
The difculty derives from the gap between the students’ perception of musical structure and
the understanding of these concepts generally held by composers and music theorists. In this
study, I suggest that in addition to systematically teaching the content of the established music
theory curriculum, instructors have more success when developing instructional material and
determining the design of their courses by considering higher-level critical thinking skills
Keywords: Critical Thinking, Music Theory Pedagogy, Bloom’s Taxonomy, D. A. Kolb,
Experiential Learning
I. Introduction
Students of music theory are often
challenged by both the complexity of the
concepts that they are expected to learn and
by the abstract nature of these ideas. For many
students, their direct experience with music,
acquired during the study of skills associated
with musical performance, does not directly
translate into the intellectual environment of
traditional music theory classes. The difculty
derives from the gap between the students’
perception of musical structure and the
understanding of these concepts generally held
by composers and music theorists.
In this study, I suggest that in addition
to systematically teaching the content of
the established music theory curriculum,
instructors have more success when developing
instructional material and determining the
design of their courses by considering higher-
level “critical thinking” skills. Through
inclusion of these skills instructors may open
a discussion concerning the reasoning and
evaluative thinking that led to the development
of many of the essential concepts of the
traditional music theory curriculum. Through
a systematic consideration of critical thinking
and a reevaluation of the teaching methods that
music theory instructors choose to prioritize,
it may be possible to provide students with a
more complete understanding of the nature
of important theoretical principles and with a
more thoughtful recognition of the aspects of
the musical structures that form the basis of the
undergraduate music theory curriculum.
II. Elements of Critical Thinking
The term critical thinking was
developed during the mid and late twentieth
century, primarily by educational theorists,
to describe an approach to academic learning
that emphasized higher-level concepts and an
awareness of the process of learning itself,
rather than a mechanical or literal-minded
memorization of facts and procedures. It was
suggested that this higher-level or critical
method of learning would lead students
60 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
to a more complete understanding of their
academic subjects and a deeper recognition
of the intellectual foundations behind the
various scholarly disciplines. Here is a possible
denition for the term critical thinking which
is proposed by the Foundation for Critical
Thinking and which may be found on the
society’s website www.criticalthinking.org:
Critical thinking is that mode of
thinking — about any subject, content,
or problem — in which the thinker
improves the quality of his or her
thinking by skillfully analyzing,
assessing, and reconstructing it.
Critical thinking is self-directed, self-
disciplined, self-monitored, and self-
corrective thinking. It presupposes
assent to rigorous standards of
excellence and mindful command
of their use. It entails effective
communication and problem-solving
abilities, as well as a commitment to
overcome our native egocentrism and
sociocentrism.
Perhaps the most important advocate for the
idea of critical thinking during the early part
of the twentieth century was the educational
reformer John Dewey, who usually referred
to the concept by the term reective thinking.
Dewey believed that it was important for
students to learn how to assess and use
information in creative and sophisticated
ways, rather than to simply memorize facts
and uninterpreted information (Hytten,
2004). He felt that mere knowledge was often
conated with wisdom to the detriment of
the educational process. Dewey criticized
the idea of an academic system based upon
the repetition of memorized texts, claiming
that “no one thing, probably works so fatally
against focusing the attention of teachers upon
training of mind as the domination of their
minds by the idea that the chief thing is to get
pupils to recite their lessons correctly” (Dewey
1998 [1933], 65).
The critical thinking movement in
American education has emphasized the
importance of higher-level thinking skills to
assist students in a number of ways to move
beyond the mastery of specic content material
in any particular class. Critical thinking skills
are also understood to benet individuals in
a wide range of life situations. Here is how
Stephen Brookeld puts it:
I argue that if you can’t think critically
your survival is in peril because you
risk living a life that—without your
being aware of it—hurts you and serves
the interests of those who wish you
harm. If you can’t think critically you
have no chance of recognizing, let
alone pushing back on, those times you
are being manipulated. And if you can’t
think critically you will behave in ways
that have less chance of achieving the
results you want. So critical thinking
is not just an academic process that
leads to good SATs, elegantly argued
essays, or experimental hypotheses that
can stand the toughest scrutiny. It is a
way of living that helps you stay intact
when any number of organizations
(corporate, political, educational,
and cultural) are trying to get you to
think and act in ways that serve their
purposes (Brookeld 2012, 1-2).
Furthermore, the ability to rationally
and objectively evaluate the process of
thinking itself should lead to a society that is
more fair and more just. Superstition, irrational
beliefs, and social bias should be greatly
reduced or eliminated within a community
in which a signicant number of individuals
are able to rationally and objectively evaluate
their own thoughts and actions. Advocates
of teaching the skills of critical thinking
maintain that the ability to be intellectually
self-critical or self-evaluative helps to develop
higher-level skills in terms of ethical thinking
and even a desire to be fair-minded by
compensating for the effects of egocentrism
FALL 2015, VOL. 30, NO. 3 61
and cultural bias (Paul & Elder 2012, 347).
In this regard, the modern critical thinking
movement shares many ideas in common with
the ethical thinking of a number of important
philosophers, especially the work of Socrates,
Francis Bacon, and René Descartes (Paul &
Elder 2012, 412-16).
III. Music Theory
The pedagogical value of teaching
critical thinking skills has been recognized in a
number of academic disciplines. For example,
Conn (2008) has developed a methodology for
teaching aesthetics and the art of Rembrandt
in such a way that the critical thinking skills
developed by the students may be easily
applied to other subject areas. It is my belief
that the undergraduate music theory curriculum
provides many instances of the type of learning
objectives for which a one-dimensional,
teacher-centered model of instruction
may result in a very shallow, supercial
understanding of the most important higher-
level concepts discussed during the course.
Critical thinking skills can guide students
toward a more reasoned and rational approach
to the questions suggested by the curriculum,
rather than simply memorizing a set of facts
and learned procedures.
The process of teaching critical
thinking skills during a music theory course
should include careful consideration of the
general applicability of these skills to other
areas of study. In the study of the natural
sciences, critical thinking is often closely
associated with the scientic method and the
ability to formulate well-structured hypotheses;
while in the social sciences critical thinking
often describes the process of identifying
ideological or social bias. Brookeld (2012,
156) has suggested that the following
characteristics are common to critical thinking
in most areas of college or university study:
“(1) identifying the assumptions that frame
our thinking and determine our actions,
(2) checking out the degree to which these
assumptions are accurate and valid, (3) looking
at our ideas and actions from several different
perspectives, and (4) on the basis of all this,
taking informed actions.”
When one looks for them, one nds
that elements of critical thinking have
inuenced a number of research projects
within the elds of music and music theory.
Davis (1995) has investigated the philosophical
and epistemological assumptions behind the
study and analysis of atonal music; Johnson
(2011) has studied the effect of instruction in
critical thinking skills on the ability of students
to create verbal descriptions of music; and
Pogonowski (1989) has described the process
of using critical thinking skills to increase the
sophistication of students’ ability to listen to
music while understanding and identifying a
number of concepts that relate to a range of
academic disciplines.
IV. Pedagogical Specics
An Exemplary Pedagogical Problem
There are a number of specic learning
objectives within the traditional music
theory sequence that often seem to present
stumbling blocks for a signicant percentage
of undergraduate students. Perhaps the most
important of these problematic topics is the
concept of secondary dominant sonorities.
(See Lively (2005) for an exploration of the
importance of considering individual student
learning styles when teaching the concept of
secondary dominant functions.) Although there
are many concepts in the undergraduate music
theory sequence that represent abstract modes
of thinking or require complex reasoning skills,
the idea of secondary dominant function seems
to present unique and important challenges
for both music theory students and instructors.
The concept of secondary harmonic functions
normally requires at least two semesters
of music theory study to fully understand;
however, the following brief description may
be helpful for understanding the pedagogical
62 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
signicance of this teaching objective in terms
of its relationship to critical thinking skills.
In the rst semester of a traditional
undergraduate music theory sequence, the
students learn to classify harmonic sonorities
or “chords” by the scale degree upon which
they are derived. In a major key, there would
be seven diatonic scale degrees from which
the following chords could be derived: I, ii,
iii, IV, V, vi, and vii°. The chords derived from
the rst, fourth, and fth scale degrees are
described as major chords and are represented
by upper case Roman numerals. The chords
derived from the second, third, and sixth scale
degrees are described as minor chords and are
represented by lower case Roman numerals.
The chord derived from the seventh scale
degree is described as a diminished chord and
is represented by a lower case Roman numeral
with º as “the diminished symbol.”
Usually beginning in the second
semester of music theory study, the students
are expected to account for the harmonic
function of each chord that is analyzed. The
chord derived from the fth scale degree
in the system described above is known as
the dominant and under certain conditions
it demonstrates a dominant function. If a
chord derived from the sixth scale degree,
for example, were to demonstrate a dominant
function in relation to the second scale degree
(from which it is ve scale degrees above), it
should be correctly analyzed as a secondary
dominant with the symbol “V/ii” (“ve of
two,” or “the dominant of two”), rather than as
“vi.” An analogy is often made to the English
language, in which a noun, such as “cloud,”
can be used as a noun (The cloud is beautiful)
or as a verb (This may cloud my opinion).
The student in this analogy would be asked
to analyze the function of the word “cloud,”
rather than simply identifying the word
“cloud” as a word spelled C-L-O-U-D.
The idea of secondary function
represents the point at which a number of
students encounter, for the rst time in the
sequence, a learning object that requires a level
of abstraction that they may not be able to
easily demonstrate. This requires the student to
make an active decision concerning how he or
she will respond to the intellectual challenge.
The student may either choose to accept that
he or she does not understand the new concept
and accept the low test scores that will follow,
or the student may decide to learn the new
concept. In that case the student must actively
begin to think at a higher level of abstraction.
A Pre-existing Schema—the Need to Identify
and Challenge an Assumption
In my experience, it seems that the
primary reason this topic presents a problem
for certain students derives from a pre-
existing schema or cognitive model that
the students have formulated during earlier
lessons in musical analysis, a cognitive
model that conicts with the “new” idea of
secondary functions. The situation that arises
when the concept of secondary function is
initially introduced to undergraduate music
theory students represents an important and
interesting pedagogical “fork in the road.” The
new information seems to be a relatively minor
level of added complexity for the instructor
and more advanced students, yet at the same
time, for other students, it represents the end
of their ability to immediately understand the
information that is being presented using their
existing method of thinking. If students have
been introduced by this point in the music
theory curriculum to Brookeld’s conception
of critical thinking as involving identifying and
challenging the assumptions we make, it may
be possible not only to improve the students’
ability to learn the new specic learning
objectives, but, more importantly, to develop a
higher-level understanding of the purpose and
capabilities of traditional musical analysis.
Bloom’s Taxonomy and Higher Level Skills
A consideration of Bloom’s taxonomy
FALL 2015, VOL. 30, NO. 3 63
(Bloom 1956) may be helpful for the
application of higher-level critical thinking
skills to the specic learning objectives
related to the study of secondary dominant
functions. According to Bloom’s taxonomy,
learning objectives may be classied into six
distinct levels of complexity: (1) Knowledge,
(2) Comprehension, (3) Application, (4)
Analysis, (5) Synthesis, and (6) Evaluation.
The taxonomy provides a pedagogical
framework from which instructors may
develop teaching strategies and procedures
that specically require increasingly abstract
levels of cognition, and thereby allow for the
development of critical thinking skills. In this
connection, it may even be a good idea to
introduce Bloom’s taxonomy to students so
that they can better appreciate that they are
being asked in their learning to move from the
lower levels of the hierarchy of thinking skills
to the higher levels.
Before proceeding to any higher-level
understanding of the concept of secondary
functions, the instructor must rst present the
information in a clear and concrete manner.
The process of presenting the information
and any evaluative work related to the direct
recall of this information would be classied
as the Knowledge level of Bloom’s taxonomy.
Critical thinking cannot begin, at least to any
meaningful extent, until a certain minimum
level of concrete knowledge is transmitted and
successfully learned.
A learning objective related to the
Comprehension level of the taxonomy could
be to ask students to explain and describe
the differences between a sonority that
would be analyzed as “V/V” and a sonority
that would be analyzed as “ii” or “II.” The
Comprehension level may represent the most
important stage of the process, if an instructor
chooses to progress sequentially through the
levels of Bloom’s taxonomy, since it is at this
step that the students are rst asked to think
critically and to evaluate their own methods of
thinking. The Comprehension level requires
the student to ask questions about the reasons
that a statement may be true or not true, rather
than simply memorizing information and
repeating it in a manner that is not interpretive.
The Application level could be represented
by a short composition project, such as the
one presented in Example 1, especially if
the student compositions are subsequently
compared and analyzed through group
discussion.
Group Discussion and a Community of Inquiry
in the Classroom
The notion of a community of
inquiry within which group discussion takes
place has been a stressed in recent years as
a means of enhancing students’ reective
thinking (Gregory, 2007). The process of
group discussion is an important strategy for
15
15
64 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
helping students to begin to think critically
about any new learning objective. Stephen
Brookeld and Stephen Preskill (2005, 21-41)
have identied the following benets of class
discussion:
1. It helps students explore a diversity
of perspectives.
2. It increases students’ awareness of
and tolerance for ambiguity or complexity.
3. It helps students recognize and
investigate their assumptions.
4. It encourages attentive, respectful
listening.
5. It develops new appreciation for
continuing differences.
6. It increases intellectual agility.
7. It helps students become connected
to a topic.
8. It shows respect for students’ voices
and experiences.
9. It helps students learn the processes
and habits of democratic discourse.
10. It afrms students as cocreators of
knowledge.
11. It develops the capacity for the clear
communication of ideas and meaning.
12. It develops habits of collaborative
learning.
13. It increases breadth and makes
students more empathic.
14. It helps students develop skills of
synthesis and integration.
15. It leads to transformation.
In particular, asking questions during
group discussions that require the students
to actively evaluate their own methods of
thinking may be among the best methods for
helping students to arrive at a deeper level
of understanding, especially for learning
objectives that are challenging for some
students. Scott Dirkse (2014, 69-84) has
suggested the following considerations for
asking discussion questions while teaching a
music theory class:
1. Questions should be directed to all
students.
2. Each question should be followed by
“wait time.”
3. Questions should be precise and
answerable.
4. Questions should be inviting, not
intimidating.
5. Questions should require student
thinking.
6. Questions should elicit maximum
student participation.
The process of completing a “Roman
numeral” analytical assignment, especially
a complex example that presents secondary
dominant sonorities in a variety of inversions,
such as Example 2, could be classied as
representing the Analysis level of Bloom’s
taxonomy.
Music theorists may recognize that the
harmonic progression provided in Example
2 represents a version of the “omnibus”
progression, in which the secondary dominant
sonorities may not really be functioning as true
dominants. This allows for an in-depth class
discussion, in which the instructor may ask
the students to think critically by attempting to
redene the meaning of the dominant function.
Active Experimentation and Testing One’s
Theories
A learning cycle, in which experience
leads to observation and reection, followed
FALL 2015, VOL. 30, NO. 3 65
ultimately by concept formation has been
described by D. A. Kolb (1984). According
to the model proposed by Kolb, a person rst
has a concrete experience and then makes
reective observations about it; these reective
observations will eventually form the basis of
abstract conceptualizations as the individual
ts the observations into generalized theories.
A person will then ultimately test these theories
through active experimentation. The Synthesis
level of Bloom’s taxonomy is an important
stage of the pedagogical process, equivalent
to active experimentation, the nal stage of
Kolb’s learning cycle.
The internal cognitive models that the
student has developed, even after the process
of critical evaluation, must ultimately be
tested against real examples of the systems
or structures that the music theory student is
expected to understand. The improvements
or corrections that the students may choose to
make to their own internal cognitive models,
after engaging in active experimentation,
represents the Synthesis level of Bloom’s
taxonomy. A learning objective related to the
Synthesis level of the taxonomy could be to ask
students to complete a realization assignment,
such as Example 3.
The Essential Role of Purpose
Richard Paul and Linda Elder’s view
of critical thinking gives a central role to
analyzing the purpose of our thinking in
particular cases (Paul & Elder, 2012). This
comports with Bloom’s Evaluation level, and
it can be implemented by asking students
to explain and describe the differences and
similarities between secondary dominant
chords, secondary seventh chords, and other
types of dominant preparation sonorities,
17
17
16
16
66 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
including augmented sixth chords. At this
stage the instructor may ask the students to
truly evaluate the knowledge that they have
learned by asking questions such as “What is
the purpose of harmonic analysis?” or “What
meaning is gained by understanding the
harmonic analysis of a musical work?” If other
analytical systems have been discussed in the
class, the instructor may even ask the students
to critically compare and evaluate “functional
analysis” in relation to other analytical
systems, such as Renaissance modality,
Schenkerian analysis, or Neo-Riemannian
analysis. At this point, the instructor may also
introduce the idea of critical thinking itself and
discuss strategies for applying the concepts
of critical thinking to any complex learning
objective.
Making the Assumption Embedded in a Pre-
existing Schema Apparent
There can be a considerable difference
between the level of understanding and
sophistication demonstrated by a group of
students who have studied the concept of
secondary functions through the use of critical
thinking skills and higher-level modes of
cognition, compared to a group who have tried
to learn these concepts only through the one-
dimensional techniques of memorization and
concrete technical procedures. The pre-existing
schema or cognitive model that I believe
interferes with the ability of some students
to easily learn and understand the concept of
secondary dominant functions is the notion
that the primary objective of musical analysis
is to arrange a discrete group of pitches into a
tertian-related “stack” and to then simply name
the “stack” by its apparent root. This cognitive
model may have served the student well during
earlier lessons related to chord inversions and
analysis; however, it entirely fails to consider
important higher-level concepts such chord
progression, harmonic function, or musical
syntax. For this reason, I believe that it may be
highly effective to introduce critical thinking
skills into the instructional sequence when
rst discussing the idea of secondary dominant
functions.
I discovered an interesting example
of the kind of one-dimensional thinking
that may hinder some students from easily
understanding secondary dominant functions
when I gave an assignment similar to Example
4 as an extra-credit test question in a second or
third semester music theory class.
These tests were given at an institution in
which “musicianship” training was not a
signicant aspect of the music theory sequence
and no specic instruction in harmonizing a
melody had been given to the students prior
to this extra-credit test question. Teaching
students to be able to successfully harmonize
a melody is, of course, one of the most
productive exercises that can be offered in
terms of the development of higher-level
critical thinking skills. Nancy Rogers’s article
18
18
FALL 2015, VOL. 30, NO. 3 67
“Interpreting and Harmonizing Melodies:
Some Formulas for Success” (Rogers 2008)
describes specic techniques for teaching the
skill of harmonizing a melody, such as learning
to “prioritize the cadence,” or learning to
recognize non-cadential harmonic and melodic
formulas.
To my great surprise, at least sixty
percent of the students answered the question
by providing the incorrect analysis shown in
Example 4, a result that was repeated in several
subsequent semesters. The analysis given in
Example 4, of course, demonstrates a complete
disregard for the concept of “harmonic
rhythm,” but more importantly, the analysis
represents a failure to employ the principles
of voice-leading and part-writing that the
students were able to successfully demonstrate
in their other work on the same test. In terms
of Bloom’s taxonomy, the higher-level skills
related to Application, Analysis, and Synthesis
were not being used. As part of the process of
the post-test review, I made the point that this
harmonization results in a bass line that creates
one hundred percent parallel octaves with the
melody. This type of analysis could only be
produced if the students were mechanically
operating on the cognitive model that the
“correct analysis” results from a “tertian stack
of a discrete group of pitches.” If the “group
of pitches” were the melody pitch alone, then
the answer would be a chord with that pitch
as the root, i.e. “garbage in— garbage out.”
(If we were to extend the analogy to English
grammar once again, this incorrect analysis
might be comparable to analyzing every word
in a sentence as a noun and as the subject, in
other words a completely non-interpretive and
non-functional analysis.)
This is a very clear example of the
importance of teaching the meaning and
purpose of musical analysis, rather than simply
the technique of giving correct answers on a
music theory test. Emphasis on the meaning
and purpose of musical analysis is perhaps
even more central to the process of studying
advanced topics, such as set theory, Neo-
Riemannian analysis, or Schenkerian analysis.
V. Conclusion
Music theory is one of the academic
subject areas in which students are often
confounded by the complexity of the learning
objectives with which they are presented.
In addition to clear and careful teaching of
the primary content material, music theory
instructors should consider the systematic
inclusion of critical thinking skills during the
process of developing the content of their
courses. Teaching students to be mindful
and deliberate thinkers not only benets
their ability to master a complex subject
such as music theory, but more importantly,
improves their ability to learn in a wide range
of subjects, and perhaps their capacity to
become thoughtful and productive citizens and
members of the world community.
References
Bloom, B. S. & Associates. 1956. Taxonomy
of Educational Objectives: The
Classication of Educational Goals.
New York: David McKay Company.
Brookeld, Stephen. 2012. Teaching for
Critical Thinking. San Francisco,
California: Jossey-Bass.
Brookeld, Stephen & Preskill, Stephen. 2005.
Discussion as a Way of Teaching, 2nd
ed. San Francisco, California: Jossey-
Bass.
Conn, Mark S. 2008. “Rembrandt’s Art: A
Paradigm for Critical Thinking and
Aesthetics.” The Journal of Aesthetic
Education 42: 68-82.
Davis, James A. 1995. “Philosophical
Positivism and American Atonal Music
Theory.” Journal of the History of
Ideas 56: 501-22.
Dewey, John. 1998 [1933]. How We Think.
68 INQUIRY: CRITICAL THINKING ACROSS THE DISCIPLINES
Author Information
Michael Lively is currently a Visiting Lecturer
in Music Theory at Southern Methodist
University. He recently served a one-year
appointment as Visiting Assistant Professor of
Music Theory at Texas Woman’s University. In
2010 he completed a Ph.D. in Music Theory at
the University of North Texas. His publications
include articles in the Journal of Music Theory
Pedagogy at the College Music Symposium.
He is also music director and organist at the
Anglican Church of Saint Peter and Saint Paul
in Arlington, Texas.
New York: Houghton Mifin.
Dirkse, Scott. 2014. “Effective Questioning
Strategies for the Music Theory
Classroom.” Journal of Music Theory
Pedagogy 27: 69-84.
Hytten, Kathy. 2004. “Democracy and
John Dewey.” In Critical Thinking
and Learning, ed. Joe L. Kincheloe
and Danny Weil, 149-53. Westport,
Connecticut: Greenwood Press.
Gregory, M. (2007). A framework for
facilitating classroom dialogue.
Teaching philosophy, 30(1), 59-84.
Johnson, Daniel C. “The Effect of Critical
Thinking Instruction on Verbal
Descriptions of Music.” Journal of
Research in Music Education 59: 257-
72.
Kolb, David A. 1984. Experiential Learning.
London: Prentice Hall.
Lively, Michael. 2005. “D. A. Kolb’s Theory
of Experiential Learning: Implications
for the Development of music Theory
Instructional Material.” Journal of
Music Theory Pedagogy 19: 77-100.
Paul, Richard & Elder, Linda. 2012.
Critical Thinking, 3rd ed. Boston,
Massachusetts: Pearson.
Pogonowski, Lenore. 1989. “Critical Thinking
and Music Listening.” Music Educators
Journal 76: 35-38.
Rogers, Nancy. 2008. “Interpreting and
Harmonizing Melodies: Some
Formulas for Success.” In AP Music
Theory: Building AP Music Theory
from the Ground Up, 32-38. The
College Board.
ELECTRONIC SUBSCRIPTION PROCESS
To subscribe electronically, go to the Philosophy Documentation Center website at www.pdcnet.org. On the website
go to the “Products” heading and click on “Journals and Series.” That will take you to a list of journals, and then click
on “
INQUIRY
” for the electronic subscription page.
INQUIRY
is published three times per year.
To subscribe by mail use this form.
MAIL SUBSCRIPTION FORM
This form can be duplicated and can be used to subscribe to
INQUIRY: Critical Thinking Across the Disciplines
and can be used by individuals and institutions.
Please enter a subscription to
INQUIRY
for:
Name:____________________________________________________________________________
Institution:________________________________________________________________________
Address:__________________________________________________________________________
City/State/Zip:_____________________________________________________________________
No. of Years COST
Individual Subscription
Print only ($40 per year) ______ ______
Online only ($50 per year — includes electronic access to
volumes 1 to the present) ______ ______
Print + Online ($64 per year — includes electronic access to
volumes 1 to the present) ______ ______
Institutional Subscription
Online only ($160 per year — includes electronic access to
volumes 1 to the present) ______ ______
Print + Online ($200 per year — includes electronic access to
volumes 1 to the present) ______ ______
Foreign Surcharge ($8.00 per year, applies to both individual and
institutional subscriptions to cover increased postage ______ ______
Single Issue ($18 each) Number of issues ______ ______
TOTAL COST: ______
MAKE CHECKS PAYABLE TO:
INQUIRY
SEND FORM AND CHECK TO: Philosophy Documentation Center
P.O. Box 7147
Charlottesville, VA 22906-7147
(800) 444-2419 (US & Canada) or
(434) 220-3300
FAX: (434) 220-3301
Web: www.pdcnet.org
Sam Houston State University
Sponsors
Dr. Richard Eglsaer, Interim Provost and Vice President for Academic Affairs
Dr. Mitchell Muehsam, Dean, College of Business
Dr. Phillip Lyons, Dean, College of Criminal Justice
Dr. Stacey Edmonson, Dean, College of Education
Dr. Ronald Shields, Dean, College of Fine Arts and Mass Communication
Dr. Abbey Zink, Dean, College of Humanities and Social Sciences
Dr. John Pascarella, Dean, College of Sciences
Dr. Rhonda Callaway, Interim Dean, College of Health Sciences
BOARD OF REGENTS
THE TEXAS STATE UNIVERSITY SYSTEM
Dr. Jaime R. Garza, Chairman ....................................San Antonio
Rossanna Salazar, Vice Chairman ...................................... Austin
Charlie Amato ................................................San Antonio
Veronica Muzquiz-Edwards ......................................San Antonio
David Montagne ................................................Beaumont
Vernon Reaser III ..................................................Bellaire
William F. Scott ................................................ Nederland
Alan L. Tinsley .............................................. Madisonville
Donna N. Williams ..............................................Arlington
Dylan McFarland, Student Regent .................................. Huntsville
Brian McCall, Chancellor ............................................ Austin
Instructions to Authors
INQUIRY: Critical Thinking Across the Disciplines is sponsored by the College of Educa-
tion, the College of Sciences, the College of Fine Arts and Mass Communication, the College
of Criminal Justice, the College of Humanities and Social Sciences, and the College of Business
Administration of Sam Houston State University. It is a forum for the discussion of issues related
to the theory, practice, and pedagogy of critical thinking across the disciplines, from precollege to
university settings.
INQUIRY: Critical Thinking Across the Disciplines invites submission of articles, responses,
reviews of the literature, notices, letters, and book reviews for consideration for publication. The
goal is to encourage an exchange of ideas about effective pedagogy in critical thinking instruction,
about methods of assessing critical thinking skills and dispositions, about systematic errors in our
thinking, about enhancing the quality of information on which we base decisions and inferences,
about common fallacies in argumentation, and about all other topics that are relevant to critical
thinking across the disciplines.
Given the diverse nature of the eld and of our readership, we ask that authors present early
in the manuscript the manner in which “critical thinking” is to be understood. Each submission
should have on a separate page the author’s name and complete contact information, including dis-
ciplinary specialty and institutional afliation. Each manuscript should be submitted in Microsoft
Word or as a pdf le as an email attachment. Send submissions to the Managing Editor, Frank
Fair, of the Department of Psychology and Philosophy, Box 2447, Sam Houston State University,
Huntsville, TX 77341-2447. The email address is criticalthinking@shsu.edu
Authors must follow the 6th edition of the Publication Manual of the American Psychologi-
cal Association. Manuscripts that fail to conform to APA style will not be considered for review.
By submitting a manuscript to INQUIRY an author is afrming that the manuscript has not been
published and is not under consideration for publication elsewhere. Authors are responsible for
obtaining and providing written permission from copyright owners for reprinting previously pub-
lished illustrations, tables, or lengthy quotes of 500 words or more. Authors are responsible for the
accuracy of the material in their manuscripts. Use of footnotes and endnotes should be limited as
much as possible.
All manuscripts must be accompanied by an abstract of 100-150 words on a separate page or
the manuscript will not be accepted for review. Include no more than three key words below the
abstract. All manuscript pages, including reference lists and tables, must be double-spaced. All
gures must be camera-ready.
INQUIRY: Critical Thinking Across the Disciplines is published three times a year in coop-
eration with the Philosophy Documentation Center. All correspondence regarding subscriptions,
renewals, change of address, back issues, sample copies, and related matters regarding the jour-
nal’s distribution should be addressed to: Philosophy Documentation Center, P.O. Box 7147,
Charlottesville, VA 22906-7147. Phone (800) 444-2419 (USA and Canada) or (804) 220-3300.
Fax: (804) 220-3301. URL: www.pdcnet.org.
For information about subscriptions, see the next to the last page of this issue.
Kevin Possin (Philosophy)
Winona State University
Maria Sanders (Philosophy and Law)
Plymouth State University
Michael Scriven (Psychology and Evaluation)
The Claremont Graduate School and
Western Michigan University
Harvey Siegel (Philosophy)
University of Miami
Keith Stanovich (Psychology)
University of Toronto
James Trel (Physics)
George Mason University
Mark Weinstein (Educational Foundations)
Montclair State University
Frank Zenker (Philosophy and Cognitive Science)
Lund University
INQUIRY: Critical Thinking Across the Disciplines ©
Sponsored by the
College of Business Administration, the College of Criminal Justice, the College of Humanities and
Social Sciences, the College of Education, the College of Fine Arts and Mass Communication, College of
Health Sciences, and the College of Sciences at Sam Houston State University
The Editorial Staff
Frank Fair, Managing Editor (Philosophy, SHSU)
Valerie Muehsam, Associate Editor (Business Analysis, SHSU)
Marcus Gillespie, Associate Editor (Geography, SHSU)
Jeff Anastasi, Associate Editor (Psychology, SHSU)
Mike Vaughn, Associate Editor (Criminal Justice, SHSU)
Daphne Johnson, Associate Editor (Curriculum and Instruction, SHSU)
Michael Henderson, Associate Editor (Art, SHSU)
Sheryl Murphy-Manley, Assosciate Editor (School of Music, SHSU)
Scott Miller, Associate Editor (Physics, SHSU)
Ben Hamby, Associate Editor (Philosophy, Coastal Carolina)
David Wright, Associate Editor (Philosophy, SHSU)
Kevin R. Anderson, Production Editor (Sam Houston Press & Copy Center, SHSU)
Janet Fair, Copy Editor (SAM Center, SHSU)
EDITORIAL ADVISORY BOARD
J. Anthony Blair (Philosophy)
University of Windsor
Linda Elder (Educational Psychology)
Foundation for Critical Thinking
Robert Ennis (Education and Philosophy)
University of Illinois
Daniel Fasko, Jr. (Educational Psychology)
Bowling Green State University
Alec Fisher (Philosophy)
University of East Anglia
Donald Hatcher (Philosophy)
Baker University
Ralph Johnson (Philosophy)
University of Windsor
Richard Mayer (Psychology)
UC Santa Barbara
Gerald Nosich (Philosophy)
Buffalo State College
Founding Editors and Layout and Cover Designers
Wendy Oxman, Robert Michael Esforme, Jeff Spradling, and Erin Back
Statements or opinions expressed in INQUIRY: Critical Thinking Across the Disciplines reect the views of the
author(s) and do not represent the policy of the College of Business Administration, the College of Criminal
Justice, the College of Humanities and Social Sciences, the College of Education, the College of Fine Arts and
Mass Communication, College of Health Sciences, or the College of Sciences at Sam Houston State University
unless so stated. INQUIRY: Critical Thinking Across the Disciplines has been indexed by the following services:
ERIC, Expanded Academic ASAP, Index Philosophicus, InfoTrac OneFile, International Bibliography of
Book Reviews of Scholarly Literature (IBR), International Bibliography of Periodical Literature (IBZ), MLA
International Bibliography, The Philosopher’s Index, PsycINFO.
©2012, INQUIRY: Critical Thinking Across the Disciplines. All rights reserved.
ISSN: 1093-1082 eISSN: 2153-9871 www.pdcnet.org/inquiryct
Printed in the United States of America
INQUIRY
C
RITICAL
T
HINKING
A
CROSS
THE
D
ISCIPLINES
Summer 2010 Vol. XXV No. 2
The College of Arts and Sciences, the College of Business
Administration, the College of Humanities and Social Sciences,
the College of Education, and the College of Criminal Justice
Sam Houston State University
ISSN 1093-1082
977109 3 108003
2 2
gniknihT lacitirC gnihcaeT ni scigoL gnitcilfnoCzapraH maroY
?edutittA dna edutitpA fo noitseuQ A :gniknihT lacitirCdna oteiN .M anA
Carlos Saiz
Harikumar Sankaran and Implications for Critical Thinking Disposition:
ocixeM weN ni nemhserF morf ecnedivEcivejirtimiD aziraM
STCAT,riaF knarF TM : Developing a New Critical Thinking
tnemurtsnI tnemssessA,masheuM eirelaV
John Miller, and
Wendy McCoy Elliott
The College of Business Administration, the College of Criminal
Justice, the College of Humanities and Social Sciences,
the College of Education, the College of Fine Arts and Mass
Communication, College of Health Sciences, and
the College of Sciences
Sam Houston State University
FALL 2015 Vol. XXX No. 3
Sam Houston
State University
INQUIRY
CRITICAL
THINKING
ACROSS
THE
DISCIPLINES
Summer 2010 Vol. XXV No. 2
INQUIRY
C
RITICAL
T
HINKING
A
CROSS
THE
D
ISCIPLINES
Summer 2010 Vol. XXV No. 2
The College of Arts and Sciences, the College of Business
Administration, the College of Humanities and Social Sciences,
the College of Education, and the College of Criminal Justice
Sam Houston State University
ISSN 1093-1082
977109 3 108003
2 2
gniknihT lacitirC gnihcaeT ni scigoL gnitcilfnoCzapraH maroY
?edutittA dna edutitpA fo noitseuQ A :gniknihT lacitirCdna oteiN .M anA
Carlos Saiz
Harikumar Sankaran and Implications for Critical Thinking Disposition:
ocixeM weN ni nemhserF morf ecnedivEcivejirtimiD aziraM
STCAT,riaF knarF TM : Developing a New Critical Thinking
tnemurtsnI tnemssessA,masheuM eirelaV
John Miller, and
Wendy McCoy Elliott
INQUIRY
C
RITICAL
T
HINKING
A
CROSS
THE
D
ISCIPLINES
Summer 2010 Vol. XXV No. 2
The College of Arts and Sciences, the College of Business
Administration, the College of Humanities and Social Sciences,
the College of Education, and the College of Criminal Justice
Sam Houston State University
ISSN 1093-1082
9
77109 3 108003
2 2
gniknihT lacitirC gnihcaeT ni scigoL gnitcilfnoCzapraH maroY
?edutittA dna edutitpA fo noitseuQ A :gniknihT lacitirCdna oteiN .M anA
Carlos Saiz
Harikumar Sankaran and Implications for Critical Thinking Disposition:
ocixeM weN ni nemhserF morf ecnedivEcivejirtimiD aziraM
STCAT,riaF knarF TM : Developing a New Critical Thinking
tnemurtsnI tnemssessA,masheuM eirelaV
John Miller, and
Wendy McCoy Elliott
FALL 2015 Vol. XXX No. 3
From the Editor’s Desk ............................................................................................................. 3
Frank Fair
Critical Thinking Instruction: A Realistic Evaluation ........................................................... 4
Donald Hatcher
Are We Asking the Right Questions About Critical Thinking Assessment:
A Response to Hatcher ............................................................................................................. 20
David Wright
Effect Size and Critical Thinking Assessment: A Response to Wright .............................. 32
Donald Hatcher
Using the Critical Thinking Assessment Test (CAT) as a Model for Designing
Within-Course Assessments: Changing How Faculty Assess Student Learning ............... 38
Ada Haynes, Elizabeth Lisic,
Kevin Harris, Katie Lemming,
Kyle Shanks, and Barry Stein
Evidence-Based Critical Thinking Exercise:
Shaping Tomorrow’s Dental Choices ..................................................................................... 49
Pauletta G. Baughman,
Gustavo M. S. Oliveira,
Elizabeth M. Smigelski, and
Vida Vaughn
Critical thinking and the Pedagogy of Music Theory ........................................................... 59
Michael Lively
