Content uploaded by Andreas Duenser
Author content
All content in this area was uploaded by Andreas Duenser
Content may be subject to copyright.
Perceptual and Motor Skills, 2007, 104, 123-140. © Perceptual and Motor Skills 2007
WARNING: SUBTLE ASPECTS OF STRATEGY ASSESSMENT MAY
AFFECT CORRELATIONS AMONG SPATIAL TESTS1
JUDITH GLÜCK AND ANDREAS DÜNSER
Faculty of Psychology
University of Vienna
KARIN STEINBÜGL AND HANNES KAUFMANN
Institute of Software Engineering and Interactive Systems
Technical University of Vienna
Summary.—In this study, preliminary to a larger experiment, 42 participants
completed four different spatial tests and, after each test, a strategy questionnaire. For
half of the participants, visualizational strategies were presented first in this question-
naire, and for the other half, analytical strategies. The order of strategy descriptions
had effects on the strategies reported and on the intercorrelations among the spatial
tests and between the spatial tests and an inductive-reasoning test. In the group first
presented with visualizational strategies, intercorrelations among the spatial tests were
higher and correlations with the reasoning test were lower than in the group first pre-
sented with analytical strategies. Bootstrap analyses with 100 random splits of the sam-
ple confirmed this result. The findings are interpreted as indications of a priming ef-
fect by the strategy descriptions which affected the way participants dealt with subse-
quent tests. Implications for strategy assessment are discussed.
The fact that different people may solve the same spatial tasks in differ-
ent ways was already recognized in early phases of the development of psy-
chometric tests and structural theories of intelligence (for an overview, see
Lohman & Kyllonen, 1983). For example, Thurstone (1938) argued that
when a test loads on more than one ability factor, it is unclear whether every
participant used all those abilities in solving the test items or whether differ-
ent individuals used different abilities to produce the same performance (see
Kyllonen, Lohman, & Snow, 1984, p. 131). Thus, from a psychometric view-
point, diversity of strategies is problematic because it affects the interpret-
ability of test scores. The instructions of many spatial tests have therefore
been designed to emphasize "truly spatial" strategies, for example, by ex-
plicitly suggesting mental rotation as the best way to solve the test items
(Schultz, 1991).
Across different domains of spatial ability, however, both interindivid-
1 Address correspondence to A. Prof. Dr. Judith Glück, Faculty of Psychology, University of
Vienna, Liebiggasse 5, 1010 Vienna, Austria or e-mail (judith.glueck@univie.ac.at). This re-
search was funded by Grant no. P16803 from the Austrian Research Foundation (FWF).
Andreas Dünser is now at HITlab NZ, University of Canterbury, Christchurch, New Zealand.
DOT 10.2466/PMS.104.1.123-140
124 J. GLÜCK, ET AL.
ual and intra-individual differences in strategy use have been reported (for
an overview, see Glück & Fitting, 2003). On the most general level, two
types of strategies can be distinguished. A person using a holistic strategy
represents and manipulates spatial information "in a spatial way," that is, he
maintains and uses information about the spatial relations between the ele-
ments in the mental representation. Examples of holistic strategies include
mentally rotating stimuli in a cube comparison test or visualizing a map while
navigating through an environment. Analytic strategies represent and manip-
ulate spatial information by reducing it to an essentially nonspatial, list-like
format. The spatial relations among the patterns on a cube can be represent-
ed as a list of relations among pairs of patterns (which Just & Carpenter,
1985, called an "orientation-free" description), and a route can be repre-
sented as a list of landmarks without any notion of the spatial layout of the
environment. Thus, analytic strategies reduce the complexity of spatial infor-
mation, focusing on parts of the object or the environment one by one rath-
er than on the object, and its spatial configuration, as a whole.
Analytic strategies have sometimes been labeled "verbal" strategies, but
they do not necessarily require verbalization. Similarly, it would be too nar-
row to label all holistic strategies "visualization," although most of them in-
volve visualization. However, holistic strategies also include other ways of
representing spatial relations, such as maintaining a sense of the direction of
a reference point while navigating an environment. Finally, analytic and ho-
listic strategies should not be viewed as mutually exclusive categories; rather,
they are the poles of a continuum, with the possibilities of intermediate and
combined strategies (see, e.g., Just & Carpenter, 1985).
Strategy Use, Task Difficulty, and Test Performance
Is there a relationship between the way someone solves a spatial task
and actual performance on the task? First of all, strategy use depends on
task difficulty. The more difficult an item or a test, the more analytic the
strategies used (Barratt, 1953; Lohman & Kyllonen, 1983; Kyllonen, et al,
1984). This pattern has also been found in a number of cognitive-exper-
imental studies on mental rotation (e.g., Yuille & Steiger, 1982; Just & Car-
penter, 1985).
In spite of this relationship between item difficulty and strategy use,
most researchers have treated strategy use as a stable personal variable, ask-
ing participants for their strategies only after they have completed a test.
Based on the relationship between task difficulty and strategy use described
above, one would expect individual differences in the difficulty level at which
people switch to analytic strategies. Individuals high in spatial ability may be
able to solve tasks in a holistic way for which low-spatial individuals need to
resort to analytic strategies. This idea is supported by a number of studies.
STRATEGY ASSESSMENT AFFECTS TEST INTERCORRELATIONS 125
For example, Cochran and Wheatley (1989) found that use of holistic strate-
gies was correlated with performance on relatively easy items. Schultz (1991)
found that using mental rotation in the Vandenberg-Kuse Mental Rotation
Test (Vandenberg & Kuse, 1978) was positively correlated with test perfor-
mance, whereas using analytic strategies was negatively correlated with test
performance.
Perhaps one reason few researchers study strategy use empirically is be-
cause it is not immediately obvious that the strategies an individual uses are
relevant for interpreting the person's performance. Thurstone's (1938) argu-
ment shows that in the early days of test development, researchers were wor-
ried about the interpretation of test scores when there is strategy heteroge-
neity. In a classical study, French (1965) showed that for participants repor-
ting analytic strategies, performance in two spatial tests had much lower load-
ings on a visualization factor than for other participants.
If an individual uses analytic strategies in a spatial test, does the test
score still reflect that person's "spatial" ability? And, if it is not spatial abil-
ity that determines this individual's performance, what is it? Reasoning
seems to be an interesting candidate. On a basic level, most types of spatial
test items require participants to decide whether two stimuli are identical
with respect to structural characteristics. This identity question can be solved
by "spatial" transformations, such as mental rotation, or by generating and
comparing "nonspatial," orientation-free descriptions. Thus, the cognitive
demand in many analytic strategies seems to be to extract rules that describe
the structure of one stimulus, and to test whether the same rules apply to
the other. Skill at extracting and applying rules is measured by tests of in-
ductive reasoning, such as matrices or letter series. Therefore, we assume that
the spatial test performance of an individual using analytic strategies should
be more highly correlated with inductive reasoning than that of a person
using holistic strategies.
Assessment of Spatial Strategies
Another reason for the lack of studies of strategy use in spatial tasks
may be that no single optimal methodology has yet been developed for the
valid and reliable assessment of strategy use. Researchers have used (a) re-
sponse times, especially in the context of mental rotation research (e.g., Mu-
maw, Pellegrino, Kail, & Carter, 1984; Just & Carpenter, 1985; Cohen &
Kubovy, 1993; Pierret & Peronnet, 1994; Grimshaw, Sitarenios, & Finegan,
1995), (b) eye movement analyses (e.g., Johnston & Pirozzolo, 1981; Deff-
ner, 1985; Just & Carpenter, 1985), (c) response or error patterns (e.g., Köl-
ler, Rost, & Köller, 1994; Hosenfeld, Strauss, & Köller, 1997; Glück, Ma-
chat, Jirasko, & Rollett, 2002), or (d) dual-task conditions (e.g., Bowers &
LaBarba, 1988). Each method has some advantages and some problems. As
126 J. GLÜCK, ET AL.
Lohman and Kyllonen (1983) argued, researchers are well advised to use
convergent evidence from several strategy-assessment methods (see also Erics-
son & Simon, 1980, 1993). For example, Just and Carpenter (1985) used
such convergent evidence from self-reports, response latencies, and eye-move-
ment data in their analysis of solution strategies on cube-comparison tasks.
Self-report data, which we used in this study, have been collected by
many researchers, and in many different ways (e.g., Barratt, 1953; French,
1965; Cochran & Wheatley, 1989; Schultz, 1991; Glück, 1999). The validity
of such self-reports is dependent on a number of factors. Based on Ericsson
and Simon's (1980, 1993) classical discussion of the validity of self-report
data, a number of aspects can be identified that are particularly relevant for
assessing strategy use in spatial tasks.
First, it is not quite clear to what degree individuals are consciously
aware of the strategies they use in spatial tasks. While people seem to have
some intuitive notion of whether they have mentally rotated a stimulus, we
do not know how valid these intuitions are. Problems of strategy conscious-
ness are particularly relevant when tasks are relatively familiar and some of
the cognitive processes involved may be automatized. Second, spatial tasks
are, by nature, visual (although there are a few exceptions), and the transla-
tion of visual information into language may be difficult. A third aspect re-
fers to the timing of the strategy assessment. Some researchers have collected
strategy data during the solution process, e.g., by thinking-aloud methods.
Obviously, these methods require large effort in content coding. Other re-
searchers have used multiple-choice strategy questionnaires presented after
every single item (Glück, 1999; Glück & Fitting, 2003). The problem with
such approaches is that participants' motivation to observe their own solu-
tion processes in detail decreases across a test, which may lead them to
check off the same strategies for all items. In addition, all closed question-
naire formats have the problem that the strategies described may not match
the way some participants solved the tasks. Taking all this together, the most
problematic method is to assess strategy by (multiple-choice) questionnaire
after the whole test has been completed. Here, participants' ideas of how the
items should be solved may have a strong influence on how they think they
solved them (Ericsson & Simon, 1980).
In the present study, we tried out a questionnaire method for strategy
assessment that should overcome some of the problems described above,
while at the same time being brief and requiring no coding. However, as we
show below, we discovered yet another problematic aspect of strategy assess-
ment by questionnaire: simple aspects such as which strategy is described
first in the questionnaire can affect how participants solve subsequent tasks.
As we interpret this as a kind of priming effect, we now briefly review the
literature on effects of priming on test performance.
STRATEGY ASSESSMENT AFFECTS TEST INTERCORRELATIONS 127
Priming Effects on (Spatial) Test Performance
In the last ten years, a large number of studies on effects of conscious
or subconscious manipulations on subsequent cognitive performance have
been published. For example, Dijksterhuis and van Knippenberg (1999)
showed that people's performance in a vocabulary test can increase or de-
crease depending on whether they have been thinking about "professors" or
"football hooligans" prior to taking the test. Several studies have shown that
activation of group stereotypes with respect to cognitive performance can
have effects on members of the respective group. Similarly, Steele and Aron-
son (1995) found that African Americans showed lower cognitive perfor-
mance if their group membership was primed, and Levy (1996) found that
older individuals performed lower if negative aging stereotypes (e.g., forget-
fulness) were activated, and higher if a positive aging stereotype (wisdom)
was activated. No studies on priming of spatial performance have been pub-
lished yet. In a study on mathematical performance, Shih and Pittinsky (1999)
showed that the performance of Asian American women increased if their
Asian group membership was emphasized prior to the test, and decreased if
their gender was emphasized. Similar effects might be expected for spatial
ability.
While the results of priming have been clearly demonstrated, it is rela-
tively unclear exactly how priming affects cognitive processes (see, e.g., Hig-
gins, Bargh, & Lombardi, 1985). Presumably, basic cognitive processes are
less susceptible to priming than strategy use, that is, the way people solve a
task. Therefore, especially in domains where different strategies lead to the
same end, such as in spatial ability, priming manipulations might well affect
participants' strategies. The present study was aimed at evaluating instru-
ments designed for a larger study. There was no attempt to prime partici-
pants' strategies. However, the order in which strategies were described in
the questionnaire was varied just to check whether there would be any dif-
ferences. This manipulation seems to have had clear effects on the way par-
ticipants solved the items of the subsequent test. This finding is important as
a "warning" to all researchers designing strategy-assessment studies.
In the following, we present the results of a preliminary study in the
context of a larger training evaluation. The aims of the current study were
(a) to get some first data on how well several tests translated into German
were applicable for Austrian secondary-school students, and (b) to try out a
new questionnaire format for assessing strategy use in all these spatial tests.
Participants were students of mechatronics, that is, a combination of me-
chanical engineering, electronic engineering, and software engineering. Thus,
they could be expected to have relatively high spatial abilities. We were in-
terested in strategy use in a high-performing group because the preliminary
study focused on successful, rather than unsuccessful, solution strategies.
128 J. GLÜCK, ET AL.
METHOD
Participants
Participants were students attending the 3rd grade (11th year of total
education) of the mechatronics class of a technical secondary school located
near Vienna, Austria. Overall, 42 students participated, three of whom were
female. Participation in the study was voluntary; one week after the test ses-
sion, participants received confidential feedback about their performance.
Materials
Spatial tests.—The participants were presented with four spatial tests.
The tests were chosen according to two criteria. First, they should represent
a broad range of task types within the subfactors visualization and mental
rotation that have been identified in analyses of the structure of spatial abili-
ties (Lohman, 1979; Linn & Petersen, 1985), and second, they should have
been used in other spatial-ability training studies so as to allow for compari-
sons of results.
In the order of presentation, we used the Purdue Spatial Visualization
Test (Guay, 1977), the Mental Cutting Test (College Entrance Examination
Board, 1939), the Mental Rotation Test (Vandenberg & Kuse, 1978; Peters,
Laeng, Latham, Jackson, Zaiyouna, & Richardson, 1995; German form by
Quaiser-Pohl & Lehmann, 2002), and the Differential Aptitude Test: Spatial
Relations (Bennett, Seashore, & Wesman, 1973). All tests are paper-and-
pencil tests that require the mental manipulation of three-dimensional stimu-
lus figures presented as two-dimensional drawings.
Each item of the Purdue Spatial Visualization Test consists of two rows.
In the first row, a sample figure (identical across all items) is shown in an
original and a rotated position. In the second row, a different stimulus fig-
ure is given, and the participant is supposed to determine what it would
look like if rotated around the same angle as the sample figure. In the Men-
tal Cutting Test, the task is to determine what the cut surface looks like if a
given figure is cut along a given plane. In the Mental Rotation Test, the par-
ticipant is supposed to determine the two out of four figures that are struc-
turally identical, although rotated, to a target figure. Each item of the Differ-
ential Aptitude Test: Spatial Relations consists of a two-dimensional figure
and four three-dimensional figures. The task is to determine which of the
three-dimensional figures results from folding the two-dimensional figure.
All tests except the Mental Rotation Test, for which a German form ex-
isted, were translated into German, and the translated test instructions were
evaluated and refined in a small pilot study. In all tests except the Mental
Rotation Test, we constructed two different short forms by dividing the items
according to odd and even item numbers; in some cases, a few items were
retained in both versions. The Mental Rotation Test was presented as a
STRATEGY ASSESSMENT AFFECTS TEST INTERCORRELATIONS 129
whole. This resulted in test lengths of 15 items (12.5 min.) for the Purdue
Spatial Visualization Test, 15 items (12 min.) for the Mental Cutting Test
(where some items were used in both groups), 24 items (6 min.) for the
Mental Rotation Test, and 25 items (12.5 min.) for the Differential Aptitude
Test: Spatial Relations. In all cases, the two test forms were comparable with
respect to means (t test p values ranging from .21 to .91) and standard devi-
ations (F test p values ranging from .51 to .97). Cronbach alphas were .43
and .66 for the two Purdue Spatial Visualization Test short forms,2 .73 and
.65 for the Mental Cutting Test short forms, .80 for the Mental Rotation
Test, and .81 and .90 for the Differential Aptitude Test: Spatial Relations
short forms.
Letter series.—In addition, participants were presented with a letter se-
ries test. As was explained above, we assumed that the spatial performance
of participants using analytic strategies might be substantially correlated with
performance in inductive reasoning tasks. To avoid correlations based on the
visual nature of stimulus materials, or on relationships between spatial ability
and mathematics, we preferred letter series over matrices and number series
tests. We used the subtest "Buchstabenreihen" (letter series) from the Wilde
Intelligence Test (Jager & Althoff, 1983), a standardized German-language
measure of inductive reasoning. The test consists of 15 items; in each item,
the task is to understand the rule by which a series of letters was produced,
and to find the next letter that continues the series correctly.
Strategy questionnaire.—In developing the strategy questionnaire, we
tried to overcome some of the difficulties of strategy self-reports described
above. First, we did not ask participants for global retrospective descriptions
of their strategies in the test they had just completed. Rather, we presented
two or more new test items of varying difficulty, and asked participants to
observe themselves while completing them, and to fill out the questionnaire
after each of these items. Second, we did not use very detailed strategy de-
scriptions, to avoid boredom and fatigue due to long questionnaires, which
could lead to inaccurate responses. As a middle course, the questionnaire
contained relatively general descriptions of two opposed poles (e.g., visualiz-
ing vs reasoning), and participants rated their strategies on an analog scale
between these two poles. Fig. 1 shows the strategy questions for the Mental
Cutting Test. The contents of the strategy descriptions for each test were de-
rived from a think-aloud pilot study with 18 participants (8 men, 10 women,
2As .43 indicates a very low internal consistency, all analyses presented in the following were
also performed using only the most reliable 9 items from each form, and the pattern of correla-
tions remained the same. However, using the short form resulted in a ceiling effect in one of
the two test forms and significant mean differences between the two test forms. Therefore, the
results presented in the rest of the article are based on the total scores in spite of relatively low
reliability.
130 J. GLÜCK, ET AL.
aged 21-39 years) who verbalized their thoughts while solving items of vary-
ing difficulty from each test. Their statements and behaviors were recorded,
transcribed, and categorized into holistic and analytic strategies. The results
were used to formulate the strategy descriptions in the strategy question-
naire. The questionnaires for the Purdue Spatial Visualization Test, Mental
Cutting Test, and Differential Aptitude Test: Spatial Relations contained two
sample test items (one easy and one difficult) and one strategy description
for the test in general; for the Mental Rotation Test, we included four sam-
ple items (two identical and two different), each consisting of only two stim-
ulus figures.
As Fig. 1 shows, the questionnaire assessed several aspects of strategy
use: visualizing vs reasoning (which most closely represents the holistic-ana-
lytic distinction); focusing on the stimulus as a whole vs on details; when
Please judge to what degree you used the two possibilities "Visualizing" vs "Reasoning
when you were working on Item A.
Visualizing: You tried to see the cut and the cut surface with your "inner eye.'
Reasoning: You thought about which attributes the cut surface needs to show.
When working on Item A, to what degree did you include the whole cut surface in your
visualizing and reasoning, or use a stepwise procedure?
If you visualized when working on Item A: To what degree did you move the objects, or
change your own perspective by moving around the objects in your imagination?
Did you directly identify the correct solution, or did you arrive at your solution by excluding
all incorrect alternatives?
FIG. 1. Strategy questionnaire for the Mental Cutting Test
STRATEGY ASSESSMENT AFFECTS TEST INTERCORRELATIONS 131
visualizing, mentally changing the object vs. changing one's own viewpoint;
when visualizing, moving stepwise or gradually. The order of aspects was the
same in all questionnaires. In addition, at the end of each strategy question-
naire participants rated how well the strategy descriptions in the question-
naire described the strategies they had used.
We used two different versions of the questionnaire in the study. In the
first version, which will be referred to as V-R, "Visualizing" was on the left
side of the scale and "Reasoning" was on the right side. Thus, participants
read the description of "Visualizing" first. The sample questionnaire in Fig.
1 is the V-R questionnaire. In the second version, R-V, "Reasoning" was on
the left and "Visualizing" was on the right side, and all other aspects were
reversed in the same way. Each participant was presented with the same ver-
sion for all spatial tests. The two strategy-questionnaire forms were combined
with the two short test forms in a balanced manner: of each test-form group,
half of the participants were presented with the V-R questionnaire, and the
other half with the R-V questionnaire.
Participants gave their responses on a 10-cm visual analog scale (see
Fig. 1). Participants' marks on the visual analog scale were measured in mil-
limetres, thus, values could be between 0 and 100. Values from the R-V
group were reversed, so that for all participants higher means reflected high-
er values in "Reasoning."
Procedure
The test session took place in the normal school context. Over a 90-
minute period, participants took the four spatial tests in the following order:
Purdue Spatial Visualization Test, Mental Cutting Test, Mental Rotation
Test, Differential Aptitude Test: Spatial Relations; after each test, they com-
pleted the strategy questionnaire. The letter series test was presented after
the last spatial test.
RESULTS
The following analyses focus on the first item of the strategy question-
naire, Visualizing vs Reasoning, for two reasons: first, this item reflects the
holistic-analytic distinction best, and second, as it was read first by the par-
ticipants, we assume it to have caused the strongest priming effect. (We
found similar group differences when analyzing the Whole vs Details dimen-
sion, which was significantly correlated with Visualizing vs Reasoning in all
tests.) As a measure of Visualizing vs Reasoning in each spatial test, the
mean of participants' responses to the Visualizing vs Reasoning item across
all sample items was used.
Strategy Questionnaire Results
We first tested for differences in self-reported strategy between the R-V
132 J. GLÜCK, ET AL.
and the V-R group. We found significant differences between the groups on
all four tests (F1,40 = 237.68, p<.01) and significant mean differences between
the tests (F3,38 = 5.68, p<.01). There was no test x group interaction (F3,120 =
.468, p = .71); thus, the mean difference between the groups was the same
across the four tests. Fig. 2 shows the means of the two groups in the strat-
egy questionnaire.
FIG. 2. Means of the two strategy-questionnaire groups in the visualizing-reasoning strat-
egy item for the four spatial tests. Scale as in the original item. Means and standard errors
As Fig. 2 shows, there was a clear effect of the order of strategy presen-
tation in the questionnaire on the strategies participants reported. Partici-
pants who rated their strategies on a scale from Reasoning to Visualizing
rated their strategies significantly closer to Reasoning than participants whose
scale was from Visualizing to Reasoning. At first sight, we interpreted this as
an indication that our participants' strategy reports were largely the result of
the questionnaire format, and therefore largely invalid. This interpretation
changed, however, when we examined the correlation matrices of the spatial
and letter series scores in the two groups.
Test Scores and Inter correlations
First, we tested whether the different versions of the strategy question-
naire affected participants' performance on any of the tests. As Table 1
shows, this was not the case. The order of presentation in the strategy scale
did not have any effects on participants' test scores. It did affect, however,
the inter correlations among the tests, as shown in Table 2.
STRATEGY ASSESSMENT AFFECTS TEST INTER CORRELATIONS 133
TABLE 1
COMPARISON OF MEAN TEST SCORES FOR Two STRATEGY-QUESTIONNAIRE GROUPS,
VISUALIZING REASONING (V-R) AND REASONING-VISUALIZING (REV) (ns = 21)
Test
Purdue Spatial Visualization
Mental Cutting Test
Mental Rotation Test
Differential Aptitude Test
Spatial Relations
Letter Series
Test
No.
Items
15
15
24
25
15
V-E
M
12.14
9.43
16.33
18.90
9.81
SD
2.22
2.80
3.94
5.08
2.60
REV
M
11.95
9.90
14.67
18.81
9.43
SD
1.86
2.62
3.69
3.43
2.60
t40
.302
.568
1.414
.071
.475
Note.—No group differences were significant at p< .05.
Inter correlations among spatial tests.—In the V-R group, inter correlations among the spatial tests were all in the .50 to .70 range (p values
ranging from <.001 to .02), except that between the Purdue Spatial Visual-
ization Test and Mental Rotation Test (r = .27, p = .23). In the REV group, all
TABLE 2
STATISTICAL RELATIONSHIPS AMONG SPATIAL TEST SCORES FOR V-R AND REV GROUPS AND
BETWEEN SPATIAL TEST SCORES AND LETTER SERIES SCORES: CORRELATIONS, FISHER'S Z
FOR GROUP DIFFERENCES IN CORRELATIONS, AND BOOTSTRAP PERCENTILES (NS = 21)
Mental Cutting Mental Spatial Letter
Test Rotation Test Relations Series
V-R
Purdue Spatial Visualization Test
Correlation
Fisher's z
Bootstrap percentile
Mental Cutting Test
Correlation
Fisher's z
Bootstrap percentile
Mental Rotation Test
Correlation
Fisher's z
Bootstrap percentile
.69t
2.484
99
REV
.02
*
3
Differential Aptitude Test: Spatial Relation;
Correlation
Fisher's z
Bootstrap percentile
V-R REV
.27 .22
.160
57 41
.51* .15
1.235
93 14
V-R REV
.50* .22
.977
61 14
.72t .22
2.052*
99 3
.45* .37
.289
64 39
V-R REV
.39 .34
.173
59 43
.05 .41
1.157
18 88
.21 .31
.322
35 63
.21 .57t
1.303
19 92
*p<.05. tp<.01.
correlations were around .20 and below (p values ranging from .34 to .93),
except that between the Mental Rotation Test and Differential Aptitude
Test: Spatial Relations (r = .37, p = .10).
Given the small sample sizes of 21 per group, we assumed that such
134 J. GLÜCK, ET AL.
differences in correlations might easily have been caused by chance. We dealt
with this problem in two ways. First, we z-transformed the correlations and
tested each correlation for group differences using Fisher's z statistic. The
results are shown in the second row for each correlation in Table 2. Second,
we ran bootstrap analyses.
Bootstrap analyses.—Bootstrap analyses (see, e.g., Mooney & Duval,
1993) are a nonparametric means to gain information about the sample dis-
tribution of statistical parameters. They can be used, for example, to eval-
uate the stability of certain parameters when samples are small. In classical
bootstrap analyses, a large number of samples of the same size as the origi-
nal sample are randomly drawn with replacement from the original sample,
and the distribution of a statistical parameter across these samples is then
evaluated. In the present case, we performed bootstrap-type analyses by di-
viding the sample into two equal halves according to 100 different random
number series. In Table 2, the third row for each correlation shows the per-
centile value of the REV and V-R correlations among the 200 random-sample
correlations. For the correlations of Mental Cutting Test with Purdue Spatial
Visualization Test and Differential Aptitude Test: Spatial Relations, the boot-
strap analysis implied a marked difference between the REV and V-R groups,
as the correlation coefficient for V-R is in the top 5% of the bootstrap dis-
tribution, and the correlation for REV is in the bottom 5%. Thus, the boot-
strap results are in very good agreement with the results of Fisher's z test.
In sum, the correlations between Purdue Spatial Visualization Test and
Mental Cutting Test and between Mental Cutting Test and Differential Ap-
titude Test: Spatial Relations were significantly lower in the REV group, who
were first presented with the description of a reasoning strategy, than in the
V-R group, who first read the description of visualizational strategy. The ef-
fect is less pronounced but also visible in the other intercorrelations; in fact,
there were no significant correlations among the spatial tests in the REV
group. The mean correlations among the spatial tests were .52 (bootstrap
percentile 98%) in the V-R group and .20 (bootstrap percentile 3%) in the
REV group.
Correlations between spatial tests and inductive reasoning.—The oppo-
site pattern showed in the correlations with the letter series test. In the V-R
group, there was a correlation of .39 (p = .08) between Purdue Spatial Visu-
alization Test and Letter Series; for all other tests, the correlations were .21
and below (p values from .37 to .82). Note that performance in the Purdue
Spatial Visualization Test could not have been affected by the scale labels
because this test was completed before participants saw the first strategy
questionnaire. In the REV group, the correlations of the spatial tests with
Letter Series ranged from .31 to .57 (p values ranging from .005 to .18). The
correlations with Letter Series were largest for the Mental Cutting Test (I =
STRATEGY ASSESSMENT AFFECTS TEST INTER CORRELATIONS 135
.41, p = .07, bootstrap percentile 88%, compared to the V-R group: r = .05,
p = .82, bootstrap percentile 18%) and for the Differential Aptitude Test:
Spatial Relations (r = .57, p = .008, bootstrap percentile 92%, compared to
the V-R group: r- .21, p = 37, bootstrap percentile 19%). The mean correla-
tion between the spatial tests and letter series was .22 (bootstrap percentile
23%) in the V-R group, and .41 (bootstrap percentile 84%) in the REV
group.
Factor analyses.—In spite of the small sample, we ran factor analyses of
the test scores within each group to get a descriptive notion of the group
differences in structure. The factor analyses were consistent with the correla-
tional analyses. In the V-R group, there was a strong first factor, explaining
54.1% of the variance, on which all the spatial tests loaded (with loadings
ranging from .67 to .94), and a small second factor, explaining 20.0%, on
which there was a strong loading of letter series (.98) and a weak loading of
the Purdue Spatial Visualization Test (.47). In the REV group, only one fac-
tor, explaining 43.8% of the variance, had an eigenvalue above 1. The
loadings of the spatial tests on this factor ranged from .50 (Purdue Spatial
Visualization Test) to .77 (Differential Aptitude Test: Spatial Relations); the
loading of letter series was .84.
Controlling for letter series.—As we have shown, in the group that first
read a description of a Reasoning (analytic) strategy, correlations among the
spatial tests were lower and correlations with the letter series test were
higher than in the group that first read a description of a visualizational (ho-
listic) strategy. If reasoning strategies in spatial tests indeed require deri-
vation and comparison of rules, then the common variance across the spatial
tests in the REV group should be accounted for by letter series, a measure of
inductive reasoning. Although there was little (and nonsignificant) common
variance to begin with, we examined how the spatial test inter correlations in
both groups changed after controlling for letter-series performance. Table 3
shows the results.
TABLE 3
SPATIAL-TEST SCORE INTER CORRELATIONS AFTER CONTROLLING
FOR LETTER SERIES VARIANCE (NS = 21)
Purdue Spatial Visualization Test
Mental Cutting Test
Mental Rotation Test
Mental Cutting
Test
V-R REV
.71t -.15
Mental Rotation
Test
V-R REV
.21 .12
.51* .03
Spatial
Relations
V-R REV
.46* .03
.73t .02
.43 .25
*p<.05. tp<.01.
Correlations in the V-R group remained virtually the same after control-
ling for letter series, whereas the correlations in the REV group, which had
been nonsignificant to begin with (ranging from .02 to .37), decreased even
136 J. GLÜCK, ET AL.
more (to a range from -.15 to .25). Thus, the small common variance among
the spatial tests that had been present in the R-V group was related to in-
ductive reasoning, whereas the much larger common variance in the V-R
group was not related to inductive reasoning. (Note, however, that the
changes in the R-V group were not significant.)
DISCUSSION
In this study, we found effects of the order of strategy descriptions in a
strategy questionnaire on strategy self-reports and on the correlational struc-
ture of spatial and reasoning tasks. Participants were presented with four
different spatial tests and an inductive-reasoning test. After each spatial test,
they filled out a strategy questionnaire. In one version of this questionnaire
(V-R), a Visualizational (holistic) strategy was presented first, whereas in the
other version (R-V), a Reasoning (analytic) strategy was presented first.
The two groups differed in self-reported strategies across all spatial
tests, with each group showing a preference in the strategy that was pre-
sented first in their questionnaire. Beyond this relatively trivial effect, there
were also marked differences in the intercorrelations among the spatial tests.
In the V-R group, intercorrelations among the spatial tests were relatively
high, especially those of the Mental Cutting Test (which was presented after
participants had completed the first strategy questionnaire) with the other
tests (.51 to .72), and correlations with letter series were low. In the R-V
group, correlations among the spatial tests were nonsignificant and, in all
cases but one, smaller than the correlations of each spatial test with letter se-
ries. These results suggest that participants who first read an analytical strat-
egy description used more analytical strategies at least in the next test they
completed, whereas participants who first read a holistic strategy description
shifted to more holistic strategies.
In interpreting these results, we first have to note two limitations. First,
the sample in this preliminary study was small (21 participants per group).
We tried to deal with the sample size issue by performing a bootstrap analy-
sis, that is, by comparing the correlations found in the two strategy-question-
naire groups to correlations obtained when the sample was randomly divided
into two groups. For the future, it seems important to investigate the gener-
alizability of the results with other samples. Second, as the participants were
students of a technical school, they were generally relatively high in spatial
ability (see the mean scores in Table 1). Interestingly, the restricted perfor-
mance variance does not seem to have lowered the test intercorrelations. We
believe that the sample's high performance may have increased the effect of
the strategy descriptions: possibly, only individuals who have a sufficient abil-
ity and strategy repertoire can easily switch strategies. This may also explain
why the priming affected the test intercorrelations, but not test performance:
STRATEGY ASSESSMENT AFFECTS TEST INTERCORRELATIONS 137
the average performance was the same in the two groups although they seem
to have solved the test items in different ways. Possibly, participants with
high spatial ability are able to solve the test items in different ways equally
well.
In spite of the small and selective sample, we believe the results have
important consequences for researchers who want to use strategy-assessment
questionnaires. Our interpretation is that the format of the strategy question-
naire primed the way participants dealt with the subsequent tests.
Participants were first presented with the strategy questionnaire after
they had completed the Purdue Spatial Visualization Test. Thus, the priming
could not have affected the way they solved the items of this test. Accord-
ingly, the Purdue Spatial Visualization Test is the only test for which the cor-
relations with letter series were similar across the two groups. Notably, how-
ever, the differences in self-reported strategy use were as large in the Purdue
Spatial Visualization Test as in the other tests. Thus, when participants, after
completing the Purdue Spatial Visualization Test, first read the strategy ques-
tionnaire, the strategy described first seems to have had an immediate effect
on how they thought they had solved the test items, as well as on how they
actually did solve the items of subsequent tests. This finding renders the use-
fulness of strategy questionnaires highly questionable: even high-ability par-
ticipants' awareness of their solution strategies seems to be rather low and,
therefore, they may be highly susceptible to scale-label effects and the like
(overview in Schwarz, 1999).
The effect found here was, however, more than a simple scale-label ef-
fect where participants simply choose the response in the questionnaire that
they have read first. Our results show that after giving such responses, par-
ticipants actually shifted their strategy use in the subsequent spatial test in
the respective direction. Thus, we believe that some kind of priming oc-
curred: just as participants who just have been thinking about university pro-
fessors solve more items in a knowledge test than participants who thought
about football hooligans (Dijksterhuis & van Knippenberg, 1999), partici-
pants who just thought that they had been using an analytic strategy actually
did use one in the next test they were given.
One puzzle that the results pose is what exactly the analytic strategies
are. If analytic strategies, as we originally assumed, largely consist of extract-
ing and comparing structural rules of the spatial stimuli, then the intercorre-
lations among the spatial tests should still have been high in participants us-
ing analytical strategies. The only difference should be that letter series
would explain the common variance of the spatial tests in the R-V, but not
in the V-R group. While we did find the latter, the intercorrelations among
the spatial tests in the R-V group were small and nonsignificant even before
controlling for letter series, whereas the intercorrelations in the V-R group
138 J. GLÜCK, ET AL.
were relatively high. The picture of intercorrelations among spatial tests in
the literature is somewhat inconsistent: Vandenberg and Kuse (1978) report-
ed correlations across ten different spatial tests between -.02 and .46, with a
mean correlation of .25. Juhel (1991) found correlations between .16 and .54
across five tests; the mean correlation was .31. Poltrock and Brown (1984)
found correlations between .35 and .65 across eight tests; the mean correla-
tion was .47. Thus, the correlations found in the V-R group (mean correla-
tion: 52) are at the upper end, whereas the correlations in the R-V group
(mean correlation: .20) are at the lower end of the distribution of correla-
tions in previous studies. A possible explanation is that while holistic strate-
gies draw upon relatively similar (visualizational) cognitive processes inde-
pendent of the type of spatial task, analytic strategies may be much more de-
pendent on specific task requirements or on personal characteristics.
Generally, we draw two conclusions from our findings. First, research-
ers who want to study strategy use need to be very careful in designing their
strategy measures. Open formats, especially think-aloud measures, seem to
be preferable to questionnaires. Even with open formats such as thinking
aloud, however, people's accounts of their own strategies in one test may af-
fect the way they solve another test. Also, in addition to self-reports, other
measures such as eye movements or response times should be collected (see
above). Perhaps we can find ways to construct test items that allow for bet-
ter strategy assessments—for example, tests where different strategies lead to
different types of errors.
Second, our results suggest that the strategies people use in spatial tasks
may be relatively easy to change, be it by test instructions emphasizing cer-
tain strategies or by more specific explicit or implicit strategy instructions.
While this clearly implies a caveat in methodological considerations, it may
also open up new possibilities for research on strategy use in spatial tasks.
The high-ability participants in the current study seem to have been easily
able to switch between strategies. The degree to which individuals are able
to comply with different strategy instructions may be an indicator of the
breadth of their available strategy repertoire. As the effectiveness of different
strategies is highly dependent on task characteristics (Lohman & Kyllonen,
1983; Schultz, 1991), the best performers across different spatial tasks may
not always be those who are best at using holistic strategies, but those who
are best at selecting the optimal strategies for each task.
To conclude, we believe that strategy use is an important and interest-
ing issue that may help us to find substantive explanations of individual
differences in spatial ability (Glück & Fitting, 2003). The optimal method of
assessing strategy use, however, has not yet been identified.
STRATEGY ASSESSMENT AFFECTS TEST INTERCORRELATIONS 139
REFERENCES
BARRATT, B. S. (1953) An analysis of verbal reports of solving spatial problems as an aid in de-
fining spatial factors. The Journal of Psychology, 36, 17-25.
BENNETT, G. K., SEASHORE, H. G., &WESMAN, A. G. (1973) Differential Aptitude Tests, Forms S
and T. New York: The Psychological Corp.
BOWERS, C. A., &LABARBA, R. C. (1988) Sex differences in the lateralization of spatial abilities:
a spatial component analysis of extreme group scores. Brain and Cognition, 8, 165-177.
COCHRAN, K. F, & WHEATLEY, G. H. (1989) Abiltiy and sex-related differences in cognitive
strategies on spatial tasks. The Journal of General Psychology, 116, 43-55.
COHEN, D., & KUBOVY, M. (1993) Mental rotation, mental representation and flat slopes. Cog-
nitive Psychology, 25, 351-382.
COLLEGE ENTRANCE EXAMINATION BOARD. (1939) Special Aptitude Test in Spatial Relations
Mental Cutting Test. Princeton, NJ: CEEB.
DEFFNER, G. (1985) Data on solution strategies from eye-movement recordings. In R. Groner,
G. McConkie, & C. Menz (Eds.), Eye movements and human information processing. Pro-
ceedings of the 23rd International Congress of Psychology, Acapulco. Pp. 309-322.
DIJKSTERHUIS, A., &VAN KNIPPENBERG, A. (1999) The relation between perception and behav-
ior, or how to win a game of Trivial Pursuit. Journal of Personality and Social Psychology,
74, 865-877.
ERICSSON, K. A., &SIMON, H. A. (1980) Verbal reports as data. Psychological Review, 87. 215-
251,
ERICSSON, K. A., & SIMON, H. A. (1993) Protocol analysis: verbal reports as data. Cambridge,
MA: The MIT Press.
FRENCH, J. W. (1965) The relationship of problem solving styles to the factor composition of
tests. Educational and Psychological Measurement, 25, 9-28.
GLÜCK, J. (1999) Spatial Strategies—Kognitive Strategien bei räumlichen Leistungen [Spatial
strategies—strategy use in spatial cognition]. Unpublished dissertation, Univer. of Vienna,
Austria.
GLÜCK, J., & FITTING, S. (2003) Spatial strategy selection: interesting incremental information.
International Journal of Testing, 3, 293-308.
GLÜCK, J., MACHAT, R., JIRASKO, M., & ROLLETT, B. (2002) Training-related changes in solution
strategy in a spatial test: an application of item response models. Learning and Individual
Differences, 13, 1-22.
GRIMSHAW, G. M., SITARENIOS, G., & FINEGAN, J. A. K. (1995) Mental rotation at 7 years: rela-
tions with prenatal testosterone levels and spatial play experience. Brain and Cognition,
29, 85-100.
GUAY, R. (1977) Purdue Spatial Visualization Tests. West Lafayette, IN: Purdue Research Foun-
dation.
HIGGINS, E. T., BARGH, J. A., & LOMBARDI, W. (1985) Nature of priming effects on categoriza-
tion. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11, 59-69.
HOSENFELD, I., STRAUSS, B., & KÖLLER, O. (1997) Geschlechtsdifferenzen bei Raumvorstellungs-
aufgaben—eine Frage der Strategic [Gender differences in spatial tasks—a question of
strategy use]? Zeitschrift fur Pädagogische Psychologie, 11, 85-94.
JÄGER, A. O., &ALTHOFF, K. (1983) Der Wilde-Intelligenz-Test (WIT). Em Strukturdiagnostikum
[The Wilde Intelligence Test (WIT). A structural assessment]. Göttingen: Hogrefe.
JOHNSTON, C. W., &PIROZZOLO, F. J. (1981) Eye movements and cognitive strategies. Perceptual
and Motor Skills, 53, 623-632.
JUHEL, J. (1991) Spatial abilities and individual differences in visual information processing. In-
telligence, 15, 117-135.
JUST, M. A., & CARPENTER, P. A. (1985) Cognitive coordinate systems: accounts of mental rota-
tion and individual differences in spatial ability. Psychological Review, 92, 137-172.
KÖLLER, O., ROST, J., &KÖLLER, M. (1994) Individuelle Unterschiede beim Lösen von Raum-
vorstellungsaufgaben aus dem IST-bzw. IST-70-Untertest, "Würfelaufgaben" [Individual
differences in solving spatial-ability tasks from the IST and IST-70 subtest "Cubes"].
Zeitschrift für Psychologie, 202, 65-85.
KYLLONEN, P. C, LOHMAN, D. F, & SNOW, R. E. (1984) Effects of aptitude, strategy training,
140 J. GLÜCK, ETAL.
and task facets on spatial test performance. Journal of Educational Psychology, 76, 130-
145.
LEVY, B. (1996) Improving memory in old age through implicit self-stereotyping. Journal of
Personality and Social Psychology, 71, 1092-1107.
LINN, M. C, & PETERSEN, A. C. (1985) Emergence and characterization of sex differences in
spatial ability: a meta-analysis. Child Development, 56, 1479-1498.
LOHMAN, D. F. (1979) Spatial ability: a review and reanalysis of the correlational literature.
(Technical Report No. 8). Stanford, CA: Stanford Univer. School of Education, Aptitude
Research Project.
LOHMAN, D. F., &KYLLONEN, P. C. (1983) Individual differences in solution strategy on spatial
tasks. In D. F. Dillon & R. R. Schmeck (Eds.), Individual differences in cognition. New
York: Academic Press. Pp. 105-135.
MOONEY, C. Z., & DUVAL, R. D. (1993) Bootstrapping: a nonparametric approach to statistical in-
ference. Newbury Park, CA: Sage.
MUMAW, R. J., PELLEGRINO, J. W., KAIL, R. V., & CARTER, P. (1984) Different slopes for different
folks: process analysis of spatial aptitude. Memory & Cognition, 12, 515-521.
PETERS, M., LAENG, B., LATHAM, K., JACKSON, M., ZAIYOUNA, R., & RICHARDSON, C.
(1995)
A re-
drawn Vandenberg and Kuse Mental Rotation Test: different versions and factors that af-
fect performance. Brain and Cognition, 28, 39-58.
PIERRET, A., & PERONNET, F. (1994) Mental rotation and mirror-image discrimination. Percep-
tual and Motor Skills, 78, 515-524.
POLTROCK, S. E., & BROWN, P. (1984) Individual differences in imagery and spatial ability. Intel-
ligence, 8, 93-138.
QUAISER-POHL, C, & LEHMANN, W. (2002) Der Mental-Rotation-Test [The Mental Rotations
Test]. Frankfurt/Main: Swets Test Services.
SCHULTZ, K. (1991) The contribution of solution strategy to spatial performance. Canadian
Journal of Psychology, 45, 474-491.
SCHWARZ. N. (1999) Self-reports: how the questions shape the answers. American Psychologist,
54, 93-105.
SHIH, M., & PITTINSKY, T. L. (1999) Stereotype susceptibility: identity salience and shifts in
quantitative performance. Psychological Science, 10, 80-83.
STEELE, C. M., &ARONSON, J. (1995) Stereotype threat and the intellectual test performance of
African Americans. Journal of Personality and Social Psychology, 69, 797-811.
THURSTONE, L. L. (1938) Primary mental abilities. Chicago, IL: Univer. of Chicago Press.
VANDENBERG, S. G., & KUSE, A. R. (1978) Mental rotations: a group test of three-dimensional
spatial visualization. Perceptual and Motor Skills, 47, 599-604.
YUILLE, J. C, & STEIGER, J. H. (1982) Nonholistic processing in mental rotation: some evi-
dence. Perception & Psychophysics, 31, 201-209.
Accepted January 3, 2007.
