The Sorting Paired Features Task: A Measure of Association Strengths

Abstract

The sorting paired features (SPF) task measures four associations in a single response block. Using four response options (e.g., good-Republicans, bad-Republicans, good-Democrats, and bad-Democrats), each trial requires participants to categorize two stimuli at once to a category pair (e.g., wonderful-Clinton to good-Democrats). Unlike other association measures, the SPF requires simultaneous categorization of both components of the association in the same trial. Providing measurement flexibility, it is sensitive to both focal, attended concepts and nonfocal, unattended stimulus features (e.g., gender of individuals in a politics SPF). Three studies measure race, gender, and political evaluations, differentiate automatic evaluations between known groups, provide evidence of convergent and discriminant validity with other attitude measures, and illustrate the SPF's unique measurement qualities.

Full text

The Sorting Paired Features Task
A Measure of Association Strengths
Yoav Bar-Anan,
1
Brian A. Nosek,
1
and Michelangelo Vianello
2
1
University of Virginia, Charlottesville, VA
2
University of Padua, Italy
Abstract. The sorting paired features (SPF) task measures four associations in a single response block. Using four response options (e.g., good-
Republicans, bad-Republicans, good-Democrats, and bad-Democrats), each trial requires participants to categorize two stimuli at once to a
category pair (e.g., wonderful-Clinton to good-Democrats). Unlike other association measures, the SPF requires simultaneous categorization of
both components of the association in the same trial. Providing measurement flexibility, it is sensitive to both focal, attended concepts and
nonfocal, unattended stimulus features (e.g., gender of individuals in a politics SPF). Three studies measure race, gender, and political
evaluations, differentiate automatic evaluations between known groups, provide evidence of convergent and discriminant validity with other
attitude measures, and illustrate the SPF’s unique measurement qualities.
Keywords: implicit measures, automatic association, automatic attitudes, attitude measures, priming
Associations between concepts are related to thought and
behavior. A recurrent good feeling after tasting something
sweet, for example, may cause a strong association between
the concepts ‘‘sweet’’ and ‘‘good.’’ This association, in turn,
may bolster the thought that sweet taste is good (attitude)
and the tendency to seek sweet tastes (behavior). Because
of their relations to thought and behavior, associations play
a prominent role in psychological theory and application
(Wyer, 2007).
Measures of association strengths use distinct procedures
and may assess a variety of associative processes. The implicit
association test (IAT; Greenwald, McGhee, & Schwartz,
1998) and evaluative priming (EP; Fazio, Jackson, Dunton,
& Williams, 1995) are both used as measures of associations
between targets and valence, but they may reflect distinct
aspects of association because of their idiosyncratic proce-
dural features (Olson & Fazio, 2003). Because psychological
constructs are unobservable and only inferred through mea-
surement, the measures themselves shape the theoretical
understanding of constructs. This interdependence of theory
and measurement encourages method diversity to parse the
variation in measurement that is construct-valid versus
method-specific (Campbell & Fiske, 1959; Nosek & Smyth,
2007). Further, if multiple methods represent different aspects
of a heterogeneous construct, then research efforts can employ
the method best suited for the theoretical question.
This article presents the sorting paired features (SPF)
task, a measure of associations that has unique properties
for research application, and may advance theoretical under-
standing of associations and its derivative social constructs –
attitudes, stereotypes, and self-concepts (Greenwald et al.,
2002). This research examines the validity of the SPF, pre-
sents its unique qualities as a measure of associations, and
demonstrates its potential in revealing findings that cannot
be detected easily by other measures.
The SPF
Measures of association rely on the fact that the processing
of a stimulus increases the accessibility of associated con-
cepts (Higgins, 1996). Association measures that use
response latency or error rates as dependent variables create
task demands for which the presence of an association will
either facilitate or impede performance. For instance, in EP,
a prime immediately precedes the presentation of a target
that is categorized as ‘‘good’’ or ‘‘bad.’’ If the prime acti-
vates ‘‘good’’ then categorizing good targets as good will
be facilitated (i.e., faster or more accurate), and categorizing
bad targets as bad will be impeded (i.e., slower or more inac-
curate). The SPF comprises a single task with four response
alternatives that represent four associations. Comparison
of the latency of performance between these responses
provides an index of their comparative association strengths.
In the SPF, four category pairs are presented in the top-
left, top-right, bottom-left, and bottom-right corners of the
screen. For example, in a task measuring associations
between pets (dogs and cats) and valence (good and bad),
the category pairs would be dogs-good, dogs-bad, cats-
good, and cats-bad. Each category pair corresponds with a
response key (e.g., ‘‘q’’, ‘‘p’’, ‘‘c’’, and ‘‘m’’) on a standard
keyboard. A block of SPF trials involves categorizing pairs
of pet-valence stimulus items into one of the four category
pairs using the response keys. Each trial presents a stimuli
pair at the center of the screen, for example, one valence
item and one pet item. Figure 1 illustrates the computer
screen for a single trial of a pet-valence task (see http://
www.briannosek.com/spf/ for a demonstration).
Participants categorize the items conjointly into one of
the four category pairs as quickly as possible. A red ‘‘X’’
appears below the stimuli after mistakes. Participants must
correct the error to finish the trial. Faster categorization for
2009 Hogrefe & Huber Publishers Experimental Psychology 2009; Vol. 56(5):329–343
DOI: 10.1027/1618-3169.56.5.329

items representing one category pair compared to another
indicates stronger associations between the first category
pair in comparison to the other category pair.
An appropriate response requires that participants attend
to the target pair stimuli, process each stimulus, categorize
each stimulus to one of the presented categories, identify
the category pair among the four available response options,
and make the behavioral response corresponding to that cat-
egory pair. The conjoint or interactive processing of the two
stimuli presents an opportunity for the processing of one
stimulus to influence the processing of the other stimulus.
Categories and stimulus pairs that are highly associated
might facilitate each other’s processing, whereas stimulus
pairs that are not associated might interfere with each other.
Effects of proactive interference may disrupt the speed and
accuracy of categorization (e.g., Craik & Birtwistle, 1971)
or slow down the appropriate behavioral response.
Features of the SPF
Simultaneous Processing of the Two
Association Units
This research validates a novel approach to measuring asso-
ciations – require participants to categorize the two relevant
stimuli simultaneously. Other tasks require processing of
just one stimulus per response. The association strength
between concepts is inferred by the presumed influence of
a task-irrelevant presentation of a second stimulus (e.g.,
priming or Stroop), or a response pairing in which items rep-
resenting two categories require the same behavioral
response on iterative occasions throughout a response block
such as the IAT, go/no-go association task (GNAT; Nosek &
Banaji, 2001), and the extrinsic affective Simon task (EAST;
De Houwer, 2003).
Direct Manipulation of Categorization of the
Stimuli
Social stimuli belong to multiple categories. Hillary Clinton
is a politician, Democrat, female, US citizen, and White.
These categories independently, or in conjunction, can influ-
ence the evaluation of Hillary Clinton. Likewise, presenting
an image of Hillary Clinton could activate evaluations of
one or more of these categories or their conjunctions. This
research tests and validates that SPF performance is influ-
enced by associations between the focal, relevant categories
used in the sorting task, and between nonfocal, incidental
categories that are not highlighted or relevant for task
performance.
Some association measures, like sequential priming,
present primes without constraining their interpretation by
the participants. Only the target stimulus is categorized
explicitly, sometimes on a relevant feature (good or bad)
and, in other cases, on an irrelevant feature (word or non-
word). Because there are no categorization constraints on
prime stimuli, priming is thought to be heavily influenced
by individual stimulus features, and individual differences
in categorization tendencies (Olson & Fazio, 2003).
Other tasks, like the IAT and GNAT, involve active cat-
egorization of all stimulus items into a defined set of super-
ordinate categories. Proper task performance requires a
specific interpretation of each stimulus item – defined by
the category label. Consequently, evaluations of stimuli
are stronger indicators of the superordinate categories than
other features of the stimulus items (Nosek, Greenwald, &
Banaji, 2007). The unique influences of stimulus items
appear to involve shaping the construal of the superordinate
categories rather than eliciting stimulus-specific effects
(Nosek, Greenwald, & Banaji, 2005).
The SPF appears to be a blend of these EP and IAT fea-
tures. Like the IAT, the SPF constrains stimulus interpreta-
tion with superordinate categories. However, because the
SPF requires processing of two stimuli at once, other fea-
tures of the stimulus items may have opportunity for mutual
influence. We predicted that the SPF would be sensitive to
associations between the superordinate categories, and to
associations between individual, item-level features that
are independent of the categories. For instance, in a task
with women-bad, women-good, men-bad, men-good as
the ‘‘focal’’ categorization groups, the processing of each
face stimulus may be sensitive to nongender properties of
the stimuli such as ethnicity. As such, in addition to mea-
suring the associations between the focal categories, the
SPF may be effective in measuring ‘‘nonfocal’’ associations
that are incidental, not obvious, or even not consciously
identified by the participant. This provides a unique meth-
odological opportunity to manipulate the focus of attention
and measure its effects on associations for various
categories.
Figure 1. Illustration of an SPF trial as it appears on the
computer screen. The gray font was actually green. The
correct response is ‘‘C’’, because the target word pleasant
belongs to the category good and the picture belongs to the
category dogs.
330 Bar-Anan et al.: The Sorting Paired Features Task
Experimental Psychology 2009; Vol. 56(5):329–343 2009 Hogrefe & Huber Publishers

Eliminating Procedural Effects Due to
Separate Response Conditions
Measures that compare two (or more) independent response
blocks are vulnerable to the effects of learning, practice, fati-
gue, distraction, interference, or changes in response strategy
from one response block to the next. Many measures, such
as the IAT or GNAT, are vulnerable to these influences
(Nosek & Banaji, 2001; Nosek et al., 2005). Like EP, the
SPF measures all four associations in a single response
block. Therefore, any irrelevant factors that change across
time will have the same effect on all the measured associa-
tions. This does not eliminate the possibility that these fac-
tors influence performance, in general, but it does eliminate
the selective influence on one association assessment com-
pared to others.
Estimation of Each Association
The IAT is constrained to a single index indicating
aggregated relative association strengths of two pairs of
associations (e.g., white-good and black-bad compared to
white-bad and black-good, Nosek & Sriram, 2007). The
SPF may enable separable assessments of the four associa-
tion strengths. The ease of response to each of the four con-
ditions (white-good, white-bad, black-good, and black-bad)
is affected by the strength of the association processed in
that condition. As such, the SPF may be able to distinguish,
for example, an individual who has comparatively strong
white-good associations from the one that has comparatively
strong black-bad associations.
Importantly, like all other association measures, the four
association strengths assessed in the SPF are not interpret-
able in isolation. Each response to a pair might be influenced
by the remaining three pairs of stimuli and labels. Scores for
the four associations are algebraically dependent. Calculat-
ing separate associations does not guarantee that the separate
assessments are valid (such as prior efforts to calculate
unique components from IAT performance, Nosek et al.,
2005). Nevertheless, the present research demonstrates that
the separate estimations add information that the IAT cannot
provide. We find that the distinct association estimates –
within the same task – have different internal consistencies,
relations with other variables, and are not symmetrical (i.e.,
the association white-good is not the exact opposite of the
association black-good or white-bad, and is not equal to
the association black-bad).
Overview of This Report
We conducted three studies to evaluate the features, reliabil-
ity, and validity of the SPF. Study 1 was a small-scale lab
study that tested the SPF as a measure of automatic race atti-
tudes. It also tested whether the SPF can measure automatic
attitudes about ‘‘nonfocal’’ attributes that were not men-
tioned explicitly in the task, nor relevant for task perfor-
mance. Study 2 was a large-scale web study that
examined the relation between the SPF (automatic race or
gender attitudes) and self-reported attitudes. It tested the
measure’s sensitivity to focal and nonfocal features of the
stimuli and convergent validity with self-reported attitudes.
Study 3 examined the relations among the SPF, the IAT,
and self-report measures of political attitudes to evaluate
convergent validity with other measures of association and
evaluation.
Finally, we demonstrate the potential theory develop-
ment benefits of the SPF by reporting a serendipitous find-
ing revealed by the SPF’s unique measurement features.
Associations of concepts with positive valence, in compari-
son to the associations with negative valence, may be more
reliable, more related to self-reported attitudes, and have lar-
ger effects on automatic evaluation.
Study 1
Method
Participants
In 1999, 16 students (8 women) at Yale University partici-
pated for course credit.
Materials
There were four groups of stimuli: 24 good words (e.g.,
‘‘wonderful’’ and ‘‘triumph’’), 24 bad words (e.g., ‘‘terrible’’
and ‘‘hate’’), 42 pictures of White people (21 females and 21
males), and 42 pictures of Black people (21 females and 21
males). The pictures were taken from 1998 to 1999 NBA
and WNBA player and coach image repositories. We
selected people that were unlikely to be recognized by any-
one but dedicated basketball fans.
Procedure
Participants performed the task in individual cubicles. The
instructions appeared on the computer screen. For each trial
(illustrated in Figure 1 with dogs and cats), two items
appeared in the middle of the screen – a face and a word.
The faces consisted of White and Black people, and the
words represented good and bad concepts. Participants cat-
egorized the face-word pairs into their appropriate category
as quickly as possible. The four category pairs (black-good,
black-bad, white-good, and white-bad) appeared at the top
left, top right, bottom left, and bottom right of the screen.
Participants put their left pinky and index finger on the
‘‘Q’’ and ‘‘C’’ keys, respectively, and their right pinky and
index finger on the ‘‘P’’ and ‘‘M’’ keys, respectively, on a
standard QWERTY keyboard.
Each pair of target stimuli remained on the screen until it
was categorized correctly. After an error, a red X appeared
below the stimuli and remained until the participant
Bar-Anan et al.: The Sorting Paired Features Task 331
2009 Hogrefe & Huber Publishers Experimental Psychology 2009; Vol. 56(5):329–343

corrected it. The next pair of target stimuli appeared 300 ms
after correct categorization.
The task consisted of four blocks, each with 48 trials. All
blocks presented the same four response options. The loca-
tions of the four category pairs were counterbalanced
between blocks. The four counterbalanced response assign-
ments for top-left, top-right, bottom-left, and bottom-right
responses were: (1) white-good, white-bad, black-good,
and black-bad; (2) white-bad, black-bad, white-good, and
black-good; (3) black-bad, black-good, white-bad, and
white-good; and (4) black-good, white-good, black-bad,
and white-bad. The order of the blocks was randomized
between participants. At the beginning of each block, the
first four trials presented one of each of the four category
pairs, to facilitate the participants’ learning of the key
assignments. The remaining trials were randomized with
the constraint that each of the four category pairs was pre-
sented an equal number of times.
Analysis Strategy
In all the studies, the response latency was the time between
the target stimuli onset and the correct response, regardless
of whether the participant made an incorrect response first.
The analyses included all trials with latency longer than
400 ms and shorter than 5,000 ms (average of 1.1% trials
removed for each participant; none had > 10% trials outside
this range). We log-transformed response latencies prior to
aggregating data (untransformed latency means are reported
in text).
Results and Discussion
Themeanreactiontimewas1,327ms(SD =318). The
mean error rate was 0.15 (SD = 0.09). The mean latency
for the four response conditions are given in Table 1. Based
on prior research of automatic racial attitudes (e.g., Nosek,
Smyth et al., 2007), we expected stronger preference for
Whites compared to Blacks, though it was uncertain
whether these preferences would be manifest as differences
in more positive associations for Whites, negative associa-
tions for Blacks, or both.
A2(race)·2 (valence) analysis of variance (ANOVA)
representing the four associations revealed a significant
interaction between race and valence, F(1, 15) = 12.70,
p<.01,g
p
2
= .45, suggesting that the two races were differ-
ently associated with the two valences. This interaction was
caused by a significant difference between the two white
associations (faster responses to white-good than to white-
bad), F(1, 15) = 35.60, p= .0001, g
p
2
= .70, and no differ-
ence between the two black associations, F< 1. Followup
comparisons found that white-good associations were not
significantly stronger than black-good, F(1, 15) < 1, but
black-bad was stronger than white-bad, F(1, 15) = 40.47,
p< .0001, g
p
2
= .73.
The ANOVA also found a main effect of valence,
F(1, 15) = 8.97, p<.01,g
p
2
= .37 (faster responses when
good was one of the concepts), and no effect of race on
the performance, F(1, 15) < 1. Given the pattern of means,
the main effect of valence may have resulted from the strong
white-good association and the weak white-bad association,
and may not be a reflection of stronger good associations, or
faster responses for pairs that contain good items, in general.
With just 16 participants, most differences were esti-
mated reliably and with strong effect magnitudes suggesting
that the SPF was effective in distinguishing association
strengths. This pattern of results provides a more nuanced
accounting of race-evaluation associations than is possible
with most other measures. We observed a positive evalua-
tion of White people, a relatively neutral evaluation of Black
people, and a stronger association for Black people with
bad, than for White people with bad. In summary, these
results are consistent with preference and evaluation effects
observed with other paradigms, with the additional benefit
of estimating separate association strengths.
Nonfocal Gender Associations
The focal categories (i.e., the categories identified by the
response labels and the basis for categorization) were Black
and White people. However, half of the black and white
faces were women and half were men. Previous research
with the IAT finds that women are implicitly preferred to
men on average (Nosek, 2005). In the SPF, it is possible
to estimate associations for nonfocal categories. The laten-
cies for each of the four Sex ·Valence conditions (Table 1)
were subjected to a gender (2) ·Valence (2) ANOVA.
There was a nonsignificant interaction effect,
F(1, 15) = 3.06, p= .10, g
p
2
=.14. A marginal effect of
gender, F(1, 15) = 4.19, p= .06, g
p
2
= .23, reflected faster
responses when men was the attitude object in the associa-
tion, rather than women. There was a main effect of evalu-
ation, F(1, 15) = 10.42, p<.01,g
p
2
= .40, reflecting faster
responses when good was the evaluative concept in the
association. The effects were driven by a significant differ-
ence between the weakest association women-bad, and all
the other three associations, g
p
2
’s > .31, suggesting that it
was more difficult to associate women with negativity than
any other association. The three stronger associations did
not differ from each other significantly. Notably, this pro-
women effect was observed even though participants did
not explicitly categorize the faces in terms of gender.
In this study, the focal category was race and we identi-
fied gender as an influential nonfocal category. The pro-
white effects were stronger than the pro-women effects.
Table 1. Study 1: Mean latency for each Valence ·Cat-
egory condition (standard deviation in
parentheses)
Attribute/category Good Bad
Black 1,330 (387) 1,287 (334)
White 1,251 (300) 1,442 (335)
Women 1,299 (334) 1,421 (356)
Men 1,278 (317) 1,302 (322)
332 Bar-Anan et al.: The Sorting Paired Features Task
Experimental Psychology 2009; Vol. 56(5):329–343 2009 Hogrefe & Huber Publishers

It could be tempting to conclude that automatic racial eval-
uations are stronger than gender evaluations. However, we
expect that the focal category has a greater influence in auto-
matic evaluations than nonfocal categories that are irrelevant
to task performance. That is, automatic evaluative process-
ing should be primarily a function of the concepts that are
driving categorization. Irrelevant stimulus features may still
provoke automatic evaluations, but not as strongly because
they do not assist with task performance, and may vary in
their accessibility. This was tested systematically with a
high-powered design in Study 2.
Study 2
In Study 2, we used a constant set of stimuli (black and
white women and men), and manipulated the focal concept
as race or gender. We compared the measurement of the
same associations – valence with race and valence with gen-
der – when either race or gender was the focal concept. If
the measured associations vary as a function of the focal
condition, then attention and deliberate categorization may
be important influences in the assessment of automatic asso-
ciations. The SPF’s ability to manipulate attention away or
toward a particular category might then provide opportuni-
ties to test whether focal or nonfocal associations have pre-
dictive validity in different circumstances.
In this study, SPF validity was tested in two ways. First,
participants varied in their social group. There were Black
participants, White participants, women, and men. We
expected stronger preferences for one’s own social group
relative to others (Tajfel & Turner, 1986). We also compared
the SPF with self-reported attitudes toward social groups,
and thermometer ratings toward individuals that belong to
these social groups. Previous research finds that some impli-
cit measures, such as EP, elicit weak correlations with self-
reported racial attitudes (Fazio et al., 1995), but this may be
due, in part, to low reliability of the measure (Bosson,
Swann, & Pennebaker, 2000; Cunningham, Preacher, &
Banaji, 2001; Olson & Fazio, 2003). Other implicit mea-
sures, such as the IAT, show weak-to-moderate correlations
with racial attitudes apparently depending, in part, on the
heterogeneity of the sample (Nosek, 2007). Self-reported
and IAT-measured gender attitudes were unrelated in
previous investigations (Nosek, 2005), so we expected no
relation here as well.
Method
Participants
Two thousand and four hundred volunteers at Project Impli-
cit (https://www.implicit.harvard.edu; see Nosek, 2005)
completed at least one measure. 2,074 participants (87%)
finished the whole session (61% women, 38% men, 1%
unknown; 69% white, 8% black, 12% other, 11% unknown;
Mage = 31.1, SD = 11.9). All participants were included
in the analyses, whether they completed all measures
or not.
1
Measures
Stimuli
The two evaluative groups were five good words (awesome,
glorious, excellent, wonderful, and pleasant) and five bad
words (horrible, awful, terrible, evil, and nasty). The attitude
objects were labeled as either Black people/White people or
women/men. The stimuli were 20 faces of famous Ameri-
cans (5 black women, 5 black men, 5 white women, and
5 white men; see supplement, http://www.briannosek.com/
spf/). We used famous people to test whether evaluations
of each individual has an influence on performance.
SPF
The SPF design was the same as in Study 1, with the follow-
ing differences: (a) the task consisted of three identical
blocks of 40 trials each, (b) the spatial locations of the cat-
egory label pairs were constant throughout the experiment
and manipulated between-subjects, (c) the inter-stimulus
interval was 250 ms, and (d) the attitude object labels were
either men/women or Black people/White people, manipu-
lated between participants.
The locations of the four category pairs were counterbal-
anced between participants. The counterbalanced response
assignments were all possible assignments with the con-
straint that the pairs of each concept (e.g., good-white and
good-black for the concept good) never appeared diagonally
separated.
Self-Reported Attitudes Toward Social Groups
Participants rated their feelings for the social groups – Black
people, White people, women, men, black men, black
women, white men, and white women – on a scale from
0 (the coldest) to 8 (the warmest). All groups appeared on
one page, in a randomized order for each participant.
Self-Reported Attitudes Toward Individuals
Participants rated their feelings toward each of the 20 indi-
viduals that appeared in the SPF, on the same thermometer
scale (0–8). The stimuli were presented 10 at a time, on two
1
There was no difference in sex, age and political identification between participants who did not complete any measure and participants
who completed at least one measure, ts < 1. People who completed the self-report but did not complete the SPF did not differ from the
people who completed the SPF in any of the self-reported measures, ps > .31.
Bar-Anan et al.: The Sorting Paired Features Task 333
2009 Hogrefe & Huber Publishers Experimental Psychology 2009; Vol. 56(5):329–343

pages that appeared one after the other (in a randomized
order).
Demographics
Participants completed a demographics questionnaire when
they registered at Project Implicit from minutes to months
before they were randomly assigned to this study. This study
used the items age, sex, race, and political identity (7-point
scale; strongly liberal to strongly conservative).
Design
All the independent variables were selected randomly for
each participant, and each was selected orthogonally to the
other variables. The overall design included 192 conditions:
(2) SPF focal categories (black/white or men/women focus
concepts) ·(6) order of measures ·(8) locations of cate-
gory pairs in the SPF ·(2) social group division of individ-
uals’ thermometer pages (black/white or men/women). Most
of the interactions between these factors are unimportant for
the present purposes.
2
Procedure
Participants completed the three measures (SPF; self-
reported attitudes about social groups, and self-reported atti-
tudes about the individuals) in a random order. There was no
effect of order on any of the analyses reported below.
Results
The mean reaction time was 1,428 ms (SD = 330) and the
mean error rate was .09 (SD = 0.07). Ninety-six participants
(4.5%) with below chance accuracy rate (0.25; three partic-
ipants) or more than 1/6 of the trials outside of the analyzed
latency range (400–5,000 ms, 93 participants) were omitted
from the SPF analyses. Because of the large sample size,
every test reported is significant with p< .0001, unless
noted otherwise.
Self-Report Measures
Summary effects for the self-report measures are given
in Table 2. When providing ratings of the social groups, par-
ticipants reported preferences for Whites and women over
Blacks and men, respectively. Averaging the individual ther-
mometer ratings of the people used in the SPF by race and
gender, we observed warmer feelings toward the Black indi-
viduals in comparison to the White individuals, t(2,206) =
36.34, d= 0.77. The 10 women were rated more positively
on average than the 10 men, t(2,206) = 6.43, d=0.13.
Despite the opposing mean ratings, the correlation
between the self-reported social-group-race-preference and
the by-exemplar-race-preference was positive r(2,116) =
.38. The correlation between the self-reported social-
group-gender-preference and the by-exemplar-gender-
preference was weakly positive, r(2,108) = .11.
Focal Measures of SPF
Table 3 displays the focal SPF effects. When the category
labels were race related, the four focal associations were
analyzed with a race (2) ·Valence (2) ANOVA. Of most
interest was the interaction, F(1, 1003) = 237.25, g
p
2
=
.19. This interaction reflected that the association white-
good was stronger than the association white-bad,
F(1, 1003) = 303.54, g
p
2
= .23, whereas the opposite was
found with associations of Black people and valence,
black-bad was stronger than black-good, F(1, 1003) =
34.97, g
p
2
= .03. A main effect of valence,
F(1, 1003) = 66.26, g
p
2
= .06, indicated faster responses
2
See supplement web material http://www.briannosek.com/spf/ for analyses of procedural factors that are not reported here.
Table 2. Study 2: Means of the self-report measures
Black White
Men Women All Men Women All
Group rating 0.78** 1.27** 1.19** 1.14** 1.69** 1.57**
Mean of individual ratings 1.06** .77** .92** .04* .47** .22**
Overall Preference
Men Women Women-Men White-Black
Group rating 1.27** 1.98** .72** .38**
Mean of individual ratings .51** .62** .11** .70**
Note. The scale ranged from 4 to 4 (rescaled from 1–9). The mean of individual ratings were the average rating of famous people that
belong to each group. The statistical test is whether the score is different than zero (nonsignificant difference from zero may still be
significantly different from other scores).
*p< .05; **p< .0001.
334 Bar-Anan et al.: The Sorting Paired Features Task
Experimental Psychology 2009; Vol. 56(5):329–343 2009 Hogrefe & Huber Publishers

when the evaluative term was ‘‘good’’ rather than ‘‘bad’’,
and a main effect of race suggested faster responses when
the attitude object was Black people, F(1, 1003) = 82.0,
g
p
2
= .08. Both effects probably resulted from the fact that
response to white-bad was slower than in all the other
conditions.
When the focal context was gender, the ANOVA gender
(2) ·Valence (2) yielded a significant interaction,
F(1, 974) = 139.13, g
p
2
= .13. This interaction reflected
that, as predicted, the association women-good was stronger
than women-bad, F(1, 974) = 101.15, g
p
2
= .09, whereas
men-bad was stronger than men-good, F(1, 974) = 57.54,
g
p
2
= .06. A main effect of valence indicated that people
were slightly faster to categorize pairs that included good
words, F(1, 974) = 4.12, p= .04, g
p
2
<.01.Therewasno
main effect of gender, F(1, 974) = 1.37, p= .24.
Known-Groups Validation
According to social identity theory and evidence, people
tend to favor their own groups compared to others, even
implicitly (Nosek, Smyth et al., 2007; Payne, Cheng,
Govorun, & Stewart, 2005; Tajfel & Turner, 1986).
Seventy-three Black and 726 White participants completed
the race SPF. As expected, when the race of the participants
was added to the above ANOVA, the three-way interaction
of Participant’s race ·Target’s race ·Valence was signifi-
cant, F(1, 797) = 53.34, g
p
2
= .06. As illustrated by Figure
2A, the interaction reflected the predicted pattern of results.
White participants categorized black-bad faster than black-
good, F(1, 725) = 38.09, g
p
2
= .05, whereas Black partici-
pants were faster to categorize black-good than black-bad,
F(1, 72) = 9.46, g
p
2
= .11. White participants categorized
white-good faster than white-bad, F(1, 725) = 280.13,
g
p
2
= .28, whereas Black participants showed no difference
between the two, F(1, 72) < 1, g
p
2
<.01.
Similar patterns were observed with the gender SPF
between men (n = 368) and women (n= 596). Adding par-
ticipant’s gender to the Target’s gender ·Valence ANOVA
yielded the expected three-way interaction, F(1, 962) =
172.71, g
p
2
= .15. As illustrated in Figure 2B, the interaction
reflected the results consistent with social identity theory for
women, and men showed no gender preferences. Females
were faster to respond to women-good than to women-
bad, F(1, 595) = 219.75, g
p
2
= .27, and faster to respond
tomen-badthanmen-good,F(1, 595) = 134.84, g
p
2
=.18.
Males did not show differences between women-good and
women-bad, F(1, 367) = 2.73, p= .10, g
p
2
< .01, and
between men-bad and men-good, F(1, 367) = 2.97,
p=.09,g
p
2
< .01. A similar pattern of strong gender pref-
erences among women and none among men occurs with
the IAT as well (Rudman & Goodwin, 2004). Males were
faster to respond to pairs that included men than women,
F(1, 367) = 33.64, g
p
2
= .08. Females did not show this
difference.
Relations Among SPF and Self-Reported Attitudes
To test the relations between the SPF and self-reported atti-
tudes, four SPF scores were computed for each participant
following methods described by Greenwald, Nosek, and
Banaji (2003). Each score represents performance in one cat-
egory pair condition in comparison to the overall perfor-
mance. Each score reflects the difference between the
participant’s mean latency in that condition and the partici-
pant’s overall mean latency in all trials, divided by the par-
ticipant’s overall standard deviation for all trials
(D
association
=(M
overall
M
association
)/SD
overall
). This individ-
ualized effect size behaves like a dominance measure assess-
ing the degree of overlap in the response distributions of one
association compared to the whole sample (Sriram, Nosek,
& Greenwald, 2008). In other applications, the Dis less vul-
nerable to extraneous influences that affect response latency
data such as cognitive fluency and task switching ability
(Cai, Sriram, Greenwald, & McFarland, 2004; Greenwald
et al., 2003; Klauer & Mierke, 2005; Mierke & Klauer,
2003). All measurement requires some acknowledgement
of ‘‘compared to what’’. Each SPF score is a comparison
of one association to all associations in the task. As such,
they are interdependent – knowing three of the scores pro-
vides sufficient information to calculate the fourth.
As can be observed in Table 4, among the four race asso-
ciations, black-good and white-good SPF associations were
the most related to self-report. Black-good correlated nega-
tively (rs=.16, .24) with the two self-reported race
preference measures, and white-good correlated positively
with them (rs = .17, .18). Black-bad and white-bad associa-
tions showed little to no relations with self-report. These re-
sults suggest that the SPF is related to self-reported attitudes.
The SPF’s unique association-specific information suggests
that the relative strengths of associations with good are bet-
ter predictors of self-report than the relative strengths of
associations with bad.
Table 3. Study 2: Mean latency for each Group ·Valence condition (standard deviation in parentheses)
Bar-Anan et al.: The Sorting Paired Features Task 335
2009 Hogrefe & Huber Publishers Experimental Psychology 2009; Vol. 56(5):329–343

One way to compute an SPF measure of race preference is
an additive combination of scores (white-good + black-bad)
– (white-bad + black-good). This score correlated with the
self-report racial preference measures at approximately the
same magnitude as the individual associations of race with
good (rs = .18, .24). In comparison, the IAT has a somewhat
stronger correlation with self-reported racial preferences on
average (e.g., r(586,139) = .31, Nosek, Smyth et al., 2007).
Figure 2. Study 2: SPF latency means by participants’ social-group (in the relevant focal conditions). Ns: Women: 596,
men: 368, white: 726, black: 73.
Table 4. Study 2: Correlations between SPF race association scores and self-reported race attitudes (correlations of
nonfocal race association scores in parentheses)
Self-report Self-report,
by ratings of
individuals
Self-report
white
Self-report
black
Self-report
white, by
ratings
of individuals
Self-report
black, by
ratings of
individuals
SPF black + good .16**
(.09**)
.24**
(.14**)
.06*
(.03)
.11**
(.08*)
.08**
(.05)
.16*
(.08*)
SPF black + bad .05
(.01)
.11**
(.05)
.01
(.02)
.06*
(.04)
.03
(.04)
.14*
(.00)
SPF white + good .17**
(.13**)
.18**
(.09**)
.11**
(.07*)
.07*
(.07*)
.10**
(.04)
.07*
(.04)
SPF white + bad .05
(.05)
.03
(.00)
.03
(.02)
.02
(.04)
.01
(.04)
.04
(.03)
Note. *p< .05; ** p< .01.
336 Bar-Anan et al.: The Sorting Paired Features Task
Experimental Psychology 2009; Vol. 56(5):329–343 2009 Hogrefe & Huber Publishers

As with the IAT (Nosek, 2005), the gender-related SPF
associations were weakly or not at all related to self-reported
gender preferences (Table 5). This probably indicates the
complexity and context sensitivity of attitudes toward men
and women (e.g., differences between sexual attraction
and friendship interests). Automatic and self-report mea-
sures may not capture that context sensitivity in the same
way.
Comparison of Focal and Nonfocal Association
Measurement
The same 20 images served as attitude stimuli regardless of
whether the focal categories were women/men or Black peo-
ple/White people. We compared the same valence-race and
valence-gender associations in the two different focal
dimension conditions (race and gender). Focality had an
effect on the association estimates (see Table 3). Adding
the between-subject focal-concept manipulation to the ear-
lier analyses, the three-factor ANOVA race (2) ·Va l e n c e
(2) ·Focal-concept (2), yielded a significant three-way
interaction, F(1, 1977) = 110.14, g
p
2
= .05. This interaction
indicates that when gender was the focal concept, the
Race ·Valence interaction decreased, F(1, 974) = 4.77,
p=.03,g
p
2
< .01, in comparison to the previously reported
large effect (g
p
2
= .19) when race was the focal concept.
The interaction reflected the fact that the nonfocal white-
bad association was slightly weaker than white-good,
F(1, 1003) = 9.14, g
p
2
<.01.
Similar findings occurred with nonfocal gender associa-
tions. The three-way interaction between race, valence, and
focal-concept was significant, F(1, 1977) = 54.16, g
p
2
=.03,
reflecting smaller differences between gender associations
when race was the focal concept. The Gender ·Va l enc e i nte r -
action was significant even when race was the focal category,
F(1, 1003) = 12.80, g
p
2
= .01. The nonfocal women-good
association was stronger than women-bad, and
F(1, 1003) = 70.11, g
p
2
= .06, and men-bad was stronger
than men-good, F(1, 1003) = 18.02, g
p
2
=.02.
The SPF measures of nonfocal attitudes were evident
with the interaction between the participants’ own social
group and the nonfocal conditions. A small three-way
Own race ·Target r a ce ·Valence interaction was found
even when the focal category was gender,
F(1, 754) = 5.73, p= .02, g
p
2
< .01. The interaction
reflected two significant differences: White participants held
stronger white-good than white-bad associations,
F(1, 689) = 12.04, g
p
2
= .02, whereas Black participants
held stronger white-bad than white-good associations,
F(1, 65) = 5.65, p= .02, g
p
2
=.08. When the focal cate-
gory was race, there was an Own-gender ·Target-
gender ·Valence interaction, F(1, 988) = 4.81, p=.03,
g
p
2
< .01. Men did not show ‘‘nonfocal’’ preference for
men or women, whereas women had stronger women-good
and men-bad than women-bad and men-good associations,
Fs(1, 988) = 59.90, 19.24, g
p
2
s = .06, .02, respectively.
The correlations in the parentheses in Tables 4 and 5 show
that the nonfocal SPF attitudes were weakly related to self-
reported attitudes. Overall, the effect of the nonfocal concept
attitudes on performance was smaller, but still reliable. We
conclude that, using the SPF, nonfocal categories can influ-
ence automatic evaluation even when another category dom-
inates attention.
Political Attitudes
More evidence that the SPF nonfocal scores are meaningful
comes from SPF preference scores between even narrower
subsets of stimuli. Three of the individual stimuli were affil-
iated with the US Republican Party, and three were affiliated
with the US Democratic Party. An SPF political preference
score calculated as ((Democrats-good + Republicans-bad) –
(Democrats-bad + Republicans-good)) showed a correlation
of r(1,972) = .18 with a self-rating of 3 (conservative) to 3
(liberal).
Single Stimuli
The most extreme test of the SPF’s ability to detect attitudes
other than the focal attributes is to examine each stimulus indi-
vidually. Participants rated explicitly each of the 20 people
Table 5. Study 2: Correlations between SPF gender association scores and self-reported gender attitudes (correlations of
nonfocal gender association scores in parentheses)
Self-report Self-report,
by ratings
of individuals
Self-report
women
Self-report
men
Self-report
women,
by ratings of
individuals
Self-report
men, by ratings
of individuals
SPF women + good .00
(.04)
.
11**
(.11**)
.04
(.03)
.04
(.02)
.08*
(.09**)
.00
(.00)
SPF women + bad .01
(.06*)
.01
(.02)
.02
(.05)
.05
(.02)
.05
(.02)
.04
(.05)
SPF men + good .00
(.00)
.16**
(.12**)
.01
(.03)
.00
(.04)
.09**
(.03)
.04
(.07*)
SPF men + bad .00
(.03)
.07*
(.00)
.01
(.00)
.01
(.04)
.06
(.02)
.00
(.02)
Note. *p< .05; ** p< .01.
Bar-Anan et al.: The Sorting Paired Features Task 337
2009 Hogrefe & Huber Publishers Experimental Psychology 2009; Vol. 56(5):329–343

who appeared in the SPF. For each stimulus person, we
computed the correlation between its SPF evaluation (i.e.,
the difference between the stimulus-good and stimulus-bad
scores), and the self-reported evaluations of the five stimulus
individuals of the same gender and race. For instance, we
compared the correlations between the SPF evaluation of
Oprah Winfrey and self-reported evaluations of Oprah
Winfrey, Beyonce Knowles, Whoopi Goldberg, Whitney
Houston, and Condoleezza Rice. For each stimulus, the stron-
gest of the five correlations should be the one that involved the
same person. On random, this should happen one in five times
– in four out of the 20 targets used in our study. However, 18
times out of 20, the SPF evaluation score of the stimulus cor-
related with the self-reported evaluation of the same stimulus
better than with the self-reported evaluation of the other four
stimuli in the same race and gender group. The chances for
that are 311 20
18

4
5
21
5
18

. Notably, the correlations
between the SPF evaluation score of each stimulus and its
self-reported evaluation were very low, an average of .06,
probably because each stimulus appeared only six times in
total (three with each valence) producing a very unreliable
estimate.
SPF Reliability
We averaged the intercorrelation of each association mea-
sure across the three blocks as an estimate of internal consis-
tency. Dividing the task into thirds underestimates the
reliability of the entire measure, so we used the Spear-
man-Brown correction to compensate (termed adjusted r;
Nunnally, 1978). The adjusted rs for the SPF association
score for each pair showed a slight but consistent advantage
for associations with good: black-good .51, black-bad .41,
white-good .44, white-bad .37, women-good .48, women-
bad .33, men-good .40, men-bad .35, and .31 and .29 for
the combined race and gender preference scores,
respectively.
Overall, these internal consistencies were lower than
some automatic association measures such as the IAT
(Nosek, Greenwald et al., 2007), single-category IAT (SC-
IAT; Karpinski & Steinman, 2006), the Brief IAT (Sriram
& Greenwald, in press), and the AMP (Payne et al.,
2005), and better than others such as EP (Bosson et al.,
2000; Cunningham et al., 2001), the GNAT (Nosek & Bana-
ji, 2001), and the EAST (De Houwer, 2003). Importantly,
however, the internal consistencies may not be directly com-
parable because of the use of different stimulus items and
number of trials across tasks. In this SPF design, the task in-
volved just 120 trials in total, 30 per association.
Discussion
Study 2 demonstrates that a variety of assessments are pos-
sible within a single SPF. The SPF was affected by the par-
ticipant’s own social group, and was related to other attitude
measures. Some of the association measures correlated with
relevant self-reported measures, demonstrating the validity
of the association measures and unique contribution of the
individual associations. For instance, associations with posi-
tive valence were more reliable and more related to self-
report than associations with negative valence. In addition,
like in Study 1, we found that the weakest association was
between White people and bad, suggesting that this associ-
ation is the most influential contributor to pro-white bias.
Nonfocal SPF assessments, including measures related
to single stimuli, also showed some validity, correlating with
matched explicit attitudes. They were also affected by the
participant’s own social group. Nonfocal categories affected
performance to a lesser degree than focal categories. This
suggests that attention and accessibility play important roles
in automatic attitude measurement, and possibly also in the
manifestation of automatic attitudes in behavior. Because the
SPF provides control of what categories are focal, it may
facilitate further research on the importance of attention
and accessibility of concepts for automatic attitudes. For
instance, people who are affected by the nonfocal category
in the SPF may have a tendency to categorize people
according to that category, and be more likely to use this cat-
egory in social judgment and evaluation (Higgins, 1996).
Study 3
We aimed to further validate the SPF as a measure of atti-
tudes, by investigating its relation with the IAT. Study 3 also
examined political attitudes to extend the application to a
new topic, one that elicits reliable correlations between
self-report and implicit measures (Nosek, 2005).
Method
Participants
Sixty-four students (38 women), enrolled in an introductory
psychology course participated for course credit. We
excluded one participant who responded randomly on the
SPF.
Materials, Design, and Procedure
Materials
The good and bad words were the same as in Study 1 (48
words). The two attitude categories were Democrats (stim-
uli: Democrats, Liberal, Left-wing, Al Gore, and Clinton)
and Republicans (Republicans, Conservative, Right-wing,
Cheney, and George Bush).
SPF
The SPF procedure was similar to the Study 1 version,
with the following changes. The first block consisted of
338 Bar-Anan et al.: The Sorting Paired Features Task
Experimental Psychology 2009; Vol. 56(5):329–343 2009 Hogrefe & Huber Publishers

10 practice trials in which the attitude objects were letters
and numbers. The next three blocks, with the political atti-
tude objects, consisted of 72 trials each. Four locations of
the four category pairs were manipulated between-subjects.
IAT
The IAT followed the ‘‘standard’’ IAT design (Nosek,
Greenwald et al., 2007), but with a single combined re-
sponse block (with 54 trials) for each pairing condition.
The order of the test blocks was counterbalanced.
Self-Report
Participants rated the categories on scales of favorability
(ranged from unfavorable = 1 to favorable = 9) and valence
(ranged from bad = 1 to good = 9) scales.
The order of the three measures was counterbalanced,
and the procedure was similar to Study 1. For all measures,
positive numbers indicated preference for Democrats over
Republicans.
Results
SPF
The mean reaction time was 1,573 ms (SD = 253) and the
mean error rate was 0.07 (SD = 0.06). A 2 (politics) ·2
(valence) found only a main effect of valence,
F(1, 62) = 5.07, p=.03,g
p
2
= .06, and no other effects,
ps > .23. The only significant difference was between the
slowest condition, Republicans-bad (M= 1,601; SD =
268), and the fastest condition, Republicans-good
(M= 1,560; SD = 306), F(1, 62) = 3.91, p= .05, g
p
2
=.06
(for Democrats-bad and Democrats-good: Ms = 1,566,
1567; SDs = 256, 278, respectively). We computed a D
score for each of the four conditions. The adjusted rinternal
consistency of the associations were Republicans-good =
.70, Democrats-good = .62, Democrats-bad = .49, and
Republicans-bad = .30 replicating the higher reliability of
‘‘good’’ associations from Study 2 (the adjusted rfor the
combined political preference SPF score was .60).
IAT
The IAT score was computed using the algorithm recom-
mended by Greenwald et al. (2003). Its mean did not differ
significantly from zero, t< 1. Its reliability was adjusted
r=.79.
Self-Report
We averaged the valence and favorability ratings for each
party to compute the attitude toward each. The difference
between the two was the self-reported preference, and it
did not differ significantly from zero, t<1.
Relations Among the SPF, IAT, and Self-Report
The SPF association Republicans-good was the most related
one to the other measures, showing a correlation of
r(63) = .36, p< .001 with the IAT and r(63) = .41,
p< .001 with self-report (Table 6). The association Demo-
crats-good showed similar correlations in the opposite direc-
tion, rs(63) = .34, .35, respectively, ps<.05. The
associations Republicans-bad and Democrats-bad showed
no correlation with any measure. For all four SPF scores,
the correlation with the IAT was not better than the correla-
tion with the self-report, Williams’ ts < 1 (Williams, 1959).
The IAT strongly correlated with the self-report, r(63) = .69,
p< .0001.
Discussion
Study 3 further validates the SPF. The SPF showed relations
with both the IAT and self-reported attitudes. The correlation
pattern does not distinguish clearly between automatic and
self-report measures. This could be counter to the claim that
the IAT and the SPF measure the same construct that is
related but distinct from self-reported attitudes. However,
besides selecting a topic that is known to elicit strong corre-
lations between IAT and self-report (Nosek, 2005), the self-
reported measures are probably more reliable than both the
SPF and the IAT (Nosek & Smyth, 2007). Therefore, a mea-
surement of the relation between the SPF and the IAT will
be underestimated to a greater degree by measurement error
compared to their relations with self-report (Nosek,
Greenwald et al., 2007).
Table 6. Study 3: Correlations between SPF scores and self-reported attitudes
Democrats-Republicans
(Study 3)
Self-reported
Democrats-Republicans
IAT Self-report
Democrats
Self-report
Republicans
SPF Democrats + good .34* .35** .29* .35**
SPF Democrats + bad .08 .08 .11 .04
SPF Republicans + good .41** .36** .40** .39**
SPF Republicans + bad .02 .12 .02 .00
Note.*p< .05; ** p< .01.
Bar-Anan et al.: The Sorting Paired Features Task 339
2009 Hogrefe & Huber Publishers Experimental Psychology 2009; Vol. 56(5):329–343

Further Evidence of Relations Between SPF and Other
Attitude Measures From Other Studies
A recent investigation (Ranganath, Smith, & Nosek, 2008)
used several implicit measures, including the SPF, to test
whether self-reported ‘‘gut feelings’’ are more similar to
self-report or implicit reactions. Using a confirmatory factor
analysis, the SPF fit well on a latent factor with the IAT,
GNAT, and self-reported gut feelings in contrast to self-
reported actual feelings. Future investigations with larger
samples can take advantage of structural equation modeling
techniques to distinguish the roles of measurement error and
construct relations to better clarify the relations among the
SPF, other implicit measures, and self-report (Cunningham
et al., 2001; Nosek & Smyth, 2007).
Other studies have used the SPF and other measures,
providing further support for convergent validity (more
details at http://www.briannosek.com/spf/). One study
examined the SPF, EP, and self-reported political attitudes
(Bar-Anan, Nosek, & Vianello, 2008). We computed com-
bined preference scores, and also separate association scores
for the SPF and EP. All SPF scores showed superior internal
consistency (mean adjusted r=.42) and relationship with
self-report (mean correlation = .32) in comparison to paral-
lel EP scores (mean adjusted r= .17; mean correla-
tion = .12). The SPF/EP correlation was r(353) = .38,
p< .01. Associations with good (i.e., Democrats-good and
Republicans-good) showed superior internal consistency
and relationship with other measures than associations with
bad (i.e., Democrats-bad and Republican-bad), but only
among the SPF scores.
A second study measured attitudes toward cats and dogs
using self-report, IAT, and SPF (Smith & Nosek, 2007a) and
found that the SPF dogs-good and cats-good scores corre-
lated with the IAT and with the self-reported feelings,
rs(63) = .38, .42, ps < .0001. The dogs-bad and cats-
bad showed weaker and unreliable correlations. The correla-
tion of the IAT with self-report, r(63) = .54, p<.01,was
not significantly better than the SPF’s dogs-good and cats-
good correlations with self-report.
A third study measured attitudes toward George Bush
and John Kerry, the US presidential candidates in the
2004 elections (Smith & Nosek, 2007b). All four SPF scores
were related to the IAT and to self-reported attitudes, but
again, the associations with good (average magnitude of
r= .44) performed better than associations with bad (aver-
age magnitude of r= .25). The correlation between IAT
and self-report, r(80) = .68, p< .0001, was not significantly
stronger than the SPF’s Bush + good and Kerry + good cor-
relations with self-report.
A fourth study provides insight into test-retest reliabili-
ties of the SPF across time. Participants completed SPF
and IAT about sweet and salty food preferences three times
-initially, one week later, and a month after the initial date
(Vianello, Bar-Anan, & Nosek, 2008). The mean internal
consistency across all occasions was r=.71. The mean
test-retest correlations were .60 between the first and the sec-
ond session, and .51 between the first and the third session.
These were not significantly different from the IAT test-ret-
est correlations in the same study (both = .64).
These additional studies provide more convergent valid-
ity evidence that the SPF relates to automatic and self-report
measures across multiple topics. The SPF’s relation to self-
report is similar in magnitude to the IAT, despite having
somewhat lower internal consistency on average (.49 across
all studies reported above). Also, across studies, SPF associ-
ations with positive valence relate to other attitude measures
and show stronger internal consistency than do associations
with negative valence.
General Discussion
The present research introduces the SPF as a measure of
association strengths. In Studies 1 and 2, the SPF effects
suggested that, overall, white-good and black-bad associa-
tions are stronger than white-bad and black-good, respec-
tively. In Study 2, this effect held for Whites and was in
the opposite direction for Blacks. Similarly, as expected,
in Study 2, women more than men had stronger female-
good and male-bad associations than female-bad and
male-good, respectively. SPF scores correlated positively
with self-reported attitudes (Studies 2–3), and with the
IAT (Study 3). The strength of the SPF relationship with
self-reported attitudes across topics mirrored that observed
with the IAT (Nosek, 2005). Gender attitudes showed al-
most no relations, race showed weak relations, and politics
showed strong relations. These results establish initial
known-groups and convergent validity for the SPF.
We also found that the SPF could measure associations
for categories that were the focal, highly accessible, targets
of categorization, and the nonfocal, less accessible, inciden-
tal features of the stimuli. In Studies 1 and 2, both the focal,
accessible concepts and the nonfocal, less-accessible con-
cepts influenced task performance, the former more so than
the latter. This supports the presumption that accessibility
plays an important role in automatic attitude activation
(Olson & Fazio, 2003).
The SPF’s ability to measure nonfocal associations pro-
vides an important benefit to the family of association mea-
sures. In Study 2, when the category labels demanded
categorization by gender, Black participants still showed
stronger automatic positivity toward Blacks over Whites
than the White participants did. The nonfocal associations
were also weakly related to self-reported preferences. We
even found a relation between the SPF score of each stimu-
lus that was used in Study 2, and the self-reported evaluation
of this stimulus. These findings suggest that the SPF pro-
vides more information than other reaction time measures
of multi-faceted attitudinal influences on a single stimulus,
and that these varied influences can occur even when atten-
tion is directed toward another category.
The SPF provides estimates of specific associations
between valence and attitude objects compared to the overall
task performance. Because the four associations comprise
the overall task, the four estimates are algebraically related.
However, these studies show that the separate associations
have distinct validity, unlike measures such as the IAT
(Nosek et al., 2005). For example, results from Studies 1
340 Bar-Anan et al.: The Sorting Paired Features Task
Experimental Psychology 2009; Vol. 56(5):329–343 2009 Hogrefe & Huber Publishers

and 2 suggest that automatic pro-white biases, especially
among White participants, are a function of both positive
associations for Whites and negative associations for Blacks,
the former more so than the latter. This suggests that both in-
group favoritism and out-group derogation may be influenc-
ing the aggregate implicit preference revealed by measures
like the IAT.
Further, the assessment of all four associations revealed
an unexpected finding that could have significant theoretical
implications. In all the studies, the focal and nonfocal asso-
ciations of the attitude objects with good were more related
to criterion variables than the association of attitude objects
with bad. In most studies, the difference was substantial.
Further analysis revealed that in all the studies, the associa-
tions with good tended to show better internal consistency
(average of adjusted racross studies = .50) than associa-
tions with bad (average of adjusted racross studies = .37).
This difference in internal consistencies is unlikely to be
caused by a quirk in the task procedure. Reliability, in this
case, is the substantive outcome that was influenced by mea-
surement of good versus bad associations. This demon-
strates a virtue of the SPF’s ability to estimate internal
consistencies for each association.
Sriram and Greenwald (in press) reported similar find-
ings with a variant task of the IAT, the Brief-IAT. In that
task, participants indicate whether the target stimuli (pre-
sented one at a time) pertains to one of the two categories
(e.g., Pepsi or good) or not. The preference score between
two attitude objects is computed by comparing two blocks
that share the same evaluative term (e.g., good) but differ
in the focal attitude objects (e.g., Pepsi vs. Coke).
Sriram and Greenwald found that the preference score
correlated with self-report only when the evaluative term
was positive (e.g., Pepsi or good compared with Coke or
good) and not when it was negative (e.g., Pepsi or bad com-
pared with Coke or bad). Sriram and Greenwald speculated
that associations with positive valence might be more impor-
tant for attitudes than associations with negative valence.
Another explanation could be the density hypothesis
(Unkelbach, Fiedler, Bayer, Stegmuller, & Danner, 2008):
Negative concepts are less related to each other than positive
concepts, and therefore they yield less consistent effects.
Building on that hypothesis, Unkelbach and colleagues pre-
dicted and found that EP sequences of negative prime fol-
lowed by negative target causes less latency facilitation
than positive-positive sequences. This effect suggests less
consistent association effects among different negative stim-
uli than among positive stimuli. A recent study in our lab
tested this account with the Brief-IAT, manipulating density
and valence of the attribute words independently (Sriram,
Bar-Anan, & Nosek, 2008). The results did not support
the density hypothesis account: We found good primacy
regardless of the density of the valence attributes. Addition-
ally, in a study with both EP and SPF (Bar-Anan et al.,
2008), only the SPF scores showed the good primacy effect.
Therefore, the density hypothesis and the good primacy
effect may be unrelated phenomena. Further investigations
will help to clarify these questions, and the SPF will be a
valuable tool in that research.
Comparison With Other Association
Measures
The summary in Table 7 compares some features of the SPF
with other measures. The SPF may be particularly useful for
separate estimations of four associations, and for manipulat-
ing the accessibility of categories during measurement. For
example, the SPF may detect whether some treatments af-
fect one association more than others, contributing to answer
questions like ‘‘does exposure to positive exemplars of a
group (Dasgupta & Greenwald, 2001) change positive asso-
ciations for that group and have no impact on associations
for comparison groups?’’ Also, as noted by an anonymous
reviewer, the SPF may be able to index inter- and intra-
individual differences in chronic accessibility of categories
by comparing the effects of categories when measured
focally or nonfocally.
The SPF presents two stimuli at the same time. This fea-
ture probably makes the SPF more sensitive to stimulus
meanings than IAT measures. At the same time, the use of
superordinate categories provide some constraints on stimu-
lus processing so that the SPF is probably less influenced by
stimulus meanings than EP and the AMP. Further, the effect
of each stimulus pair may be influenced by unique proper-
ties elicited by their conjoint meaning that are not activated
when they are considered separately. For example, a picture
of a toad and the word ‘‘stool’’ may activate the concept
mushroom when neither stimulus independently would do
so. Whether such features are influential and, by extension,
useful or a hazard for measurement depends on the
Table 7. Comparison of procedural features of the SPF and other association measures
Feature Yes No
1 Requires categorization
of all stimuli (attributes and attitude-objects)
to explicitly defined categories
SPF, IAT, GNAT, BIAT,
ST-IAT, SB-IAT
EP, AMP, EAST
2 All associations are measured in
the same response block
SPF, EP, AMP,
SB-IAT, EAST
IAT, GNAT, BIAT, ST-IAT
3 Requires the processing of two
stimuli conjointly
SPF IAT, GNAT, BIAT, ST-IAT,
SB-IAT, AMP, EP, EAST
4 Requires four categories SPF, IAT, SB-IAT, EAST GNAT, BIAT, ST-IAT, AMP, EP
Note. ST-IAT is the single-target IAT (Wigboldus, Holland, & van Knippenberg, 2004).
Bar-Anan et al.: The Sorting Paired Features Task 341
2009 Hogrefe & Huber Publishers Experimental Psychology 2009; Vol. 56(5):329–343

researcher’s goals. Part of the continuing investigation of the
SPF properties will be to identify such influences to clarify
its range of appropriate measurement applications.
Limitations and Next Steps
Application of the SPF is constrained by requiring an explicit
categorization of stimuli that belong to four categories. Also,
while the SPF’s internal consistency is about average for
implicit measures, there are others that exceed it such as
the IAT. Further, there may be unidentified extraneous influ-
ences that reduce the SPF’s internal validity. For example, if
stimuli from one category are systematically easier to process
and categorize than the other category, then faster responses
of joint pairs that include one versus the other could emerge
that are not reflective of association strengths between the
categories. Further innovations may identify and redress such
possible procedural and psychometric limitations.
Additionally, it would be useful to further evaluate the
construct validity of the SPF, and examine whether the four
association estimates have distinct predictive validity. This
research suggests that they will – especially comparing the
predictive validity of good versus bad associations. Also,
this research tested the SPF as an attitudes measure –
associations with positive and negative valence. Like other
association methods, the SPF may also measure self-
concept, stereotypes, and other nonattitudinal associations.
Summary
We presented the SPF as a new measure of associations. The
SPF is novel by requiring categorization of two stimuli simul-
taneously to produce a single response. As a result, the con-
nection between the stimuli is particularly relevant to task
performance. Unlike priming measures, a focal category is
explicitly mentioned and essential for proper task perfor-
mance. This provides control overthe focal, attended concepts
throughout the task, and ca n provide a means for manipulating
other associations nonfocally. Unlike the IAT, the SPF mea-
sures all the associations in a single response block eliminat-
ing history-related artifacts like practice or change in response
strategy across blocks. Because of its unique features and abil-
ities, theSPF may serve certain research needs that are noteas-
ily accommodated by other measures of automatic attitudes,
and help to promote the study of associations and their relation
to attitudes, thought, and action.
Acknowledgments
The authors thank Colin Tucker Smith and Fred Smyth for
their help in this research, and Jesse Graham, Kate
Ranganath, and N. Sriram for their helpful comments. The
authors also thank Jeff Hansen for his technical expertise.
This research was supported by a grant from the National
Institute of Mental Health (R01 MH68447) to Brian Nosek.
References
Bar-Anan, Y., Nosek, B. A., & Vianello, M. (2008).
A comparison between the SPF and evaluative priming
Unpublished manuscript (http://www.briannosek.com/spf/).
Bosson, J. K., Swann, W., & Pennebaker, J. W. (2000). Stalking
the perfect measure of implicit self-esteem: The blind men
and the elephant revisited? Journal of Personality and Social
Psychology, 79, 631–643.
Cai, H., Sriram, N., Greenwald, A. G., & McFarland, S. G.
(2004). The implicit association test’s Dmeasure can
minimize a cognitive skill confound: Comment on McFar-
land and Crouch (2002). Social Cognition, 22, 673–684.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and
discriminant validation by the multitrait-multimethod matrix.
Psychological Bulletin, 56, 81–105.
Craik, F. I. M., & Birtwistle, J. (1971). Proactive inhibition in
free recall. Journal of Experimental Psychology, 91,
120–123.
Cunningham, W. A., Preacher, K. J., & Banaji, M. R. (2001).
Implicit attitude measures: Consistency, stability, and con-
vergent validity. Psychological Science, 12, 163–170.
Dasgupta, N., & Greenwald, A. G. (2001). On the malleability of
automatic attitudes: Combating automatic prejudice with
images of admired and disliked individuals. Journal of
Personality and Social Psychology, 81, 800–814.
De Houwer, J. (2003). The extrinsic affective Simon task.
Experimental Psychology, 50, 77–85.
Fazio, R. H., Jackson, J. R., Dunton, B. C., & Williams, C. J.
(1995). Variability in automatic activation as an unobtrusive
measure of racial attitudes: A bona fide pipeline? Journal of
Personality and Social Psychology, 69, 1013–1027.
Greenwald, A. G., Banaji, M. R., Rudman, L. A., Farnham,
S. D., Nosek, B. A., & Mellott, D. S. (2002). A unified theory
of implicit attitudes, stereotypes, self-esteem, and self-
concept. Psychological Review, 109, 3–25.
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998).
Measuring individual differences in implicit cognition: The
implicit association test. Journal of Personality and Social
Psychology, 74, 1464–1480.
Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003).
Understanding and using the implicit association test: I. An
improved scoring algorithm. Journal of Personality and
Social Psychology, 85, 197–216.
Higgins, E. T. (1996). Knowledge activation: Accessibility,
applicability and salience. In E. T. Higgins & A. W.
Kruglanski, (Eds), Social psychology: Handbook of basic
principles (pp. 133–168). Guilford: New York.
Karpinski, A., & Steinman, R. B. (2006). The single category
implicit association test as a measure of implicit social
cognition. Journal of Personality and Social Psychology, 91,
16–32.
Klauer, K. C., & Mierke, J. (2005). Task-set inertia, attitude
accessibility, and compatibility-order effects: New evidence
for a task-set switching account of the implicit association
test effect. Personality and Social Psychology Bulletin, 31,
208–217.
Mierke, J., & Klauer, K. C. (2003). Method-specific variance in
the implicit association test. Journal of Personality and
Social Psychology, 85, 1180–1192.
Nosek, B. A. (2005). Moderators of the relationship between
implicit and explicit evaluation. Journal of Experimental
Psychology: General, 134, 565–584.
Nosek, B. A. (2007). Implicit-explicit relations. Current Direc-
tions in Psychological Science, 16, 65–69.
Nosek, B. A., & Banaji, M. R. (2001). The go/no-go association
task. Social Cognition, 19, 625–666.
342 Bar-Anan et al.: The Sorting Paired Features Task
Experimental Psychology 2009; Vol. 56(5):329–343 2009 Hogrefe & Huber Publishers

Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2005).
Understanding and using the implicit association test: II.
Method variables and construct validity. Personality and
Social Psychology Bulletin, 31, 166–180.
Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2007). The
implicit association test at age 7: A methodological and
conceptual review. In J. A. Bargh (Ed.), Social psychology
and the unconscious: The automaticity of higher mental
processes (pp. 265–292). Psychology Press: New York.
Nosek, B. A., & Smyth, F. L. (2007). A multitrait-multimethod
validation of the implicit association test: Implicit and
explicit attitudes are related but distinct constructs. Experi-
mental Psychology, 54, 14–29.
Nosek, B. A., Smyth, F. L., Hansen, J. J., Devos, T., Lindner, N. M.,
Ranganath, K. A., et al. (2007). Pervasiveness and correlates
of implicit attitudes and stereotypes. European Review of
Social Psychology, 18, 36–88.
Nosek, B. A., & Sriram, N. (2007). Faulty assumptions: A
comment on Blanton, Jaccard, Gonzales, and Christie (2006).
Journal of Experimental Social Psychology, 43, 393–398.
Olson, M. A., & Fazio, R. H. (2003). Relations between implicit
measures of prejudice: What are we measuring? Psycholog-
ical Science, 14, 36–39.
Payne, B.K., Cheng, C. M., Govorun, O., & Stewart, B. (2005).
An inkblot for attitudes: Affect misattribution as implicit
measurement. Journal of Personality and Social Psychology,
89, 277–293.
Ranganath, K. A., Smith, C. T., & Nosek, B. A. (2008).
Distinguishing automatic and controlled components of
attitudes from direct and indirect measurement. Journal of
Experimental Social Psychology, 44, 386–396.
Rudman, L. A., & Goodwin, S. A. (2004). Gender differences in
automatic in-group bias: Why do women like women more
than men like men? Journal of Personality and Social
Psychology, 87, 494–509.
Smith, C. T., & Nosek, B. A. (2007a). Affective focus increases
the concordance between implicit and explicit attitudes.
Unpublished manuscript.
Smith, C. T., & Nosek, B. A. (2007b). Unpublished manuscript.
Sriram, N., Bar-Anan, Y., & Nosek, B. A. (2008). Unpublished
manuscript.
Sriram, N., & Greenwald, A. G. (in press). The brief implicit
association test. Experimental Psychology.
Sriram, N., Nosek, B. A., & Greenwald, A. G. (2008). Scale
invariant contrasts of response latency distributions. Unpub-
lished manuscript.
Tajfel, H., & Turner, J. C. (1986). The social identity theory of
intergroup behavior. In S. Worchel & W. G. Austin (Eds.),
The psychology of intergroup relations (pp. 7–24). Chicago:
Nelson-Hall.
Unkelbach, C., Fiedler, K., Bayer, M., Stegmuller, M., &
Danner, D. (2008). Why positive information is processed
faster: The density hypothesis. Journal of Personality and
Social Psychology, 95, 36–49.
Vianello, M., Bar-Anan, Y., & Nosek, B. A. (2008). Reliability of
the SPF: Test-retest and internal consistency. Unpublished
manuscript (http://www.briannosek.com/spf/).
Wigboldus, D. H. J., Holland, R. W., & van Knippenberg, A.
(2004). Single target implicit associations. Unpublished
manuscript.
Williams, E. J. (1959). The comparison of regression variables.
Journal of the Royal Statistical Society, Series B, 21,
396–399.
Wyer, R. S. (2007). Principles of mental representation. In
A. Kruglanski & E. T. Higgins (Eds.), Social psychology:
Handbook of basic principles (pp. 285–307). New York:
Guilford.
Received June 12, 2008
Revision received September 10, 2008
Accepted September 16, 2008
Yoav Bar-Anan
Department of Psychology
University of Virginia
P.O. Box 400400
Charlottesville
VA 22904
E-mail baranan@virginia.edu
Bar-Anan et al.: The Sorting Paired Features Task 343
2009 Hogrefe & Huber Publishers Experimental Psychology 2009; Vol. 56(5):329–343