Incidental Learning During Information Retrieval: A Hypertext Experiment

Abstract

A study was conducted in a hypertext environment in order to examine the effects of the type of fact-retrieval search (browsing vs. index-using) on incidental learning. It was hypothesized that browsing would lead to more incidental learning; from a cognitive viewpoint, it would appear that browsers would have more opportunity to make connections between the various articles, which would result in a more complete understanding of the material. No significant results were found at the level of overall performance; however, by examining individual questions and the log files of various subjects, insight was gained into the search process. These trends were discussed in terms of implications on the use of hypertext for education. Many suggestions for designers of hypertext databases were given, along with suggestions for further research. 1 This material is based upon work supported under a National Science Foundation Graduate Fellowship. 1 INTRODUCTION There is a growing awareness th...

Full text

Incidental Learning During
Information Retrieval:
A Hypertext Experiment
Tricia Jones1
Department of Computer Science, University of Maryland
College Park, MD, USA 20742
Abstract
A study was conducted in a hypertext environment in order to examine the effects
of the type of fact-retrieval search (browsing vs. index-using) on incidental
learning. It was hypothesized that browsing would lead to more incidental learning;
from a cognitive viewpoint, it would appear that browsers would have more
opportunity to make connections between the various articles, which would result
in a more complete understanding of the material. No significant results were found
at the level of overall performance; however, by examining individual questions and
the log files of various subjects, insight was gained into the search process. These
trends were discussed in terms of implications on the use of hypertext for
education. Many suggestions for designers of hypertext databases were given,
along with suggestions for further research.
1This material is based upon work supported under a National Science Foundation Graduate Fellowship.

1 INTRODUCTION
There is a growing awareness that current educational software often is not based on strong
educational or cognitive theories, and does not reflect the cutting edge of software development.
To create better educational software, some educators and software developers are beginning to use
hypertext environments (see Yankelovich, Meyrowitz, and van Dam (1985), pp. 22-28, for an
overview of the systems in development and in use at Brown University). While these and other
hypertext applications are blossoming, research regarding their efficacy has not kept up the pace.
In addition, research examining how users search in such an environment has been minimal. The
current study explored the use of two navigational aids, embedded menus and indexes, within
Hyperties, a hypertext system for personal computers. In particular, the interaction between the
navigational aid and incidental learning was examined. Conclusions will be drawn regarding the
appropriateness of the hypertext environment for educational software, and implications will be
made for designers of general purpose hypertext systems.
1.1 Hypertext
Hypertext can be compared to `electronic books', yet it is inherently much more than just an
electronic analogy to our familiar concept of a book. One essential characteristic of hypertext is the
connectivity of information. One author describes it as “linking together discrete blocks to form
webs of information, following different paths through the information webs, and attaching
annotations to any block of information” (Yankelovich, Meyrowitz, and van Dam, 1985, p. 19).
An extension of hypertext is hypermedia, which can include graphics, sound, videodiscs, or
animation – all interconnected.
Why should a teacher choose a software system built upon a hypertext framework? As
Marchionini and Shneiderman (1988) point out, due to the relative newness of hypertext, “we have
sparse evidence for it's effectiveness, let alone proven principles to guide design” (p. 71). Why
should the use of hypertext be promoted for learning?
One of the major assets of such a system is the ability to make explicit links between concepts.
Such explicit links may lead to an increase in knowledge, as the users gain an understanding of the
information's underlying structure. This notion of connectivity is one of the main selling points of
hypertext, since it corresponds to cognitive models proposed by research psychologists.
Hypertext systems, with their emphasis on related ideas and webs of information, should lead to a
better understanding of concepts and how they are related. Such knowledge is beneficial to one
who seeks to understand a particular domain. Ausubel has given us the notion of “anchoring ideas

to an ideational structure.” Cognitive psychologists promote the notion of spreading activation. If
knowledge is represented cognitively as an associative network or hierarchy of information, then
systems which are organized along similar lines would seem to be desirable. Browsing, with its
decreased cognitive load, might allow more short-term and episodic memory resources to be freed
for the processing of incidental facts. By emphasizing the existing interrelationships among ideas,
it would be easier to tie together related webs of information.
One of the aims of the present research is to determine if browsing strategies have an effect on a
user's overall grasp of the knowledge structure in a database. If incidental learning can be
enhanced through the use of hypertext, and through the use of navigational aids such as embedded
menus, there should be many fields in which to apply this technology. Educational software is one
arena. Kreitzberg and Shneiderman (1988) give direction to this idea by suggesting types of
instruction for which Hyperties will be appropriate, including dynamic glossaries, familiarization,
and diagnostic problem-solving. Perhaps students could read a lesson in the hypertext
environment, followed by a comprehension test, or a vocabulary test, or some other appropriate
measure. When an incorrect response is given, the system could return the student to a point in the
text where he or she could reexamine the relevant information and hopefully gain a broader
perspective on the item in question. If education is, as George Landow puts it, the act of seeing
connections and following links (Yankelovich, et al, p. 16), and if hypertext systems encourage the
understanding of the interconnections between concepts, then these systems might have a
substantial role in the development of educational software. If they are beneficial in “observing
existing connections and making new connections” (Yankelovich, et al, p. 16) between various
disciplines, then they can contribute to an interdisciplinary understanding of the world.
1.2 Information Seeking in Text
Methods of seeking information range from highly structured, analytical searching to random
forms of browsing. Search is a well-defined discipline in computer science; however, searching
for information within text – topic falling within the realm of information science – is not nearly as
formalized; therefore, much remains to be learned. For example, it is an open question how
previous knowledge of the task domain influences search strategies. Similarly, the effect of the
user interface is relatively unknown. The present study examines two specific search methods,
index-using and browsing, within the hypertext environment.
Indexes or tables of contents are familiar metaphors from the realm of ‘hard-copy books’. Most
literate people feel quite comfortable with the concept. Browsing, on the other hand, does not
seem to be as well-defined or concrete. Browsing can be defined as “an exploratory, information-
seeking strategy that depends on serendipity. It is especially appropriate for ill-defined problems

and for exploring new task domains.” (Marchionini and Shneiderman, 1988, p. 71). Marchionini
defines browsing as “a highly interactive process with multiple decision points which depend on
feedback to help determine what to do next.” He goes on to explain various browsing techniques,
ranging from random and informal to systematic and formal. He gives three broad reasons why
people resort to browsing (Marchionini, 1988a, p.3):
First, they browse because they cannot or have not defined their search objective.
Second, people browse because it takes less cognitive load to browse than it does
to plan and conduct an analytical, optimized search... Third, people browse because
the information system supports and encourages browsing.
The browsing examined in this study falls into the third category. Many environments support and
encourage this form of search, such as the stacks in a library. Hypertext systems in general, and
the use of embedded menus in Hyperties in particular, also tend to support and encourage
browsing.
An electronic-text system with powerful full-text search capabilities is Grolier’s Electronic
Encyclopedia on CD-ROM. Experiments conducted with the system, involving elementary school
students and high school students, suggest that novices benefit from having hypertext features
which allow browsing. Elementary school students, with less proficiency in the search techniques
and insufficient mental models of the system, tended to use the low cognitive load browsing
strategies. High school students using an analytical approach were no more effective in finding
information than those using a scan and select (browsing) strategy (see Marchionini (1988b) and
Marchionini and Shneiderman (1988)).
A pilot study involving a simple fact-retrieval task was conducted using the Hyperties system, with
the primary aim of identifying research questions and methods (Wang, Liebscher, and
Marchionini, 1987). Two experiments were performed; one examined display formats (print and
electronic), the other examined two search strategies (index-using and browsing). Subjects in the
print condition were more efficient; however, no significant difference was found in mean number
of questions correctly answered. In the second experiment, subjects in the Index group tended to
view fewer articles and screens during the search, but no significant difference was found in terms
of success. The index users rated the system higher in terms of ease, speed, and lower in level of
frustration.
The above experiment involved a highly specialized student group, consisting of graduate level
Library Science students, all with considerable experience in reading printed text, as well as in

using indexes and similar searching aids. As discussed by Feibel (1984), established habits of
users play a large role in learning situations. This notion seemed to be reflected in the subjective
satisfaction scores. Furthermore, many of the Browse group were found to have migrated to use
of the index. Rather than adapting to the new situation they were learning, the subjects turned to a
more common method – the index. This study attempted to identify significant differences
between library science graduate students and undergraduates from the psychology pool in terms
of performance, both in browsing or use of an index. Furthermore, search strategies of the two
populations were compared.
1.3 Incidental Learning
Incidental learning takes place when acquisition of knowledge occurs despite a lack of intention. A
large body of literature exists concerning incidental learning. Primary producers of the research are
educators and psychologists. Studies can also be found in paradigms as diverse as medicine
(incidental learning of child-oncology interns in two different hospitals), and political science
(knowledge of New York political campaigns gained by residents of New Jersey contrasted by
geographical region and media population).
The traditional psychological literature is full of experiments involving lists of words – not full-text
situations. Classic studies examined word recognition and the determination of frequencies of
words. Many studied the effects due to age differences (generally attributed to the development of
selective attention). Many, including Craik and Tulving, lean towards an ‘elaboration of encoding’
theory, and examine the effects of depth of processing and episodic memory (see Lindberg and
Wickens, 1979). Dixon studies the effect of personality factors such as motivation and locus of
control on incidental learning conditions (see Dixon and Cameron, 1976). Another study
investigated “pupillary dilation as a possible index of task difficulty for various orienting tasks
during incidental learning” (Krinsky and Nelson, 1981).
Educational psychologists seem to have a less diverse range of research experiments, but more
diverse results. Their research tends to focus on the impact of behavioral objectives and pre-
organizers on the amount of intentional and incidental learning. Some researchers state that
behavioral objectives increase intentional learning and decrease incidental; some claim both are
increased. To help sort this out, Klauer (1984) attempted a meta-analysis of 23 reports generated
during the decade of 1970 to 1980. He analyzed variables such as text length, test difficulty, and
type, level, and number of objectives.
His main finding was that
“giving behavioral objectives, learning directions, or questions before an in-

structional text is read leads to some improvement in the learning of goal
relevant material; however, these preinstructional acts impede the learning of
goal irrelevant material (i.e. incidental learning)” (Klauer, 1984, p.323).
Barker and Hapkiewicz (1979) also attempted to account for various types and levels of behavioral
objectives. The two types they focused on were knowledge level (e.g. the learner will accurately
define X) and evaluation level (e.g. the learner will effectively compare and contrast the theories of
Y and Z). These obviously require different levels of understanding from the student. Whereas
Klauer simply concluded that general (i.e. knowledge level) objectives “produce negative effects
on incidental learning”, Barker an Hapkiewicz studied the interaction of the level of the objective
and the level of knowledge subsequently measured. They found that the learning objectives did
not depress incidental learning at lower levels; however, such learning was depressed when the
questions required a higher level of understanding than the objective demanded. In other words,
subjects given an evaluation level objective (“compare Y and Z”) probably could define X, but
subjects given a knowledge level objective (“define X”) could not effectively evaluate the theory of
Y in the context of Z. They concluded that students given evaluation level objectives must obtain a
better understanding of the basic information in order to reach their objective. This is not in
contradiction with the theory that behavioral objectives steer one's attention away from goal-
irrelevant details, since the lower-level details are relevant to a higher goal.
The task in our experiment was primarily a fact-retrieval task. Wang, Liebscher, and Marchionini
have developed a taxonomy to classify questions used in fact-retrieval. This same taxonomy was
utilized in the selection of questions for the current experiment. The taxonomy will be explained
fully in the next section; but in light of the previous discussion on learning objectives it deserves
mention here that the level of fact-retrieval question posed may have an effect on the knowledge
acquired. Questions which are more complex and less focused may lead to more incidental
learning, as more information must be processed in order to obtain the correct answer. It is
conjectured that such questions might be better explored through browsing than by use of an index
(Wang, Liebscher, and Marchionini, 1987, pp. 6, 14).
One recent study involving incidental learning from a full-text environment (as opposed to word
lists) investigated the effects of text features on vocabulary acquisition (Herman, Anderson,
Pearson, and Nagy, 1987). The unique focus of this experiment has direct implications for the
users of hypertext systems in creating databases.
Educators have hypothesized that meanings of words can be acquired through normal reading of
text, with no emphasis on vocabulary learning. This is known as the incidental acquisition
hypothesis. Herman points out that there is little clear empirical evidence for this hypothesis, and

offers a possible explanation as to why this might be. She contends that the acquisition occurs in
small increments, with portions of a word’s meaning being learned from context rather than a full
grasp of the complete definition being obtained. Previous research may have failed by requiring
students to generate a complete, adult-level understanding of the word.
With this in mind, the authors devised a measure sensitive to various levels of word knowledge.
They produced one study which provided convincing evidence for the incidental acquisition
hypothesis. A follow-up study gave further support. However, discrepancies in the results led to
further reflection. It was determined that anomalies could possibly be a result of the quality of the
text. This insight led to a study in which the authors attempted to systematically control three
different text features through successive rewriting of a text. These features are macrostructure,
microstructure, and the completeness of the explanation of concepts and the relations between
them. It was found that the degree of incidental learning was significantly influenced by the
version of the text that was read. Macrostructure revisions – more complete titles, topic sentences
– did not produce significant results, nor did the combination of macrostructure with
microstructure revisions (making temporal and logical relations explicit). However, students
reading the elaborated version, containing the above improvements as well as an elaboration of key
concepts and presentation of concrete examples, gained more word knowledge, regardless of
ability or any other factor.
Hypertext is an approach which involves a more definite and elaborate presentation of concepts and
their interrelationships than can be found in straightforward presentations of the same material.
Designers of hypertext databases should be aware that concepts which are elaborated in this
manner will be more memorable. Short text is not necessarily better; rather, it is critical that a text
“convey important information precisely, with interconnections fully explained at a level of
specificity appropriate for readers who do not know much about the subject matter” (Herman et al.,
1987, p. 281). The functionality of hypertext makes this quite easy; in Hyperties, for example,
selection of a highlighted term through the embedded menus can lead to a more elaborate
explanation of a concept or an example of an obtuse phenomenon. Readers who are already
familiar with the concept need not follow such paths, but those who are unsure of their meaning
can explore until they are satisfied.
2 EXPERIMENT
2.1 Introduction and Hypothesis
The experiment used a 2 x 2 design with navigational aid (index vs. browse) and subject type

(CLIS students vs. psych pool) as between-subjects factors. The dependent measures were
success in finding the answer to a question and score on an incidental learning task.
The hypotheses were:
1. Navigational aid will have no effect on search success.
2. Library Science students in the index group (having more familiarity with the use of
searching aids such as indexes) will perform better than index-using subjects from
the psychology pool on search tasks.
3. Browsing will have a positive effect on incidental learning; that is, the browse treat-
ment group will have a significantly better score on an incidental learning measure.
2.2 Subjects
A total of 40 subjects were used. Eighteen subjects were graduate students in the College of
Library and Information Sciences (CLIS); eighteen subjects were volunteers from the
undergraduate psychology subject pool. Four subjects were obtained by personal contact, and
were treated as part of the psychology pool group.
2.3 Materials
System: A version of the Hyperties 2.3 browser was used on IBM PCs having two disk drives.
This version of Hyperties captured selections made by users, and noted the time (in seconds)
between choices.
Database: The Hyperties database was compiled by downloading articles related to environmental
pollution from Grolier's Electronic Encyclopedia. A search was done for articles containing the
word ‘pollution’. The encyclopedia presented a list of article titles, along with a frequency count
for the word ‘pollution’. Articles with a high frequency count were downloaded to a floppy disk,
as were articles with a low frequency but which seemed to be directly related to other articles. For
example, Recycling only had a frequency count of one, but was relevant to Land Pollution and
Pollution Control.
In order to structure the text files into a database, the Environmental Pollution article was
selected as the introduction. This article, like many others, was quite lengthy. To produce shorter
articles and take advantage of hypertext capabilities, the outline provided by the Electronic

Encyclopedia was used to divide the article logically. The main sections were water, thermal, land,
pesticide and herbicide, radiation, noise, and air pollution. The first few paragraphs were rewritten
and retained as the introductory article. References were made to the various forms of pollution
and to pollution control. The main sections of the original entry were divided among the three
experimenters, who rewrote and restructured them according to the outlines. Bibliographies from
the original entries were maintained at the ends of articles. If a file was split up into subarticles, the
bibliography was maintained in the article at the head of the network.
Internal links to other articles were determined with the following procedure. First, a concordance
was constructed for each file, giving the frequency count for each word in the article. Words with
a high frequency were examined for clues to other, related articles. This method was not complete
in itself, since phrases were broken into their constituent words (e.g. there would be no consistent
measure for the frequency of the phrase ‘pollution control’). Furthermore, higher frequencies
which could be attributed to the use of synonyms and derivations would be obscured. To
circumvent these problems, articles were read over with attention paid to key points. Important
details that led to other articles were noted. Further assistance was generated by the “(See XYZ)”
citations present in the text of the encyclopedia entries. The experimenters met and drew a map of
articles and links. Where any obvious connections were being neglected, refinements were
suggested and implemented.
Titles for articles were generated by maintaining section headings present in the encyclopedia, or
by expanding them into more complete ideas. Definitions were produced with the aid of key points
in the article.
Questions: The experimenters each generated a set of questions for the set of articles they had
constructed. Questions were then classified according to the taxonomy developed by Wang,
Liebscher, and Marchionini. Five criteria are used to classify questions:
complexity : the number of concepts (facets) represented in the question.
specificity : the variability of a correct or appropriate answer. A three point scale was
used in the studies; the higher the score, the more open-ended the answer could be.
focus : the determinability of the primary facet. A three point scale was used; the
higher the score, the harder it is to choose a main entry point.
path : the length, in terms of articles selected, of the optimal route to find the
information needed to answer the question.
accessibility : the difficulty of finding the right path to an answer. A three point scale
was used; the higher the score, the more difficult it is to find the path.

________________________________________________________________________
| Complexity Specificity Focus Path Accessibility |
|________________________________________________________________________|
|
| Practice 1 2 1 1 1 1 |
| Practice 2 3 1 2 2 3 |
|________________________________________________________________________|
| Question 1 2 1 1 2 2 |
| Question 2 2 2 2 2 1 |
| Question 3 4 1 3 2 3 |
| Question 4 3 1 3 2 2 |
| Question 5 4 1 2 4 2 |
________________________________________________________________________
Table 1: Taxonomy Rankings for Target Questions
______________________________________________________________________________
___
_ _C_omplexity _ Specificity _Focus _ Path _ Accessibility _
______________________________________________________________________________
____
_ Question 1 _ _ 2 _ 2 _ 1 _ 2 _ 2 _
_
__________________________________________________________________
_ Question 2 _ _ 3 _ 1 _ 2 _ 4 _ 2 _
_
__________________________________________________________________
_ Question 3 _ _ 1 _ 1 _ 1 _ 2 _ 1 _
_
__________________________________________________________________
_ Question 4 _ _ 3 _ 3 _ 1 _ 2 _ 2 _
_
__________________________________________________________________
_ Question 5 _ _ 2 _ 2 _ 2 _ 2 _ 3 _

______________________________________________________________________________
_____
_ Question 6 _ _ 3 _ 1 _ 2 _ 2 _ 1 _
_
__________________________________________________________________
_ Question 7 _ _ 2 _ 1 _ 1 _ 2 _ 2 _
_
__________________________________________________________________
_ Question 8 _ _ 3 _ 1 _ 3 _ 3 _ 3 _
_
__________________________________________________________________
_ Question 9 _ _ 3 _ 1 _ 2 _ 2 _ 2 _
_
__________________________________________________________________
_ Question 10 _ _ 3 _ 1 _ 2 _ 2 _ 2 _
______________________________________________________________________________
____
Table 2: Taxonomy Rankings for Incidental Questions
According to Wang, et al.,
"complexity and specificity are generic [system-independent] and objective;
focus is generic but subjective; path is objective but system-based; accessi-
bility describes the interaction of the user's knowledge base, the search task,
and the system". (1987, p. 6)
All questions generated were ranked independently by the experimenters. Agreement on the
ratings was checked before selecting a question. Target questions were chosen by examining the
rankings and attempting to mix levels of difficulty. The first five incidental learning questions were
a result of the choice of target questions; the sentence preceding a target was used to construct an
incidental learning question. The last five posttest questions were picked randomly from available
questions. The questions selected can be seen in Appendix A; the taxonomy rankings for the
questions are in Tables 1 and 2.

2.4 Procedure
Subjects were trained in the use of the system by watching a demonstration of a search for the
answer to the question ‘Why is there a continued need for treatment in the reclamation of
chemically polluted land?’. The Index treatment groups were only shown how to use the index.
The highlighted terms in the index were explained as “clues to other index entries.” They were not
told that these items were selectable. Similarly, the Browse groups were not shown the index.
After the demonstration, subjects were given ten minutes to look up two practice questions in order
to familiarize themselves with the use of the system.
Clarification was given to anyone needing it, but no clues regarding search strategies were given.
Some of the CLIS subjects in the Browse group were noted to be using the index; the group was
then specifically told not to use the index. All groups were explicitly told that there was no need to
hurry when looking up answers. Subjects were primarily run in groups, although a few were run
individually due to scheduling conflicts.
After completion of the two practice questions, subjects were given five questions, one at a time,
for five minutes each. Subjects were instructed to push the F1 key after answering a question.
This generated a timestamp uniquely indicating ‘task completion’ in the log file on the computer
and returned the user to the outermost level on the system. If a subject finished before time was
up, they were instructed to wait until told to begin the next question. Upon completion of all the
searches, a short questionnaire was handed out, which contained the ten incidental learning
questions (see Appendix B) and a short set of questions regarding background and experience.
2.5 Grading
From the log files, search times were calculated for each question. An alternative measure to
search times can be thought of as ‘success’. This was determined by the correctness of the answer
recorded by the subject. For reasons discussed below, this may even be a better measure for the
current experiment. If the answer was the same as our intended answer, it was given a score of 1;
otherwise, it received a 0. The questions had been worded virtually identically to the sentence
containing the desired answer; therefore, incorrect answers were a result of following an incorrect
path.
The incidental questions were scored by two different measures. The first can be called ‘complete

learning’, in which only the precise answer was awarded a score of 1. Anything else resulted in a
score of 0.
An incremental knowledge measure, inspired by Herman, et al., involved partial credit. This was
somewhat subjective and involved assigning fractions of a point to answers which reflected a
portion of the correct answer, or which contained related information from the target article. An
example of this phenomenon occurred many times in question 5: the answer ‘agriculture’ was
awarded 1 point, ‘erosion’ received .75 points. Erosion was mentioned as the result of strip
mining and agriculture in the target article. If a subject wrote an R beside a question, indicating
they remembered seeing the fact but did not remember the answer, a score of .1 was awarded.
3 Results
Data on the search questions were analyzed according to performance on each individual question
as well as collectively across all five question.
CLIS students had marginally faster total times than the subjects from the psychology pool, but
this result was not significant (F = .505, p = .5, df = 1, 36). Neither could
Navigational Aid
_Index___Browse_____totals__
CLIS _ 2.6 _2.8 _ _2.7 _
_(1.2)___(1.3)_______(1.2)___
Psych _ 2.1 _2.6 _ _2.4 _
pool __(1.4)___(1.4)_______(1.4)___
_2.3 _2.7 _ _2.5 _
totals _(1.3)___(1.3)_______(1.3)___
Table 3: Success Measure for Questions 1-5
Mean Number Correct per Subject

any significant effect be attributed to navigational aid. Although some significant results were seen
when examining individual questions, search times were determined to be an inaccurate measure
for two reasons. Inadvertently, not all treatment groups were given the same amount of time per
question. For example, all CLIS subjects had been given six minutes for question 1; CLIS index-
users had been given 6.5 minutes for question 5; some of the Psych browsers had been given six
minutes for question 4. Furthermore, there is the question of correctness. More than a dozen
subjects wrote down an incorrect answer for question 1. A smaller number (6) responded with an
incorrect answer on question 2. The logs for these subjects would indicate a shorter time on the
question, yet they had not actually completed the task. If we were to attempt to normalize the
search times as a percentage of total time given to the group, it would tend to reduce the variability
of the time measure, but it would not nullify the fact that some people had a longer amount of time
to look up an answer. In addition, it doesn’t correct the second problem mentioned above, that of
incorrect answers. It is for these reasons that ‘success’ should be considered a more accurate
measure of the searches done, and used in subsequent analysis.
Basic statistics for the success measure are shown in Tables 3 and 4. Under this measure, CLIS
index subjects outperformed their psych pool counterparts, and CLIS browsers outperformed
psych pool browsers. Furthermore, psych pool browsers outperformed their companions in the
index group. As with the time measure, no significant effect was found overall for either of the
independent variables or their interaction. However, if we examine the percentage of subjects with
a correct answer on each individual question, the following pattern is observed:
Q2 : Navigational Aid F = 6.42 p = .02 df = 1, 36
Q3 : Navigational Aid F = 8.37 p = .006 df = 1, 36
Q5 : Navigational Aid F = 6.46 p = .02 df = 1, 36
For question 2, browsing was superior by a factor of seven to four. Browsers were also more
successful in finding the answer to question 3. On the other hand, the index was clearly a better
access method for question 5. As a matter of fact, every one of the CLIS subjects in the index
treatment group found the answer to question 5. All of the CLIS browsing subjects found the
answer to question 3, but the effect of subject type for this question was not significant.
A table of overall means for the incidental learning questions can be found in Table 5, showing the
number of correct answers within each treatment. On the ‘true’ incidental

Question 1 Question 2 Question 3
_Index____Browse____totals__ _Index___Browse_____totals__
_Index____Browse____totals__
CLIS _ 0.0 _ .22 _._11 _ _.56 _.89 _ _.72 _ _.56 _ 1.0 _._78 *
* _
_(0.0)____(.44)_____(_.32)___ _(.53)___(.33)_______(.46)___
_(.53)____(0.0)_____(_.43)_*
*__
Psych _.36 _ .18 _._27 _ _.54 _.91 _ _.72 _ _.36 _ .73 _._54 *
* _
pool _(.50)____(.40)_____(_.46)___ _(.52)___(.30)_______(.46)___
_(.50)____(.47)_____(_.51)_*
*__
_.20 _ .20 _._20 _ _.55 _.90 _ _.72 _ _.45 _ .85 _._72 *
* _
totals __(.41)___(.41)_____(_.40)___ _(.51)___(.31)_______(.45)___
_(.51)____(.37)_____(_.48)_*
*__
Question 4 Question 5
_Index____Browse____totals__ _Index___Browse_____totals__
CLIS _ .44 _ .22 _._33 _ _1.0 _.44 _ _.72 _
_(.53)____(.44)_____(_.48)___ _(0.0)___(.53)_______(.46)___
Psych _.18 _ .36 _._27 _ _.64 _.45 _ _.54 _
pool _(.40)____(.50)_____(_.46)____ _(.50)___(.52)_______(.51)____
_.30 _ .30 _._30 _ _.80 _.45 _ _.62 _
totals __(.47)___(.47)_____(_.46)___ _(.41)___(.51)_______(.49)___

Table 4: Mean Success Measure for Individual Target Questions
Proportion of Subjects with Correct Answer
Incidental Learning Incremental Incidental Learning
Questions 1-10 Questions 1-10 Questions 1-5
_Index____Browse____totals__ _Index____Browse____totals__
_Index____Browse____total*
*s__
CLIS _ _1.56 _ 0.90 _ _1.22 _ _ _2.71 _1.68 _ _2.19 _ _ _2.38 _1.54 _ _1.96*
* _
___(0.88)___(1.27)___(_1.11)_ _ ___(1.16)___(1.17)___(_1.25)_ _
___(0.87)___(1.19)___(_1.1*
*0)_ _
Psych _ _1.27 _0.82 _ _1.04 _ _ _2.44 _2.09 _ _2.26 _ _ _2.20 _2.06 _ _2.13*
* _
pool ___(0.47)___(0.87)___(_0.72)_ _ ___(0.61)___(0.73)___(_0.68)_ _
___(0.71)___(0.69)___(_0.6*
*9)_ _
_ _1.40 _0.85 _ _1.12 _ _ _2.56 _1.90 _ _2.23 _ _ _2.28 _1.83 _ _2.06*
* _
totals _(_0.68)___(1.04)___(_0.91)_ _ ___(0.88)___(0.95)___(_0.96)_ _
___(0.78)___(0.96)___(_0.8*
*9)_ _
Table 5: Incidental Learning
Mean Number Correct per Subject
learning questions (1-5, where subjects were virtually guaranteed to see the answer i they correctly
answered the target question), no significant effects were found. When all ten questions are taken
into account, navigational aid is a significant factor (F = 5.45, p = .025, df = 1, 36). Examining

Table 5, we see that the index group outperformed the browsers on the incidental learning task.
This was contrary to expectations.
4 Discussion
While it is perhaps discouraging that there were no significant results at the overall level, insight
into the design process can be gleaned by examining the effects that did occur at the level of
individual questions.
Success was not found to be a significant measure for overall performance. However, examining
individual questions shows a number of effects. This indicates that different techniques are
appropriate for the various information seeking tasks. Hayes and Williges (1986) present a three-
way division of search tasks, calling them explicit, clued, and non-clued. An explicit task is one in
which the target is “essentially revealed” by the question. In terms of the taxonomy discussed
above, this would correspond to a question with an accessibility of 1. Clued tasks are those in
which a clue to the file is provided, corresponding to an accessibility of 2. An example of such a
question in the present experiment is question 1. A non-clued question is one which provides no
information regarding the path to the answer. An example of such a question is number 8 on the
post-test; in terms of the taxonomy, the accessibility ranking would be large.
4.1 Analysis – Question 1
From Table 1, we can see that question 1 has a focus rating of 1 and an accessibility of 2. This
would seem to imply that most subjects should find the answer. However, if we examine the
success scores (see Table 4), we determine that only eight subjects found the correct answer! In
particular, note that none of the CLIS index group found it.
There were numerous problems associated with this problem that should be mentioned here. The
web of articles for pollution control was created by splitting the file along the guidelines of the
encyclopedia entry. This included major subtopics of air treatment systems, water treatment
systems, problems of pollution control, and the EPA, with the air treatment systems article
containing the answer. The introductory paragraph to this pollution control web was a section
about general approaches to pollution control. Two of the four methods discussed within that
article did directly mention air pollution. Therefore, many of the subjects were misled into writing
down an answer from a sentence stating “this method of pollution control is most effective...”,
when in fact that method was not even specifically related to air pollution control.

Many subjects began this search with the air pollution articles and then got lost in the information
web. This was partially due to the fact that a link had been misplaced, causing the phrase
‘abatements of air pollution’ to reference the global air pollution article, instead of the intended air
treatment systems article. Unfortunately this was not detected during the testing performed by the
experimenters, or in pilot tests.
Another common problem was that index users never connected the phrase ‘air treatment systems’
with the concept of air pollution control. This title was provided by Grolier's; apparently it did not
fit into the context that the users were building for pollution control. None of the CLIS index
group found the right answer; 4 out of 11 subjects from the psych pool index group did.
Examining the log files for these four, we find that only one of them made air treatment systems
his first choice. Two of the subjects made a browsing selection. It was obvious, from watching
the index-users perform their tasks, that the index entry was not a clear enough clue. Furthermore,
the fact that only two people were able to use the index to find the answer is a strong indicator that
something was awry. It appears that this question, for the index-users at least, corresponds to a
‘non-clued’ question. A few comments regarding the structure of the index and its entries were
made by the subjects. It was suggested that entries be better organized along the lines of “Air
Pollution, Global”, “Air Pollution, Local” (the existing situation being “Global Air Pollution",
"Local Air Pollution”,...).
Four browsers found the correct answer, with five more citing the misguided answer. Question 1,
as stated, gives a pair of clues: one being air pollution, the other being pollution control. By
examining the log files, it appears that many browsers in a clued situation tend to take the first path
presented which fits a clue. Due to the faulty structure of the database, the first clue in this case led
to an unfruitful search. This shed some insight into the behavior of users without a clearly defined
search goal. More study is needed to determine exactly how choices are made.
4.2 Analysis – Question 2
Now let us examine the questions for which a statistical significance was discovered. Question 2
had a significant main effect for navigational aid. The focus for this question was 2; accessibility
was 1. Browsers outperformed those in the index group. The search path for browsers was
clearly outlined within Hyperties: choose land pollution from the introduction, then select soil
misuse, whose definition contained a clear reference to strip mining. On the other hand, many
index users selected articles with clear references to mining, such as agricultural and mining waste
disposal, rather than soil misuse. Again, the title for this article was carried over from the
Electronic Encyclopedia. Within the index, it had no context to give it the intended meaning; in

other words, there was no overt ‘clue’. As a result, index-users had more difficulty in finding the
correct answer. The accessibility rating of 1 (implying a clued situation) which we gave to the
question seems to apply only to the browsing path.
4.3 Analysis – Question 3
Interestingly, the third question had been given accessibility and focus ratings of 3, yet it was also
easier for browsers. All of the CLIS browsers found the correct answer; two-thirds of them were
finished in under three minutes. Some of the browsers went straight from air pollution to local air
pollution. It may be that the word ‘London’ triggered an association with cities and ‘local’-ness, it
may be that some had previously read the article and remembered seeing the answer. Index users
seemed to frequently choose L as the entry point into the index; it would be insightful to know how
many of them were looking for an entry on London. The quickest search was only 51 seconds; it
was conducted by a CLIS index-user. Her first choice from the index was local air pollution.
Examining the log file indicates that the article had not previously been read; in fact, neither had the
main air pollution article. It would be interesting to know what intention or knowledge was behind
this search.
It seems that questions 2 and 3 are clear indicators of the use of context in linking clues during
browsing. In cases where the index titles were out of context, little information was given to the
index-users to help them on their path.
4.4 Analysis – Question 5
Question 5 also showed a significant main effect for navigational aid. The accessibility of this
question was ranked at 2. For this question, however, the index group was the winner, with a
mean success rate of .8 as compared to .45 for the browsers. All the CLIS index subjects found
the correct answer, as did 64% of the psych pool index-users. Of those who found the correct
answer with the aid of the index, 67% of the CLIS and 57% of the psych pool subjects made
recycling their first choice from the index.
For this question, contextual clues were not as clear for the browsers; consequently, their
performance suffered. The correct path for question 5 was land pollution to solid wastes to
recycling. It seems logical to have a direct link from pollution control to recycling, and perhaps
even to have a direct link from land pollution as well. However, neither of these existed in the
original structure of the encyclopedia, so neither was brought into the Hyperties database. Again,

the browsers were presented with two clues at the introduction level. Those who followed the
wrong clue (associating recycling with pollution control, rather than litter with land pollution)
tended not to find the answer. In fact, nearly 80% of the browsers who found the right answer
took the correct path from their initial step. Interestingly, this corresponds to the first of a pair of
clues, which supports the notion that users are persistent in following their first path.
4.5 General Analysis
It appears, given the results cited above, that the interaction of context and clues does have an
impact on the paths people take through a database. More study is needed, however, to determine
exactly what this interaction is.
Using the complete learning measure of incidental learning, where only fully correct answers are
given a score of 1, we find that index users scored nearly twice as well as the browsers (see Table
5). Likewise, if we consider the incremental learning measure, we still find index-users
outperforming the browsers. However, the margin drops to only 37%.
It seems intuitive that the presence of context in selections through the use of embedded menus
should lead to a better overall understanding of the knowledge contained in the database. This
hypothesis, however, is not substantiated by the data. It could be an artifact of the test; that is, the
type of questions asked may not be the most appropriate measure of incidental knowledge.
Another explanation is that the search for particular facts tended to suppress other learning. This is
supported by research involving behavioral objectives discussed above. Since the questions tend
to reflect ‘knowledge-level’ behavioral objectives, it is reasonable that only a small degree of
incidental learning occurred.
Questions at an ‘evaluation level’ might have triggered more incidental learning. However, it does
not explain the difference between index-users and browsers. This could be a result of previous
knowledge, reading skills, or various other factors not controlled in this experiment.
There is another explanation as to why the intuitive notions were not supported (assuming, of
course, that our intuition is correct), and that is that the results were an artifact of the task.
Subjects were focused on looking up isolated bits of information. It may be that the context effects
are not significant for incidental learning in such a situation. If, on the other hand, this database
had been a unit on environmental pollution used in a classroom-type situation, in which students
were responsible for all the information contained within it, rather than scattered bits and pieces,
the desired effect might have been found. In other words, it may be the case that context clues in
Hyperties only have a significant impact when a complete body of information must be

understood.
5 Conclusions
5.1 Impact for practitioners
Many of the problems with this experiment offer clear advice to designers of hypertext systems,
particularly those using Hyperties. The designers of the database under discussion had not actually
written the material contained within it. Many of the problems encountered in the experiment can
be attributed to the fact that articles were imported from another source. Furthermore, they had not
been written in such a manner as to convey the underlying network of associations. In such a
situation, it is imperative that the whole body of articles and their interrelationships be understood,
and then rewritten and restructured to convey the intended connections. In our case, using an
approach where individual articles were treated outside the context of the gestalt information
content of the database resulted in missing links. References that logically should have been made
were overlooked due to our focus on the structure of the entries from the encyclopedia. The
perceived impact of hypertext results from the ability to convey the structure of information; it
became clear that oftentimes extra effort must be expended in order to make this structure clear.
Another issue for designers is the index. The index in the database we used was not consistent in
its naming of articles. Two areas where consistency was practiced are the water pollution web and
the reclamation web. Titles for articles splintered out of the Grolier's reclamation entry all began
with the word ‘Reclamation’. This was mainly due to a lack of creativity, as well as an attempt to
use unique names, but it resulted in a clearer representation of the contents of the articles. Such a
method was not used uniformly by the three designers, or even by this writer. From research with
menu structures, we know that consistency is an important design issue; this must be carried over
as a design principle of indexes within Hyperties and other hypertext systems with an index. The
index can serve as an aid to understanding, so it is important to work on the index (and other such
aids) in order to take full advantage of their capabilities.
Hyperties provides users with definitions of selectable items so that they can determine whether or
not to follow a particular path. If these definitions are well-written in order to convey the meaning
and structure of the article, accessibility is increased. Within hypertext systems, it is important
that links between concepts are clearly defined and represented to the user; otherwise, the result
may be ‘hyperchaos’.
Education has been discussed as an application area for hypertext systems. This experiment

sought to provide evidence that Hyperties and its embedded menus leads to an increase in
incidental learning, thus lending support to the use of hypertext in educational settings.
Unfortunately, such support cannot be garnered from the current research. However, this result
alone does not necessarily imply that hypertext is not a good tool for educational software.
Clearly, results are specific to the database used and the task required. We sought to examine the
navigational aid; perhaps an even more important issue than the effect of such aids that arose from
this work is that the database must form a strong, cohesive, logical entity. This was not the case
with the environmental pollution database; perhaps it was this factor that suppressed the incidental
learning.
It was noticed that browsers had different strategies for highlighted terms; for example, some
scanned the highlighted terms for meaningful connections. Others would read an entire article
before traversing any paths. Still others would follow the highlighting in order – sort of a depth
first search strategy. It is not clear if these methods were a result of the task, that is, attempting to
answer specific questions, or a reflection of individual backgrounds and training. However, from
close analysis of search patterns, their results, and subsequent incidental learning, we should be
able to understand what methods are most effective, if indeed a difference exists. More
experimentation is needed in this realm. Once an ‘optimal’ method for a particular task domain is
determined, however, users should be trained in its use. For example, if it is better to follow a
path in its context rather than reading an entire article and then following paths in order, students
should be taught how to apply this method.
5.2 Suggestions for future researchers
Many experiments can be generated from the framework of this research. Domains outside of
pollution could be examined for incidental learning effects. Perhaps some subjects are better
treated in this manner; perhaps the material needs to be either familiar or entirely new for there to be
a significant incidental learning effect. Since behavioral objectives have shown to decrease
incidental learning, it might be worthwhile to study subjects who are given access to a database but
have no specific framework to follow. These subjects could be divided into two groups by
navigational aid. Maybe then the results would be those we hypothesized for the current work.
Another experiment would be to compare users with no specific task to those given a set of facts to
look up.
It might be the case that users need some sort of ‘road map’ to keep track of their place in the
database, as well as to visualize the underlying structure. It may be that this sort of guide is what
is needed to make hypertext fully compatible within a traditional educational setting. It would be
worthwhile to examine incidental learning and search paths in such situations. One could compare

subjects given such a map to those who have no orienting framework to guide them.
To explore the issue of index consistency, two versions of a database could be constructed,
differing only in their index. Subjects would be given a set of questions which rely on information
present in index entries to guide the search. It is hypothesized that the index which is both
consistent while reflecting the underlying structure would be a better aid.
Another possible study is to use a Hyperties database in the manner alluded to above; that is, to
create a unit of instruction that subjects must read and understand in order to complete their task.
Incidental learning could then be measured, comparing subjects which used embedded menus to
those who had only the index, or perhaps who had read a ‘linear’ text which had not been broken
up into constituent articles.
Much more research is needed to study search habits of individuals, both in guided and undefined
settings. One possible approach is to require users to conduct ‘think-aloud’ searches, in order to
see what thought patterns they are using, and how these can be facilitated by search aids. In order
to truly understand the interaction of previous knowledge with information, we need some measure
of a person's understanding of the structure of information. A network diagram seems like a good
first pass, but a more robust measure should be designed.
5.3 Theoretical implications
It appears that people presented with an interface which encourages browsing tend to use low
cognitive load search methods. This finds support in log files which show that the first item
selected is often the first presented. When a path is explicitly defined, it will usually be followed.
If it is clued, but the clues are vague, it is easy to get misled. In the absence of clues, an individual
will tend to examine all possible paths.
It is fairly clear from the discussion of individual questions that context clues play a large role in
defining the search task in a full-text environment. Goals and plans are derived from a person's
understanding of the underlying relationships. This understanding directs the way an individual
conducts a search. No definite model of a user's performance in full-text information retrieval has
been formulated.
6 Acknowledgements

This paper was originally prepared as a student project for CMSC 828 at the University of
Maryland, under the direction of Ben Shneiderman. Many thanks go to Gary Marchionini for
valuable input in both practical and theoretical issues; to Peter Liebscher and Xianhua Wang for
their involvement in the development and implementation of the experiment; and to my husband
Matthew for believing in it even when I didn't.
Hyperties is available from Cognetics Corporation, 55 Princeton-Hightstown Road, Princeton
Junction, NJ 08550, USA.
The Grolier's Electronic Encyclopedia was used with the written consent of the publishers.
.

REFERENCES
1. Barker, D. and Hapkiewicz, W.G. 1979. The effects of behavioral objectives on relevant and
incidental learning at two levels of Bloom's taxonomy. Journal of Educational Research,
72(6): 334–339.
2. Dixon, P.N. and Cameron, A.E. 1976. Personality and motivational factors on an intentional-
incidental learning task. Psychological Reports , 39:1315–1320.
3. Feibel, W. 1984. Natural phrasing in the delivery of text on computer screens : discussion of
results and research approaches. Proceedings of the NECC . Della T. Bonnette (Ed.),
160–167.
4. Gagne, R.M. 1986. Instructional technology : the research field. Journal of Instructional
Development , 8(3): 7–14.
5. Hayes, B.C. and Williges, R.C. 1986. Defining search strategies in information retrieval.
Proceedings of the 1986 IEEE International Conference on Systems, Man, and Cybernetics,
1108-1112.
6. Herman, P.A., Anderson, R.C., Pearson, P.D. and Nagy, W.E. 1987. Incidental acquisition
of word meaning from expositions with varied text features. Reading Research Quarterly,
22(3): 263–284.
7. Kreitzberg, C. and Shneiderman, B. 1988. Restructuring knowledge for an electronic
encyclopedia, Proceedings of the Tenth International Ergonomic Association.
8. Krinsky, R. and Nelson, T.O. 1981. Task difficulty and pupillary dilation during incidental
learning. Journal of Experimental Psychology: Human Learning and Memory , 7(4):
293–298.
9. Lindberg, M. A. and Wickens, D.D. 1979. Are incidental learning tasks measuring
elaboration of coding, or just overloading retrieval cues? Bulletin of the Psychonomic Society,
13(1): 47–49.
10. Marchionini, Gary. 1988a. An invitation to browse: Designing full-text systems for novice
users. Canadian Journal of Information Science, in press.
11. Marchionini, Gary. 1988b. Making the transition from print to electronic encyclopedias:
Assimilation and accommodation of mental models, International Journal of Man-Machine
Studies, in press.
12. Marchionini, Gary and Shneiderman, Ben. 1988. Finding facts vs. browsing knowledge in
hypertext systems. IEEE Computer, 21(1): 70–80.
13. Wang, Xianhua, Liebscher, Peter and Marchionini, Gary. 1987. Physical format and search

strategy in hypertext: Effects on information seeking, University of Maryland Human
Computer Interaction Lab Technical Report (CAR-TR-353, CS-TR-2006).
14. Yankelovich, Nicole, Meyrowitz, Norman, and van Dam, Andries. 1985. Reading and
writing the electronic book. IEEE Computer, October 1985: 15–30.

Appendix A – Search Questions
Practice Question 1. Give 3 examples of high-level sources of noise pollution.
Practice Question 2. At what rate is the concentration of carbon dioxide in the atmosphere
increasing?
1. Which of the four methods of air pollution control is most efficient?
2. Why does strip mining cause land pollution?
3. How many deaths were attributed to the severe fog of 1952 in London?
4. What nutrients may accelerate the natural aging process of lakes?
5. According to a 1975 report, how much did litter decrease in Oregon due to their recycling law?
Appendix B – Incidental Learning Questions
Knowledge Questions This questionnaire relates to information in the database. You may or may
not have seen the answers to these questions, depending on your search path. Please answer
according to the best of your knowledge. If you remember seeing the fact during your search, but
do not remember the answer, write an `R' beside the number of the question.
1. Which of the four methods of air pollution control is most commonly used?
2. When did Oregon's litter law (the first of its kind in the United States) go into effect?
3. What is smog?
4. What is the effect of a high concentration of suspended solids in rivers and navigational
channels?
5. What is the other major cause of land pollution besides strip mining?
6. What are the two safest types of storage sites currently used for radioactive nuclear wastes?
7. When was the EPA established?
8. How many people were killed in the Aberfan disaster of 1966?
9. Who maintains the International Registry of Potentially Toxic Substances?
10. About what percentage of all water consumption, excluding agricultural uses, is for cooling or
energy dissipation?