Content uploaded by Mica R. Endsley
Author content
All content in this area was uploaded by Mica R. Endsley on Sep 30, 2014
Content may be subject to copyright.
THE DESIGNER’S SITUATION AWARENESS TOOLKIT:
SUPPORT FOR USER-CENTERED DESIGN
Debra G. Jones, Mica R. Endsley, Mark Bolstad
SA Technologies, Inc.
Marietta, GA
Gil Estes
BGE Software
Marietta, GA
Designing systems to support SA involves three phases: an analysis of SA requirements,
the application of SA-oriented design principles, and the measurement of SA during
design evaluation. The Designer’s Situation Awareness Toolkit (DeSAT) provides
support to the designer for each phase of this process through both tutorials and
application specific tools. The tutorials cover the Goal Directed Task Analysis (GDTA)
methodology for delineating SA requirements, SA oriented design principles, and the
Situation Awareness Global Assessment Technique (SAGAT) methodology. The
application specific tools include a tool for documenting GDTAs, a Checklist tool to
assist designers in evaluating how well a design concept has met the relevant SA
requirements, a Design Guidance application, and a tool for creating and administering
SAGAT queries (Super SAGAT). Through these tools, DeSAT provides a
comprehensive approach for improving the designer’s ability to create designs based on
sound SA oriented design principles.
INTRODUCTION
The key to enabling users to cope with the
plethora of data characteristic of today’s
“information age” is to design systems that support
the user in building and maintaining a high level of
situation awareness (SA). Current technologies
often do not support operators as they seek to
develop SA; consequently operations are left
vulnerable to error. Problems with SA have been
found to be a major contributor to incidents and
accidents. For example, problems with SA were
found to be the leading causal factor in a review of
military aviation mishaps (Hartel, Smith, & Prince,
1991). In another study of accidents among major
air carriers, 88% of those involving human error
could be attributed to problems with situation
awareness (Endsley, 1995b). A similar review of
errors in other domains (such as air traffic control or
nuclear power) shows that this problem is not
limited to aviation, but is faced by complex systems
in a variety of domains.
Successfully enhancing SA through system
design requires that designers understand how to
combine and present the vast amounts of data
available in order to support the user’s ability to
develop SA. As true SA only exists in the mind of
the human operator, presenting large amounts of
data will do no good unless it is successfully
transmitted, absorbed and assimilated in a timely
manner by the person to form situation awareness.
One of the key benefits of looking at situation
awareness is that it tells the designer how all the
available data needs to be combined in order to
facilitate user understanding. Instead of loading the
user down with 100 pieces of miscellaneous data
provided in haphazard fashion, situation awareness
requirements provide guidance as to what the user’s
real comprehension and projection needs are. By
evaluating the SA requirements, system designers
can determine the best approach for bringing those
100 pieces of data together to form meaningful
information and to create groupings of data that can
be easily absorbed and assimilated in time critical
situations.
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 48th ANNUAL MEETING—2004 653
In the past, the responsibility for correctly
integrating all the relevant information fell on the
user. This task left them overloaded and susceptible
to missing critical factors. Thus, providing systems
that support the SA process is essential meeting a
critical challenge in today’s complex environments.
Meeting this challenge requires a structured
approach that incorporates SA considerations into
the design process, including determining SA
requirements, designing for SA enhancement, and
measuring SA in design evaluation. The Designer’s
Situation Awareness Toolkit (DeSAT) was created
to support designers in effectively addressing these
three aspects of the design cycle. Figure 1 shows
how DeSAT will support each phase of the SA-
oriented design process to provide significant aid to
the designer within the system design process. Each
of the three major subcomponents of DeSAT will be
discussed separately, although they are all
functionally linked and support each other.
Requirements
Analysis
Technology
Analysis
Human Factors
Design Guidelines
& Standards
Design
Concepts
Final
Design
Evaluation
DeSA T
DeSA T
DeSA T
Figure 1: DeSAT and the system design process
SA Requirements Analysis. As a first step,
DeSAT guides the designer through the
identification of situation awareness requirements
for the application being considered. In order to
create effective designs, the designer must
understand what information (as opposed to data!)
the individual operators need as well as what
information their teammates need for a particular
mission. An awareness of the operator’s own state
as well as the state of their teammates, the systems,
and the current operational situation all contribute to
the operator’s overall SA.
SA Design Guidelines. Most significantly,
however, DeSAT does not simply provide a tool for
documenting these SA requirements, but provides a
significant advantage in tying these requirements to
the design process. Today’s designers find a
significant gap between the requirements analyses
they create and the translation of those requirements
into design realities. DeSAT’s role is to help bridge
this gap. The development of a system design for
successfully providing the multitude of SA
requirements that exist in complex systems is a
significant challenge. DeSAT features design
guidance to assist designers through this process.
SA Measurement. Finally there are many
concepts and technologies that are currently being
developed and touted as enhancing SA. Prototyping
and simulation of new technologies, new displays
and new automation concepts are extremely
important for evaluating the actual effects of
proposed concepts within the context of the task
domain. The Situation Awareness Global
Assessment Technique (SAGAT) has been
successfully used to provide this information by
directly and objectively measuring operator SA in
evaluating avionics concepts, display designs, and
interface technologies (Endsley, 1995a). The
SAGAT portion of DeSAT, called SuperSAGAT,
supports designers in systematically evaluating their
design concepts. SuperSAGAT allows designers to
easily customize SAGAT queries for their particular
domain, based on the SA requirements analysis for
that domain.
SA REQUIREMENTS ANALYSIS
The SA requirements analysis component of
DeSAT consist of a tutorial describing how to
delineate SA requirements, a tool to help document
these requirements, and a tool for assessing if all of
the SA requirements have been incorporated into the
design.
GDTA Tutorial
The Goal Directed Task Analysis (GDTA)
addresses the need for delineating user SA
requirements as part of the design process. The
GDTA is a form of cognitive task analysis that
delineates the user’s goals, relevant decisions, and
associated SA requirements for a particular domain.
This tutorial walks the user through why a GDTA is
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 48th ANNUAL MEETING—2004 654
needed, how to construct a GDTA, and what
techniques will help ensure successful data
collection and organization. Basic principles and
techniques are described and examples provided
regarding the use of the technique to ascertain SA
requirements from subject matter experts (Figure 2).
Figure 2: Screenshots from the GDTA Tutorial
GDTA Tool
Using the Goal Directed Task Analysis
approach provides a consistent way to document the
full range of SA requirements as they relate to
specific decisions and goals in a particular domain.
Using a standard format for documenting these
elements facilitates the knowledge elicitation during
interviews with subject matter experts and supports
communication of these requirements between
designers during concept development. This tool
was designed specifically for documenting GDTAs
and supports the designer in quickly creating graphic
representations of the GDTA (Figure 3).
Figure 3: GDTA Tool
GDTA Checklist Tool
The GDTA checklist tool assists designers in
evaluating a concept to ensure that the design meets
all the users’ SA requirements (Figure 4). Drawing
from a GDTA previously entered in the database via
the GDTA tool, this tool presents the user with a list
of goals from which the user can select one or more
goals to evaluate. Once a goal or set of goals has
been selected for analysis, the tool presents the user
with the associated decisions and SA requirements.
The user can then use this list as a checklist to
determine if any SA requirements have not been
accounted for in the design concept. After the user
has worked through the checklist, the tool provides a
summary of what SA requirements have been
omitted relative to each goal.
Figure 4: GDTA Checklist Tool
SA DESIGN GUIDELINES
The SA Design Guidelines portion of DeSAT
includes a tutorial and a guidelines application tool.
Both tools are designed to support designers in
applying SA oriented design principles to the design
process.
SA Oriented Design Tutorial
The SA Oriented Design tutorial delineates 50
design principles relevant to designing systems to
support SA. This tutorial provides basic background
information and a detailed explanation of the
relevant design principles for users who may have
little or no background in this area (Figure 5). It is
divided into 3 modules for ease of use. The first
module covers general SA oriented design issues
and issues related to managing uncertainty. The
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 48th ANNUAL MEETING—2004 655
second module covers issues related to complexity
and alarms. The third module covers issues related
to automation and supporting SA in team operations.
Figure 5: Screenshots from the Design Guidelines
Tutorial
Design Guidelines Tool
The Design Guidelines tool provides the
designer with SA Oriented Design guidance (Figure
6). Based on existing theories of SA, available
empirical evidence, and existing guidelines in the
fields of human-computer interaction and human
factors, a taxonomy was created describing issues
relevant to designing systems to support SA. This
taxonomy provided the structure for further
developing design guidance for supporting designers
in creating systems that effectively support SA.
Using the taxonomy as an organizing feature, this
tool allows users to quickly select a topic and view
relevant questions, design guidance, design issues,
testing issues, examples, and the references from
which the guidance was aggregated. Topics include
general SA principles, as well as guidelines for
designing to support SA through visual displays,
auditory displays, alarms and alerts, controls,
automation, team operations, managing uncertainty,
and managing complexity.
Figure 6: Design Guidelines Tool
SA MEASUREMENT
The SA measurement part of DeSAT includes
two tools. The first is a tutorial describing the
SAGAT methodology. The second is
SuperSAGAT, an easy to use tool for creating and
administering SAGAT queries.
SAGAT Tutorial
The Situation Awareness Global Assessment
Technique (SAGAT) tutorial presents a technique
for objectively assessing user SA (Figure 7). This
tutorial describes the basic principles behind
SAGAT and how to administer SAGAT to
participants during an experiment. Further, it
explains how to analyze SAGAT data to obtain an
evaluation of user’s SA, both at a high level and
with respect to individual aspects of the system.
Figure 7: Screenshots from the SAGAT Tutorial
SuperSAGAT
The ability to effectively measure the impact of
a design on user’s SA is an essential element of SA-
Oriented Design. One proven approach to
objectively measuring SA is the Situation
Awareness Global Assessment Technique (SAGAT)
(Endsley, 1995a). SAGAT provides an unbiased
assessment of a person’s SA by directly querying
them regarding their current knowledge of the
various elements in an environment and comparing
their responses to the actual state of the
environment. The SAGAT methodology involves
freezing a man-in-the-loop simulation at random
points in time, removing information from the
participant’s display, and administering a battery of
queries to the participant. These queries are
developed based on a thorough analysis of the
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 48th ANNUAL MEETING—2004 656
participant’s SA requirements. Although these
queries can be administered via pencil-paper,
collecting the data on a computer reduces the time
and effort required to analyze the data. The
drawback to computerized data collection is that
creating a program to run the queries can be time
consuming and difficult.
To ease this difficulty, SuperSAGAT was
developed to allow the user to quickly and easily
create and administer SAGAT queries. The data
collected with this computer based tool is stored in a
file that can easily be retrieved and formatted for
statistical analysis. This program has three
components: SAGAT Designer (Figure 8), SAGAT
Compiler (Figure 9), and SAGAT FileAdapter.
SAGAT Designer allows users to create a series of
SAGAT queries; SAGAT Compiler administers the
queries to the participant; and SAGAT FileAdapter
assists the user in linking image files to the program
when the program has been transferred from one
operating system to another.
Figure 8: Screenshot from SAGATDesigner
Figure 9: SAGATCompiler
SUMMARY
The Designer’s Situation Awareness Toolkit
provides a unique set of tools to assist designers in
creating systems that effectively support the user’s
ability to build and maintain an adequate level of
SA. DeSAT provides support to the designer for
each phase of the SA oriented design process
through both tutorials and application specific tools
(Figure 10). Further, by stocking the GDTA tool
and SuperSAGAT with existing versions for select
domains, the designer is provided with a foundation
that can be adapted as needed for other domains. By
supporting the designer during each phase of the SA
Oriented Design process, DeSAT provides a unique
tool for developing human centered systems by
providing a comprehensive approach for improving
the designer’s ability to create designs based on
sound SA oriented design principles.
Figure 10: DeSAT and the SA Oriented Design
Approach
REFERENCES
Endsley, M. R. (1995a). Measurement of Situation
Awareness in Dynamic Systems. Human
Factors, 37(1), 65-84.
Endsley, M. R. (1995b). Toward a theory of
situation awareness in dynamic systems.
Human Factors, 37(1), 32-64.
Hartel, C. E., Smith, K., & Prince, C. (1991).
Defining aircrew coordination: Searching
mishaps for meaning, Sixth International
Symposium on Aviation Psychology. Columbus,
OH.
ACKNOWLEDGEMENTS
This work was supported by NASA Langley
Research Center (#NAS1-02069).
¾ GDTA Tutorial
¾ GDTA Tool
¾ GDTA Checklist Tool
¾ SA Oriented Design
Tutorial
¾ Design Guidelines Tool
SA Requirements
Analysis
SA-Oriented Design
SA Measurement ¾ SAGAT Tutorial
¾ SuperSAGAT
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 48th ANNUAL MEETING—2004 657
