Content uploaded by Benjamin Scott Goldberg
Author content
All content in this area was uploaded by Benjamin Scott Goldberg on Apr 28, 2015
Content may be subject to copyright.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2013
2013 Paper No. 13162 Page 1 of 12
Creating an Advanced Pedagogical Model to Improve Intelligent Tutoring
Technologies
Jingjing Wang-Costello, Ron W. Tarr, Luisa M.
Cintron, & Hong Jiang
Cintron
Benjamin Goldberg
RAPTER Laboratory
Institute for Simulation & Training
University of Central Florida
U.S. Army Research Laboratory-Human Research &
Engineering Directorate-Simulation & Training
Technology Center
Orlando, FL
Orlando, FL
{jwang, rtarr, lcintron, hjiang}@ist.ucf.edu
benjamin.s.goldberg@us.army.mil
ABSTRACT
Computer-Based Tutoring Systems (CBTS) are effective learning tools with a high degree of customizability. However, their
application in the training community is limited due to high development costs, limited reuse, and a lack of standards
(Sottilare, et al. 2012). To remedy this issue, the U.S. Army Research Laboratory is developing an open-source modular
program called the Generalized Intelligent Framework for Tutoring (GIFT). GIFT provides a set of tools to author, deliver,
and evaluate intelligent tutoring applications. An essential component of GIFT is a domain-independent pedagogical module
that manages instruction based on a learner’s unique information. The purpose of this pedagogical module is to tailor and
induce intervention via empirically-based generic instructional strategies. The goal of this research is to create an algorithm
in the form of a decision tree within the pedagogical module, which will inform adaptation based on generalized
characteristics associated with the learner and domain being trained.
The authors previously presented a list of learner characteristics (e.g., learner motivation, working memory capacity, prior
knowledge, etc.) that form the basis of this pedagogical model development (Goldberg et al., 2012). For each identified
variable, validated psychometric instruments were selected and threshold levels established (i.e., score designates high/low
groupings). Based on this information, the authors developed an extensive database of empirically validated instructional
strategies. Each strategy was mapped to the four categories of Merrill’s (1994) Component Display Theory (CDT):
Expository generality (general rules), Expository instance (specific examples), Inquisitory generality (recall knowledge), and
Inquisitory instance (apply knowledge). This development resulted in a pedagogical model that provides recommended
generalized strategies for incorporation in the CBTS authoring process. The authors will present work associated with the
model development, highlighting a detailed use-case of its implementation within a specific training instance. In addition, the
authors will also present the results from initial model validation.
ABOUT THE AUTHORS
Jingjing Wang-Costello, Ph.D. obtained a Ph.D. in Applied Experimental Psychology and Human Factors (2011) from
University of Central Florida, a Master’s degree in Telecommunication and Network Management (2004) and a Bachelor’s
degree in Information Science (2002) from Syracuse University. She is currently a Post-Doctoral Researcher with the
Institute for Simulation & Training RAPTER lab.
Benjamin Goldberg, Ph.D. is a member of the Learning in Intelligent Tutoring Environments (LITE) Lab at the U.S. Army
Research Laboratory’s (ARL), Human Research and Engineering Directorate (HRED), Simulation and Training Technology
Center (STTC) in Orlando, FL. He has been conducting research in the Modeling & Simulation community for the past five
years with a focus on adaptive learning and how to leverage Artificial Intelligence tools and methods for adaptive computer-
based instruction. Currently, he is the LITE Lab’s lead scientist on instructional strategy research within adaptive training
environments. Dr. Goldberg is a Ph.D. graduate from the University of Central Florida in the program of Modeling &
Simulation. Prior to employment with ARL, he held a Graduate Research Assistant position for two years in the Applied
Cognition and Training in Immersive Virtual Environments (ACTIVE) Lab at the Institute for Simulation and Training. Dr.
Goldberg’s work has been published across several well-known conferences, with recent contributions to both the Human
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2013
2013 Paper No. 13162 Page 2 of 12
Factors and Ergonomics Society (HFES) and Intelligent Tutoring Systems (ITS) proceedings, and to the Journal of Cognitive
Technology.
Ronald W. Tarr is a senior research faculty member at the University of Central Florida and Program Director of the
Research in Advanced Performance Technologies and Educational Readiness (RAPTER) Lab at the Institute for Simulation
and Training (IST). Ron leads a team of inter-disciplinary researchers who function as analysts, planners, integrators and
designers of the advanced applications of Simulation & Learning Technologies for enhancing human performance.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2013
2013 Paper No. 13162 Page 3 of 12
Creating an Advanced Pedagogical Model to Improve Intelligent Tutoring
Technologies
Jingjing Wang-Costello, Ron W. Tarr, Luisa M.
Cintron, & Hong Jiang
Cintron
Benjamin Goldberg
RAPTER Laboratory
Institute for Simulation & Training
University of Central Florida
U.S. Army Research Laboratory-Human Research &
Engineering Directorate-Simulation & Training
Technology Center
Orlando, FL
Orlando, FL
{jwang, rtarr, lcintron, hjiang}@ist.ucf.edu
benjamin.s.goldberg@us.army.mil
INTRODUCTION
In the military training community, there has been a call for point-of-need training in environments where human
instructors are unavailable or unpractical to use (Sottilare et al., 2012). Past research has suggested that Computer-
Based Tutoring Systems (CBTS) could be an effective training method when utilized properly (VanLehn, 2011).
More specifically, Bloom (1984) stated that human tutoring has an effect size of d = 2.0 as compared to classroom
teaching. CBTs or intelligent tutoring systems on average have produced an effective size of d = 0.31 (Kulik and
Kulik, 1991; VanLehn, 2011). However, despite 50 years of research, CBTSs have not been widely adopted by the
military training community or the general education system. According to Picard (2006), constraints such as high
development cost, limited reuse capability, a lack of standards, and their inadequate ability to adapt to the users have
severely hindered the growth of CBTSs. Specifically, the often complex and ill-defined military training
environment has further hampered the usage of CBTS’ applications in the military (Sottilare et al., 2012). CBTSs
are often built as domain specific, one-of-a-kind solutions that teach specific knowledge areas. However, this type of
framework makes reusing and restructuring a CBTS difficult. To minimize development cost and improve the
capability of CBTS for reuse, the U.S. Army Research Laboratory is in the process of developing an open-source
modular program called the Generalized Intelligent Framework for Tutoring (GIFT). Under the GIFT architecture, a
set of tools is available for instructors to author, deliver, and evaluate intelligent tutoring applications. An essential
component of GIFT is a domain-independent pedagogical module that manages instruction based on a learner’s
unique information. The purpose of this pedagogical module is to tailor and induce interventions via empirically-
based generic instructional strategies. The goal of this research paper is to present GIFT’s engine for Macro-
Adaptive Pedagogy (eMAP), an algorithm in the form of a decision tree that is able to inform adaptation based on
generalized characteristics associated with the learner and the targeted domains. In addition, the results of a
preliminary validation study to exam the implementation of the eMAP are also included in this paper.
THE GIFT FRAMEWORK
GIFT Overview
The Generalized Intelligent Framework for Tutoring (GIFT) is an open-source architecture under development by
the U.S. Army Research Laboratory and is the transition target for the work described. The framework is a domain-
independent, service-oriented architecture that is designed to support the authoring, execution, and evaluation of
empirically based pedagogical functions (Goldberg et al., 2012). Centered on information pertaining to an
individual’s Knowledge, Skills, and Abilities (KSAs), GIFT is intended to manage instruction by tailoring content
and guidance around the strengths and weaknesses of a particular learner. To tailor instruction effectively, strategies
need to be based on both historical information linked to a learner (e.g., trait-based information for macro-
adaptation; Goldberg et al, 2012) and real-time interaction within the learning environment (i.e., state-based metrics
related to performance and affect for microadaptation). This enables GIFT to tailor instruction prior to system
interaction based on what is already known about the learner, and to adapt instruction in real-time based on
progression and performance within a lesson.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2013
2013 Paper No. 13162 Page 4 of 12
Figure 1. Generalized Intelligent Framework for Tutoring (GIFT)
To achieve these capabilities, GIFT is modularly designed with components common to all CBTSs: (1) a Learner
Module comprised of information on individual difference variables used to inform adaptation and performance
states, (2) a Pedagogical Module used to manage strategy/adaptation selection based on the individual’s traits and
performance states received from the Learner Module, (3) a Domain Module that directs the specific training content
and strategies to carry out along with models of expert performance for assessment purposes, (4) a Sensor Module
used to monitor cognitive and affective states that impact learning (e.g., engagement, boredom, confusion), and (5) a
Learning Management System (LMS) to store and collate learner profiles based on outputs from the Learner Module
(See Figure 1; Sottilare et al., 2012). Each module performs separate processes that are associated with the tutoring
effect chain (Sottilare, 2012), where data are used to infer learner states that manage the selection of instructional
strategies intended to influence performance and retention of domain-relevant content (see Figure 2).
Figure 2. Adaptive Tutoring Learning Effect Chain
As GIFT is designed to be domain-independent, defining instructional strategies creates a unique challenge as their
context must be generalizable enough to be applied across multiple domains. Because of this, distinctions were
made between strategies and tactics in that strategies provide high-level pedagogical recommendations (e.g., use
highly visual content), while tactics specify the content or adaptation to implement based on the domain being
instructed (e.g., play video on interrogation techniques; Goldberg et al, 2012). The pedagogical module uses state
information provided by the learner module to determine when a GIFT intervention/adaptation is required and
available trait information to select specific strategies intended to maximize the effectiveness of the instructional
session. For each strategy identified in the pedagogical module, there must be a defined tactic in the domain module
that will be implemented when called upon. To ease this burden, GIFT’s pedagogical module provides a generalized
strategy along with assistance on how to author a tactic based on the context of the recommendation.
With the pedagogical module being comprised of empirically based strategies found to influence learning outcomes,
data must be established to manage strategy selection based on individual differences associated with a data variable
of interest. As reported in Goldberg et al. (2012) an extensive literature review of instructional strategy focused
research was conducted to identify methods found to consistently impact learning outcomes and to determine the
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2013
2013 Paper No. 13162 Page 5 of 12
variables that can be used as selection criteria. This requires three components to be in place. First, individual
difference variables that will be data inputs to the pedagogical module must be identified. These variables were
derived from the literature and are composed of trait characteristics that are rather static in nature (e.g., motivation,
self-efficacy, memory capacity). Second, there must be instruments available to collect data to inform trait values.
These values are what ultimately dictate the selection of strategies among a bank of choices. The third component is
the inclusion of metadata to form generalized descriptors of content and interventions that act as the basis for
selection criteria in terms of choosing specific strategies among a bank of choices.
The Component Display Theory
The eMAP adopted the CDT as its grounding theory. CDT is a set of concepts that describes the conditions,
methods, and outcomes of instruction (Merrill, 1994). It helps in the organization of instruction, along with the
sequencing and presentation of content appropriate for learners. Furthermore, CDT prescribes relationships that can
be used to guide the design and development of learning activities. Thus, CDT was chosen to simplify the
development of instruction in GIFT as it provides the basis for appropriately selecting instructional modules and
their organization (Merrill, 1994).
The CDT model has several unique features that could significantly benefit the pedagogical module (Merrill, 1994).
For instance, it can be used to guide the design and development of learning activities; it provides individualized
instruction in less structured environments; it allows learners to have control over the content; the strategy
components are chosen to fit learners’ momentary state aptitudes and their more permanent trait aptitudes; and it
prescribes instructional conditions based on the types of the desired learning outcome. These instructional
conditions, known as CDT’s Presentation Forms, provide the basic building blocks for the instruc tional strategies
present in the eMAP. CDT indicates two paths when it comes to content as depicted in the Primary Presentation
Forms (See Figure 3): Content can be presented (expository); or the instructor asks the student to remember or use
the content (inquisitory). The content can represent a general case (generality) or it can represent a specific case
(instance). Therefore, instruction can be divided into four categories: Expository generality – present a general case
(Rule); Expository instance – present a specific case (Example); Inquisitory generality – ask the student to
remember or apply the general case (Recall); and Inquisitory instance – ask the student to remember or apply the
specific case (Practice). These four categories can be used as high-level metadata descriptors to label training
content, with each category applying different pedagogical practices inherent to the learning process. Therefore,
instructional strategies can be explicitly defined and categorized within each component of the CDT. This
association allows an instructional designer to understand what a piece of content is intended to provide in a lesson
context (i.e., this video provides an example for enabling objective x), and further instructional strategies can be
defined to inform when this piece of material is most suitable for use. With a framework for organizing content and
applying metadata descriptors, a model is required to determine selection criteria and to perform conflict resolution.
Figure 3. The Component Display Theoretical Model
The Decision Tree
This research effort created an algorithm in the form of a decision tree for authoring the eMAP within GIFT’s
pedagogical module, which informs adaptation based on general learner characteristics and information about the
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2013
2013 Paper No. 13162 Page 6 of 12
domain being instructed. Specifically, the decision tree informs the selection of instructional strategies based on
known information about the learner (e.g., learner motivation, learning style, previous experience, etc.). The
resulting strategies were identified through an extensive literature review of empirically based research, in an
attempt to produce a list of commonly applied strategies found to reliably impact learning outcomes. These
strategies were analyzed and classified into the following learners’ characteristics: learner motivation,
cognitive/learning styles, prior knowledge/experience, learner ability, etc) (see Goldberg et al., 2012 for full list of
sources of adaptation). Subsequently, they were categorized into Rule, Example, Recall, and Practice based on the
CDT. Below is an illustration of the strategies appropriate for learners with low motivation (see Figure 4). The
resulting strategies identified for a specific learner serve as inputs to the domain module for selection of an explicit
tactic to implement for a lesson (see Goldberg et al, 2012). Essentially, the eMAP provides individualized strategy
recommendations for selection or creation of content within each quadrant of the CDT. During its initial
development, a preliminary study was executed to serve as design guidelines in implementation and to assess the
effect of the decision tree in a training context. The first iteration of the eMAP module is available is the current
publicly available release of GIFT.
Figure 4. A Sample of the Decision Tree
A PRELIMINARY STUDY TO EXAMINE eMAP IMPLEMENTATION
The present study was intended to assess whether a course that was customized for an individual based on his or her
motivation level (one of the targeted learner characteristics of the pedagogical module) would yield superior
learning outcomes than a training course that was designed for the general class. For this study, three versions of
land navigation course content were used. The control group course content was directly adopted from an Army land
navigation course. The experimental content had two versions: high motivation learner course content and low
motivation learner course content. These two courses were developed based on the same Army land navigation
content, but integrated training strategies from the proposed eMAP that were more suitable for either high
motivation learners or low motivation learners. Detailed description regarding course content and the grouping of
participants can be found in the material section of this paper. It was hypothesized that participants in the
experimental groups (high motivation learner and low motivation learner groups) would perform significantly better
than the control group.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2013
2013 Paper No. 13162 Page 7 of 12
To investigate whether utilizing instructional strategies to accommodate individual differences (i.e., learner
motivation) could produce better learning outcomes, the study employed a mixed between-within group
experimental design. The first Independent Variable (IV1) was the different types of groups: the control group and
the two experimental groups (high motivation learner and low motivation learner groups as deemed by the
Motivation Strategies for Learning Questionnaire). The second Independent Variable (IV2) was test types: pretest
and posttest. The Dependent Variable (DV) was students’ performance on the baseline knowledge assessment test
and the post knowledge test. Performance was measured by the percentage of correct answers on both tests.
A total of 30 participants were recruited for this experiment (14 males and 16 females), with age range between 19
and 38. They were recruited from the student body of the University of Central Florida. Monetary compensation was
offered to participants as recruitment incentives. Additional demographic information can be found in Table 1. Out
of the 7 participants who reported to have previous experience with land navigation, two participants reported that
they have self-taught themselves map reading as a hobby, one participant performed as a navigator for her cross-
country racing team, and one participant attended geography classes that taught map design. This data was used to
exam potential correlations with participants’ test performance.
Table 1. Demographic Information
Control Group
Experimental Group –
Low Motivation
Experimental Group
– High Motivation
N
10
9
11
Age
19-38 years
21-38 years
19-34 years
Gender
F = 7; M = 3
F = 5; M = 4
F = 4; M = 7
Average Years of Education
15.8 years
14.4 years
15.1 years
Numbers of Participants with
Previous Experience on Land
Navigation
2
2
3
Materials
The Motivation Strategies for Learning Questionnaire
The Motivation Strategies for Learning Questionnaire (MSLQ) (Pintrich, Smith, Garcia, & McKeachie, 1991) is a
self-report instrument designed to assess college level students’ motivational orientation and their use of different
learning strategies for an academic course. It has two sections, a motivation section and a learning strategies section.
Since the scales in the MSLQ were designed to be used either as a whole or used separately, the present study only
used the motivation section to assess students’ motivation towards a land navigation course. Students rated
themselves on a 7-point Likert scale from “not at all true to me” to “very true to me”. The total points were added
together to determine a participant’s motivation level. The present study set 150 points as the threshold to determine
if the participant was motivated to learn about land navigation. If a participant scored higher than 150 points, he/she
was assigned to the high motivation experimental group. An individual who scored equal or lower than 150 points
was assigned to the low motivation experimental group. To score higher than 150 points on the motivation section of
the MSLQ, participants would rate themselves as 5 or above on each of the items.
The Knowledge Test (Baseline and Post)
The knowledge test consists of 26 questions (22 multiple-choice questions, three short answers, and one fill-in-the
blank item) assessing participants’ knowledge on land navigation. The multiple choice questions are worth one point
each. The short answers are five points each, and the fill-in-the blank question is worth three points. The baseline
test and the posttest contain the same questions but in different orders. The knowledge test was graded based on the
percentage of correct answers.
The Self-guided Computer-based Course
The self-guided computer-based course was developed in MS Office Power Point. The lesson content was composed
of materials for training basic land navigation skills linked to map reading and terrain association, and the course
was configured into three versions. For the control group, the content was directly adopted from a land navigation
course from the Army using its original slides and lecture notes. For the experimental groups, using the Army land
navigation course as a foundation, the low motivation learner group course was developed based on the instructional
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2013
2013 Paper No. 13162 Page 8 of 12
strategies from the eMAP. For example, low motivation learners may benefit from using consistent screen format
(Song and Keller, 2001), using graphs and animation (Mayer and Gallini, 1990), and using examples from content
and situations familiar to the learner (Song and Keller, 2001). Thus, the computer-based course for the low
motivation learner course uses a consistent presentation format, contains abundant pictures and graphs, and utilizes
examples that are relatable to the learner. A recording of an instructor was also used to accompany each slide to help
low motivation individuals pay attention to the course content (see Figure 5). Similarly, the high motivation learner
group content was also created based on instructional strategies derived from GIFT’s pedagogical module. For
instance, learners with high motivation have been found to learn better when the course content contains rich linking
technologies (Shin, Schallert, and Savenye, 1994), uses fewer graphics (Mayer and Gallini, 1990), and gives user
control of navigation and pacing of navigation (Bill, 1990). Therefore, the computer-based course for high
motivation learners include additional reading materials via web links, uses less pictures and graphs, and enable
users to move through the course at their own pace (please see Figure 6 for content sample). Much of the previous
research informing the eMAP is based on studies conducted over a decade ago. With the testbed component
provided by GIFT’s modularity, the architecture can support reexamining many of the relationships found from
earlier work for validation purposes within more technologically advanced training environments.
Figure 5. Low motivation learner course content sample
Figure 6. High motivation learner course content sample
Feedback Questionnaire
The feedback questionnaire consists of 15 questions. The first 10 items asks participants to rate their experience of
the course on a 5-point Likert scale from “strongly disagree” to “strongly agree”. Few examples of these items
include “Navigating through the course was easy”, “I enjoyed the instruction”, and “the course materials were
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2013
2013 Paper No. 13162 Page 9 of 12
engaging.” Item 11 to 15 are short answer items. Participants were asked to answer each question in one or two
sentences. Few examples of these items are “Which aspects of the course contributed the most to your learning and
why?” and “Has the course helped you to improve your skills?”
Procedure
Participants were randomly assigned to either the control or the experimental conditions. Upon arrival, participants
were asked to read an informed consent document and given opportunity to ask questions prior to proceed to the
study. All participants received a demographic questionnaire, a motivation questionnaire, and a baseline knowledge
test on Land Navigation (pre-test). Then, the control group participated in a 45-minute self-guided computer-based
Land Navigation course (the control group version) using a Dell laptop with 15-inch display. As for the
experimental groups, depending on their scores on the Motivation Strategies for Learning Questionnaire,
participants were assigned either to the high or the low motivation group. The high motivation group received the
high motivation learner course content and the low motivation group received the low motivation content via the
same Dell laptop computer. Both low and high motivation courses were 45-minutes long. When participants
(experimental and control groups) completed the self-guided computer course, they received a post-training
knowledge test and a feedback questionnaire which asked them about what they liked and disliked about the course.
Results
Analyses were conducted using SPSS 20.0 for Windows. An alpha level of .05 was used for all analyses. Before
analyses were performed, the data was screened for any potential issues that could affect the results of the statistical
analyses (i.e. transcription errors, missing data, etc). The log files were individually examined to ensure the data was
valid and complete for proper analysis.
Correlations
Since Motivation was the learner characteristic selected for this study, a Pearson correlation test was conducted
between learner motivations and the Knowledge test performance to examine the relationships among test
performances and learner’s self-reported motivation levels. There was a significant positive correlation between
learner’s motivation and the baseline knowledge test scores, r = .397, p < .05. The results also showed a strong
correlation between learner’s motivation and post-knowledge test scores, r = .459, p < .05. These two correlations
suggest that motivation is indeed a critical learner characteristic that could influence learning outcomes. In addition,
a significant positive correlation between prior experience with land navigation and the baseline knowledge test
scores was also found, r = .393, p < .05. However no significant correlation was observed between previous
experience and the post-knowledge test scores, thus participants who reported to have previous experience in land
navigation were not treated differently from the rest of the subjects. It may also signify the effectiveness of the
training materials, in that novices with no prior experience produced performance scores not significantly different
from those subjects with previous exposure to the training domain.
Performance Outcomes within Groups
Three separate paired-sample t-tests were conducted to compare participants’ performance from the baseline
knowledge test to the post-knowledge test for the high motivation, the low motivation, and the control group. For the
high motivation group a significant difference was found, t(10) = -10.71, p < .000. Participants scored significantly
higher on the post-knowledge test than the baseline test. Similarly significant differences between the baseline
knowledge test and the post-knowledge test were also found for the low motivation group, t(8) = -2.84, p = .022, and
the control group t(9) = -6.28, p < .000.
Performance Comparison between Groups
A one-way between-group ANOVA was conducted to examine the performance differences on the post-knowledge
tests among the high motivation, low motivation, and control groups. There was a statistically significant difference
among the three groups, F (2, 27) = 7.00, p = .004. Post-hoc comparisons using the Tukey HSD test indicated that
the mean score for the high motivation group was significantly higher than the low motivation group. The control
group also performed significantly better than the low motivation group on the post-knowledge test (see Table 2 for
average group means on the post-knowledge test). The performance differences between the high motivation group
and the control group were not significant. No significant performance differences were found among the three
groups for the baseline knowledge test.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2013
2013 Paper No. 13162 Page 10 of 12
Table 2. Knowledge Test Percentage Scores for Each Groups
High Motivation Group
Low Motivation Group
Control Group
Baseline Knowledge
Test
M = 24.78 % (SD = .06)
M = 22.78 % (SD = .06)
M = 26 % (SD = .07)
Post-Knowledge
Test
M = 51.14 % (SD = .10)
M = 33.33 % (SD = .10)
M = 46% (SD = .12)
Feedback Outcomes
No significant differences were found among the three groups for the first 10 items of the Feedback Questionnaire
(all p > .05). However, it is worth noting that the high motivation group rated most of the questionnaire items higher
(in the 4 range) than the other two groups. The low motivation group rated most of the items the lowest (in the 3
range). For the open ended questions, item 13 asked participants “if you could change this course, what changes
would you make?” High motivation group preferred “More quizzes” and “use more user controlled interaction”. The
low motivation group would like to see “more examples, graphics, and videos.” As for the control group, they wish
they could have “animations”, “some practice”, and “examples”, which were part of the high and low motivation
groups’ content, but wasn’t provided for the control group.
Table 3. Mean Feedback Outcomes
High Motivation
Group
Low Motivation
Group
Control Group
1=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree
1. Navigating through the
course was easy to understand.
4.27
3.67
3.9
2. The flow of the course was
satisfactory.
3.91
3.44
3.8
3. I enjoyed the instructions.
3.82
3.33
3
4. The course materials were
understandable.
4
3.89
3.7
5. The course materials were
engaging.
3.36
3.44
3.5
6. There was appropriate
feedback throughout the course
activities.
4
3.11
3.1
7. The instruction kept my
attention.
3.55
3.56
3.1
8. The combination of audio,
pictures, and text was helpful in
understanding course concepts.
4.09
4.22
4
9. I found the example to be
relevant and meaningful.
4.18
3.33
4
10. Overall, how would you
rate the course?
4.09
3.22
3.6
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2013
2013 Paper No. 13162 Page 11 of 12
Discussion
All three groups showed significant improvement from the baseline knowledge test to the post-knowledge test.
These results suggested that regardless of which group the participants were in, they all improved significantly from
the baseline to the post-test.
When comparing the post-training outcomes across the three groups, the high motivation group and the control
group performed significantly better than the low motivation group. This result may suggest that Motivation as a
learner characteristic can critically influence, as well as predict, training outcomes. For this study, the high
motivation group scored significantly better than the low motivation group. This result supported the early
hypothesis that the high motivation would excel when proper instructional strategies were used to construct the
course. However, it was not expected that the control group also performed significantly better than the low
motivation group. Further analysis showed that 7 out 10 participants in the control group were high motivation
learners (they scored above 150 points on the MSLQ). Because of this high motivation factor, they may learn better
than the low motivation group. Further evidence could be seen via the correlation between learner motivation score
(from the motivation questionnaire) and the knowledge test outcome. Strong positive correlations were found
between motivation scores and baseline and post knowledge test performance. As for low motivation individuals,
the results of this study suggested using motivation training strategies alone may not be enough to boost training
effectiveness. Adding more comprehensive training strategies that deal with other learner characteristics may be
more effective at improving low motivation learners’ training outcome. In addition, interaction with training content
over a more sustained period of time that represents a real-world course may influence performance outcomes.
As for the feedback questionnaire outcomes, there was an interesting trend with the participants’ response from item
1 to item 10. Although not statistically significant, the high motivation group rated most favorably regarding their
land navigation course as compared to the low motivation group who rated least favorably toward their version of
the course. These results followed the same pattern as their post-knowledge test performance where the high
motivation group performed the best and the low motivation group performed the worst. In addition, when
participants were asked “if you could change this course, what changes would you make?”, the high motivation
group asked for more quizzes and more user controlled interaction, whereas the low motivation group would like to
see more examples, graphics, and videos. These responses further suggest that the instructional strategies adopted
for this study matched the students’ preferences. The approach in which the eMAP recommended strategies were
translated in domain tactics should be further assessed to determine if the resulting course met requirements linked
to a strategy’s description. Identifying the optimal approach for authoring domain specific implementations of an
eMAP instructional strategy is an area of research that needs further exploration and refinement. Its implementation
across more interactive training systems must also be examined.
CONCLUSION
The goal of this paper was to present the GIFT eMAP, an algorithm in the form of a decision tree within the
pedagogical module. The decision tree involved many learner characteristics derived from empirical literature,
which produces training strategy recommendations to target each learner variable identified. A preliminary
validation study was conducted to exam one of these characteristics – learner motivation. The results suggested that
learner motivation was a critical factor to consider when designing CBT courses: high motivation individuals may
excel by introducing appropriate motivation training strategies alone. In contrast, low motivation learners may
require a combination training strategies that target multiple learner characteristics (i.e., learner style, prior
experience, knowledge type, etc.). Future studies are needed to investigate how learners’ performance changes when
training strategies for a combination of learner characteristics were introduced. Further exploration in the various
applications used for education and training must also be performed to identify variations in strategy execution.
ACKNOWLEDGEMENTS
This research was sponsored by the US Army Research Laboratory and conducted by personnel from the US Army
Research Laboratory. The views and conclusions contained in this document are those of the authors and should not
be interpreted as representing the official policies, either expressed or implied, of the US Army Research Laboratory
or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government
purposes notwithstanding any copyright notation herein.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2013
2013 Paper No. 13162 Page 12 of 12
REFERENCES
Bill, R. L. (1990). The role of advance organizers, learner control, and student’s locus-of-control on acquisition of
pharmacokinetic concepts and attitudes towards computer-assisted instruction. (Doctoral dissertation).
Retrieved from ProQuest. (Assess Number: 9031295).
Bloom, B. S. (1984). The 2 sigma problem: the research for methods of group instruction as effective as one-to-one
tutoring. Educational Researcher, 13, 4-16.
Goldberg, B. et al. (2012). Use of evidence-based strategies to enhance the extensibility of adaptive tutoring
technologies. Proceedings from Interservice/Industry Training, Simulation, and Education Conference
(I/ITSEC).
Kulik, C., & Kulik, J. (1991). Effectiveness of computer-based instruction: An updated analysis. Computers in
Human Behavior, 7, 75-91.
Mayer, R., & Gallini, J. (1990). When is an illustration worth ten thousand words? Journal of Educational
Psychology, 82, 715–726.
Merrill, M. D. (1994). Instructional Design Theory. Englewood Cliffs, NJ: Educational Technology Publications.
Picard, R. (2006). Building an Affective Learning Companion. Keynote address at the 8th International Conference
on Intelligent Tutoring Systems, Jhongli, Taiwan. Retrieved from
http://www.its2006.org/ITS_keynote/ITS2006_01.pdf
Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A Manual for the Use of the Motivated
Strategies for Learning Questionnaire (MSLQ). Ann Arbor, MI: National Center for Research to Improve
Post-Secondary Teaching.
Shin, E. C., Schallert, D. L., & Savenye, W. C. (1994). Effects of learner control, advisement, and prior knowledge
on young students’ learning in a hypertext environment. Educational Technology Research and Development,
42, 33–46
Song, S. H., & Keller, J. M. (2001). Effectiveness of motivationally adaptive computer-assisted Instruction on the
dynamic aspects of motivation. Educational Technology Research & Development, 49(2), 5-22.
Sottilare, R. A., Brawner, K. W., Goldberg, B. S., & Holden, H. K. (2012). The generalized intelligent framework
for tutoring (GIFT). Orlando, FL: U.S. Army Research Laboratory – Human Research & Engineering
Directorate (ARL-HRED).
Sottilare, R.A. (2012). Considerations in the development of an ontology for a Generalized Intelligent Framework
for Tutoring. International Defense & Homeland Security Simulation Workshop in Proceedings of the I3M
Conference
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring
systems. Educational Psychologist, 46(4), 1-25.
