Content uploaded by Kazuhisa Seta
Author content
All content in this area was uploaded by Kazuhisa Seta on Jul 10, 2018
Content may be subject to copyright.
1
特集論文 「実践 Linked Open Data」
Content-Dependent Question Generation using LOD
for History Learning in Open Learning Space
Corentin Jouault Osaka Prefecture University
jouault.corentin@gmail.com, http://ks.kis.osakafu-u.ac.jp/john
Kazuhisa Seta (affiliation as previous author)
seta@mi.s.osakafu-u.ac.jp, http://ks.kis.osakafu-u.ac.jp
Yuki Hayashi (affiliation as previous author)
hayashi@kis.osakafu-u.ac.jp, http://www2.kis.osakafu-u.ac.jp/˜hayashi/
keywords: question generation, linked open data, semantic open learning space, inquiry based learning, history learning
Summary
The objective of this research is to use current linked open data (LOD) to generate questions automatically
to support history learning. This paper tries to clarify the potential of LOD as a learning resource. By linking
LOD to natural language documents, we created an open learning space where learners have access to machine
understandable natural language information about many topics. The learning environment supports learners with
content-dependent questions. In this paper, we describe the question generation method that creates natural language
questions using LOD. The integrated data is combined to a history domain ontology and a history dependent question
ontology to generate content-dependent questions. To prove whether the generated questions have a potential to
support learning, a human expert conducted an evaluation comparing our automatically generated questions with
questions generated manually. The results of the evaluation showed that the generated questions could cover more
than 80% of the questions supporting knowledge acquisition generated by humans. In addition, we confirmed the
automatically generated questions have a potential to reinforce learners’ deep historical understanding.
1. Introduction
The current state of linked open data (LOD) provides a
large amount of content. It is possible to access seman-
tic information about many domains. In this paper, we
aim to clarify the potential of LOD as a learning resource.
Our hypothesis is that it is possible to generate meaningful
content-dependent questions in an open learning space by
using the current state of LOD sources.
Questions from the teacher are an important and integral
part of the learning and can deepen learners’ understand-
ing [24]. More specifically, in the history domain, ask-
ing questions itself encourages learners to form an opinion
and reinforce their understanding [11, 23].
Learners also naturally ask questions themselves dur-
ing their learning. They contribute to acquiring new in-
formation and also to clarifying their misunderstandings.
However, they cannot always generate good questions by
themselves [22].
Because the quality of the learning is dependent on the
quality of the questions [05], asking good questions is im-
portant for performing satisfying learning. Learners are
required to generate good questions to perform good qual-
ity of learning, although when they learn with a teacher,
they can rely on the teachers’ questions. This is one of the
difficulties of learners performing their learning by them-
selves.
Therefore, research on technology enhanced learning
has shown that current learning environments can support
learners in self-directed learning by giving inquiry ques-
tions [06]. Questions also provide scaffolding which min-
imizes the difficulties that the learners encounter during
self-directed learning [10].
Jouault and Seta proposed a learning environment built
to enhance self-directed learning in the domain of history
[13, 14, 15]. We plan to embed a question generation func-
tion into it for supporting self-directed learning in an open
learning space.
Our approach combines a natural language open learn-
ing space (Wikipedia) reinforced by semantic information
from two LOD sources, DBpedia [03] and Freebase [04].
By integrating these resources, we create a “semantic open
2
人工知能学会論文誌 31 巻1号SP1-F（2016 年）
learning space” that allows the system to understand the
semantics of information.
The first issue to be clarified to build the question gen-
eration function, which is applicable to an open learning
space, is (a) how we build scalable and reliable knowledge
resource based on LOD and (b) how we build history do-
main ontology and history dependent question ontology to
generate content-dependent questions.
The second issue to be clarified is whether the quality of
the questions generated by the system is sufficient to sup-
port history learning. Although we mentioned the impor-
tance of questions, we should not provide them to learners
if the automatically generated questions are not meaning-
ful enough to deepen historical understanding, because it
might have negative effects on the learners. Therefore, we
must first of all evaluate the quality of the automatically
generated questions themselves before using it.
In this paper, first, we arrange the requirements that
should be satisfied by the knowledge resource. Second, we
show our method to identify history domain concepts and
systemize history domain question ontology. Third, we
describe the evaluation on the topic of ‘World Wars’ com-
paring the automatically generated questions with ques-
tions generated manually by experts to confirm that the
automatically generated questions can support learning.
2. Related Works
One of the important issues of learning support system
is how to realize compatibility of open learning space and
content-dependent support. Advantages of self-directed
learning, a typical learning style in an open learning space,
include a positive effect on learners’ motivation because
they can follow their interests. Learners also can keep
control of their learning, study at their own rhythm by
spending more time on their interests and focusing on their
weaknesses. One of the disadvantages is that learners in a
self-directed learning situation in an open learning space
are confronted to a large amount of information not eas-
ily manageable for unskilled learners. They can become
overwhelmed by the quantity of information and tend to
lose their way without control [31].
Thus, to support learners in self-directed learning in open
learning space, previous research lead to the creation of
systems such as the Navigation Planning Assistant [16],
which provides an environment used to describe learners’
learning plans and state of understanding to prompt their
self-regulation in an open learning space. The limitation
of this system is that its support is content-independent
due to the difficulty of working with natural language in-
formation on the Web. This problem is difficult to over-
come.
On the other side, to provide content-dependent advice,
learning materials can be prepared in advance in a spe-
cific closed domain. This is the case of Betty’s Brain [02],
which uses a concept map in an environment for learn-
ing by teaching, or Kit Build method [09], which pro-
vides a knowledge externalization environment for build-
ing a concept map using pre-defined kits and supporting
the learner during the concept map construction. How-
ever, for both systems, the preparation requires a consid-
erable amount of time even for constructing a closed learn-
ing space. It is not possible to use the same method in an
open learning space because there is too much learning
materials. Furthermore, previous research shows that pro-
viding manually defined terms representing learning ac-
tivities, even without providing answers, has the effects
of prompting learners’ internal self-conversation to under-
stand the contents not explicitly described in a textbook.
As a fact, learners could get higher marks for the prob-
lems whose answers are not described in a textbook with
this method [25, 26].
Our approach tries to build an automatized method that
makes content-dependent support compatible with open
learning space. In the system using our method, we do
not control on the learning materials but the process mak-
ing them machine understandable can be applied automat-
ically. Therefore, it can be applied to an open learning
space, thus, making the system able to generate support
depending on the content of the learning materials.
One of the premising methods to support in an open
learning space is to ask meaningful questions to learn a
specific domain. Inquiry based learning in an open space
is recognized as an useful strategy to avoid learners los-
ing their way and disturbing their learning processes [10].
Notable research is Web-based Inquiry Science Environ-
ment (WISE) [27], which provides support in self-directed
learning. Learners using WISE gather information to an-
swer an inquiry. Learners are trained in designing solu-
tions, debating subjects, and critiquing the resources they
learn. However, preparing all the inquiries in advance re-
quires manual processing by specialists.
To lighten this burden, many other research projects aim
to generate questions automatically. Most methods can
be classified into two categories: syntactic and semantic
approaches.
Syntactic approaches alter declarative sentences to turn
them into questions. An early example of this approach
is work by Wolfe [32]. More recently is work by Heil-
man [08], which generates factual questions by manipu-
Content-Dependent Question Generation using LOD for History Learning in Open Learning Space
3
lating complex sentences. Other notable research includes
CEIST system [33], which analyzes sentences and manip-
ulates the syntax trees to generate questions. One of the
weaknesses of this approach is that it also generates in-
valid questions without understanding of typical context
of the domain, although it is applicable to wide range of
fields.
Semantic approaches try to analyze the meaning of the
sentence to generate questions. Research by Agarwal [01]
analyzed natural language paragraphs by using discourse
cues to generate types of questions: why, when, give an
example, and yes/no. Lindberg et al. [19] used semantic
role labelling of sentences and generate the questions by
using construction rules. Also notable is work by Mazidi
and Nielsen [20], which is close to Lindberg ’s method
but is domain-independent and is also able to provide an-
swers to the generated questions. One of the advantages of
the semantic approach is that most of the generated ques-
tions are valid based on the understanding of concepts in
a domain.
Our method adopts a semantic approach to question gen-
eration that uses the LOD and ontologies to create content-
dependent question about any historical topic. The main
advantage of using LOD to generate questions instead of
using natural language sentences is that there cannot be an
error in the processing of natural language, thus reducing
the number of meaningless questions.
3. A Use Case of Question Generation Func-
tion
Here, we illustrate our system as a use case of the ques-
tion generation function to understand its roles to support
learning, although we do not focus on its learning effects
on learners in this paper.
The question generation function is designed to support
learners during self-directed learning of history by being
embedded into a learning support system, although it is a
general function independent of specific use cases.
Figure 1 shows a screen image of the learning environ-
ment where learners can perform their self-directed learn-
ing with question generation function. It is composed of
four windows:
(a) Question window: It displays a list of questions gen-
erated by the system.
(b) Document window: It displays the learning mate-
rial (Wikipedia document) selected by the learner.
The colors of the concepts in the text correspond to
dynamic highlighting automatically generated by the
system. Orange corresponds to concepts related to
the concept associated to the Wikipedia document.
Green corresponds to concepts known by the learner.
All other concepts are colored in blue.
(c) Answer window: Learners use this window to an-
swer their active questions selected in (a). They can
write their answers to deep questions (described in
Section 4・4) in natural language.
(d) Concept map window: It is used for the learner to
manage the concept map. The display of the con-
cept map is built for history learning. The concepts
in the middle are events. Events are represented on
an automatically generated timeline. The start and
end date of each event is displayed in gray inside the
concept. The size of each event node depends on
the length of the event. Other concepts are colored
in blue. The lines between two concepts are rela-
tions with the type of relation written at the center of
each line. Learners represent their answers to shal-
low questions (described in Section 4・4) by adding
relations to the concept map.
When learners use the system, they first set a main topic
for their learning, e.g. World War I (WWI), World War
II (WWII), Roman Empire, etc. Then, a document in
the Wikipedia about the topic appears in the window (b).
When learning from the document, learners externalize
their knowledge by building the concept map in window
(d). For building their concept maps, they can select click-
able concepts in the documents (Wikipedia pagelinks) to
add them to the concept map. Thus, the system can also
understand the contents of the learner’s concept map.
The system can detect errors in the learner’s concept
map by checking the differences with its own concept map.
The system is also able to identify which questions are an-
swered by the learner. The system has a potential to man-
age errors in several ways to make learners aware of their
own errors. The details are out of focus in this paper.
Then, at any time they want, they can request questions
from the system by clicking the “Generate Questions” but-
ton in window (a). The system generates a list of questions
that can direct their learning activities. The learners then
choose a question they find interesting, and continue their
learning to answer it.
In history learning, not only in self-directed learning but
also in classroom learning, it is important for learners to
build their own image of history [12]. It requires not only
memorization of basic information but also knowledge in-
tegration activities for learners to interpret the social back-
ground at that time period. Giving suitable questions in
the contexts of individual learners’ states have an effect
of making learners aware of the importance of acquiring
4
人工知能学会論文誌 31 巻1号SP1-F（2016 年）
Fig. 1 System interface
new knowledge and to integrate it into their knowledge.
The system generates questions adapted to the learner’s
understanding states represented as the learner’s concept
map. In order to realize the adaptability of the questions,
the system builds its own concept map (Fig. 2) based on
LOD to find out the target knowledge to ask by comparing
it with the concept map of the learner (Fig. 1 (d)). The
system’s concept map contains not only all the concept in-
stances and relation instances in the learner’s concept map
but also more semantic information about them. Thus, the
system focuses on the differences between the two con-
cept maps to generate questions adaptively for individual
learners’ states.
The difference between the two concept maps includes a
lot of concepts and relations that the learner did not study
yet. Thus, to find suitable targets for the questions, the
system adopts a simple ranking algorithm like page rank-
ing. It simply considers a factor of popularity i.e. how
many other concepts refer to the concept.
Figure 2 shows a part of the concept instances and se-
mantic relation instances among them which make the sys-
tem able to understand the chronology including the con-
text. The concepts colored in pink are only in the system’s
concept map. In chapter 4, we describe the construction
process of the system’s concept map.
The aims of embedding the function in the system from
the viewpoint of learning objectivesare two-fold:
(A) Support the development of the domain understand-
ing of the subject to be learned. The system supports
learners to help them learn the important historical
events.
(B) Support the development of learning skills for self-
directed learning of history. The system helps learn-
ers to become aware of the important points during
the study of a historical period.
Regarding objective A, the questions provide support to
help learners not only memorize historical eventsbut also
understand them. In history learning, an understanding of
chronology is necessary [30]. Chronology is defined by
Smart [28] as “the sequencing of events/people in relation
to other and existing knowledge of other, already known,
events/people.” Learning history is not only remembering
a series of facts. Learners need not only to know events,
but also to understand their context.
In our system, the learner can represent the context in-
formation in the Fig. 1 (d). The purpose of the concept
Content-Dependent Question Generation using LOD for History Learning in Open Learning Space
5
Fig. 2 System’s concept map
map window is to make learners describe their understand-
ing explicitly. Research showed that building a concept
map deepens the understanding of learners [21]. Build-
ing a chronology also reinforces learners’ historical un-
derstanding [30].
Regarding objective B, we expect questions to have an
effect of making learners become aware of the importance
of setting objectives in self-directed learning. In our sys-
tem, learners need to set a learning objective by choosing
a question before performing learning activities. Then,
learners need to gather information to be able to answer
the questions. Repeating the process prompts learners to
raise awareness of the importance of inquiry based self-
directed learning of history.
Many researchers pointed out that inquiry based learn-
ing has a positive effect in training learning skills, espe-
cially self-regulated learning skills in open space learning
[10, 27, 34]. Thus, iterating above activities can be ex-
pected to improve self-regulated learning skills.
The system we developed is a client-server application.
The system uses a mirror server loaded with dumps from
LOD (DBpedia and Freebase) to fasten the data gathering
process. The client is implemented as a Java application.
The client requests data from the server using the server’s
SPARQL endpoints. The client uses the data requested to
generate the system’s concept map and the questions.
The next chapter describes how the system generates the
questions.
4. Question Generation Method
To develop a meaningful question generation function
in a historical learning domain, we need to build a scal-
able and reliable knowledge resource containing both ma-
chine understandable and human readable learning materi-
als. Then, we need to build a history domain ontology and
history dependent question ontology to generate meaning-
ful questions for history learning.
4·1 Building integrated LOD
The problem to be solved is to build a reliable knowl-
edge resource for history learning that has both seman-
tic information and natural language information. Natu-
ral language information is required for learners to use as
learning materials. The requirements for the knowledge
sources are:
(a) Unity: The semantic information should represent
the contents of the natural language documents as
much as possible.
(b) Scalability: The natural language and semantic in-
formation should cover most historical important events
that the learners may want to study. Because our sys-
6
人工知能学会論文誌 31 巻1号SP1-F（2016 年）
tem is used for self-directed learning, they set their
learning topics and learn freely by themselves, e.g.
learning about WWI, the American Civil War, etc.
(c) Reliability: The semantic information represented
should be reliable enough to provide meaningful con-
cept instances and context information to learners,
especially about important concept instances.
Regarding (a), for the natural language learning materi-
als, we selected Wikipedia because its fast evolution and
growth give reliable information about a huge number of
topics. In the system, Wikipedia documents are shown
as learning materials in Fig. 1 (b). In addition, if errors
are present, they are corrected with time . An advantage
of selecting Wikipedia is that two semantic knowledge re-
sources (DBpedia [03] and Freebase [04]) are available.
Both of the DBpedia and Freebase projects aim to create a
semantic copy of the knowledge on Wikipedia. Thus, the
requirement of (a) is satisfied by using Wikipedia and its
semantic resources.
Regarding (b) and (c), we built a semantically enriched
resource by combining the above two semantic resources
to give meaningful questions that followlearners’ individ-
ual interests on their learning topics.
The main difference between the two projects is:
•DBpedia’s information is automatically extracted from
Wikipedia; it deals with all of the topicsin Wikipedia.
It covers even minor topics as opposed to Freebase,
while the quality of its information is not as high as
the human made semantic information in Freebase.
•Freebase’s information is provided by humans; the qual-
ity of information is higher than that of DBpedia. While
it has rich semantic information about major topics,
the semantic information about minor topics, but im-
portant for advanced learners, is poor.
We understand that DBpedia and Freebase have differ-
ent characteristics of scalability and reliability. Fig. 3
shows an example of the relation instances defined for two
concept instances: “First Battle of the Marne” and “1st
Army (German Empire).” These concept instances appear
in red. The top part of the figure shows the graphical rep-
resentation of the relation instances defined for the two
concept instances for Freebase (on the left) and DBpedia
(on the right).
The first concept instance “First Battle of the Marne”
is an important event during WWI. Thus, a lot of rela-
tion instances have been defined on Freebase. DBpedia
also has relation instances, but not as many as Freebase.
On the other side, the second concept instance “1st Army
(German Empire)” is minor for most learners, although
advanced learners might have interest. For this concept
instance, the advantage of DBpedia is particularly visible
since Freebase did not define any relation instances about
it.
By following the above consideration, we developed a
method to satisfy the (b) Scalability and (c) Reliability re-
quirements that combines the two semantic information
resources. The bottom part of Fig. 3 shows the combined
relation instances with color depending on their prove-
nances. The system embeds the combined and integrated
information as its own concept map (as shown in Fig. 2).
By integrating two semantic information resources, we
get semantic information that includes the major concept
instances with reliable rich semantic information most learn-
ers should learn and that also covers minor concept in-
stances that might be of interest for advanced learners.
It still maintains the relations with Wikipedia documents,
thus satisfying the (a) Unity requirement.
4·2 Building the history domain ontology
To enable semantic processing for the integrated data,
we need to build an ontology for historical concepts and
relations. We, however, need to consider that both DBpe-
dia and Freebase have respective original class and type
definitions.
•DBpedia: It has an ontology which has been manually
created based on the most commonly used infoboxes
within Wikipedia.
•Freebase: It does not have hierarchically organized on-
tology. It defines types for respective topic categoriza-
tions by domain. The organization by domain is de-
signed to make it easier for users unfamiliar with on-
tology to understand.
Because DBpedia ontology specifies the hierarchy of
concept classes, we simply used its hierarchy and mapped
the Freebase types to the DBpedia classes by defining equiv-
alents. The advantages are:
(i) The data extracted from DBpedia can be easily ap-
plied to the ontology.
(ii) Even the data extracted and integrated from Free-
base gets semantically enhanced by referring to the
ontology.
Figure 4 shows a portion of the history domain ontol-
ogy. Here, we mainly describe the concept class defini-
tions, although we also specify relation classes the same
way as concept class definitions.
In the top left, the types in each domain defined in Free-
base are shown, e.g. the type ‘fr:military conflict’ (F:C1)
and ‘fr:military person’ (F:C3’) are in the ‘fr:military’ do-
main.
Content-Dependent Question Generation using LOD for History Learning in Open Learning Space
7
Fig. 3 Combination of Freebase and DBpedia information for two concepts (Concept map built by the system)
The top right shows classes defined in the DBpedia on-
tology and their hierarchy, e.g. it specifies that the class
‘dbpedia-owl:MilitaryConflict’ (D:C1) is a sub-class of the
class ‘dbpedia-owl:Event’ (D:C0), and both classes ‘dbpedia-
owl:Person’ (D:C3) and ‘dbpedia-owl:Organisation’ (D:C4)
are a subclass-of the class ‘dbpedia-owl:Agent’ (D:C2).
The bottom shows a hierarchy of concept classes de-
fined in the history domain ontology. Their hierarchy is
adopted from that of DBpedia, e.g. it specifies that both
of the concept classes ‘Person’ (S:C3) and ‘Organization’
Fig. 5 Internal representation of Wilhelm II in Freebase, DBpedia,
and their information merged in the system
(S:C4) are a sub-class of the concept class ‘Agent’ (S:C2).
Furthermore, the equivalents for both semantic resources
can be seen under each concept class definition, e.g. the
types ‘dbpedia-owl:Person’ (D:C3), ‘fr:person’ (F:C3), and
‘fr:military person’ (F:C3’) specified as the equivalents un-
der the definition of ‘Person’ (S:C3). According to this
definition, the data integrated from a concept instance of
‘fr:military person’ (F:C3’) and a concept instance of ‘fr:
person’ (F:C3) on Freebase are both dealt as a concept in-
stance of ‘Person’ (S:C3) in the system.
It also specifies how the system merges the data of two
concept instances as an integrated concept instance, even
if the system can judge the sameness of the two concept
instances by referring to the Wikipedia URI.
Figure 5 shows an example of an internal representation
of combined concept instances about “Wilhelm II” based
on the definition of concept class ‘Person’ (S:C3).
Furthermore, the missing concept class or type infor-
mation is complemented, e.g. when gathering informa-
tion about the concept instance “Central Powers”, DBpe-
dia missed the information that it is an instance of the
class ‘dbpedia-owl:Organisation’ (D:C4), while on Free-
base, “Central Powers” is given the type ‘fr:organization’
(F:C4). By specifying that the class ‘dbpedia-owl:Organisation’
(D:C4) and the type ‘fr:organization’ (F:C4) are equiva-
lent under the definition of the concept class ‘Organisa-
tion’ (S:C4), the information can be complemented in the
8
人工知能学会論文誌 31 巻1号SP1-F（2016 年）
Fig. 4 History domain ontology linking Freebase and DBpedia
combined information by referring to the concept defini-
tion of ‘dbpedia-owl:Organisation’ (D:C4).
For building the history domain ontology, the way we
identify enough valid concept (relation) classes related to
history and how we set appropriate equivalents are impor-
tant issues, especially because of the huge size of the DB-
pedia ontology (currently, 685 classes and 2,795 different
properties). Our method to find history related concept
(relation) classes and to set appropriate equivalents is sim-
ple and partially automatic and the approach is based on
real instance data:
(i) First, we identified a huge number of history related
pairs of concept (relation) instances from the two
semantic resources, each of which shares the same
URI, by referring to Wikipedia categories of history.
(ii) Then, the concept (relation) types for both Freebase
and DBpedia set for the concept (relation) instances
were gathered (the classes with the same label are
set as equivalents automatically).
(iii) Finally, we manually associated the equivalent of
classes without equivalents and checked the auto-
matically set equivalents.
In the current version of the history domain ontology,
we specify 121 concept classes and 282 relation classes.
4·3 Building the history dependent question ontology
The system can understand the semantic information to
create questions that can support history learning. Fur-
thermore, the system requires an understanding of a mean-
ingful question’s structure to generate meaningful history
domain questions.
To understand the structure and function of a question,
we refer to Graesser’s taxonomy [07] to build an ontology
for the history domain. This taxonomy describes domain-
independent question types that are meaningful to sup-
port learning. This taxonomy is used by many research
Content-Dependent Question Generation using LOD for History Learning in Open Learning Space
9
Fig. 6 History dependent question ontology and examples of natural language question generated
projects [17, 18].
Figure 6 shows the history dependentquestion ontology
and examples of the natural language questions generated.
The top part of the Fig. 6 shows the History Dependent
Question Ontology (HDQ Ontology) which is divided in
two parts. The left part shows the definition of the domain
independent question concept classes based on Graesser’s
taxonomy. The right part shows the history domain ques-
tion concept classes that we defined.
The HDQ Ontology specifies that there are four major
categories of questions, i.e. (a) ‘Description Question’,
(b) ‘Method Question’, (c) ‘Explanation Question’ and (d)
‘Comparison Question’. Furthermore, more specialized
question concept classes are defined, e.g. ‘Concept Com-
parison’ question, ‘Judgement’ question and ‘Improvement’
question are defined as a subclass of the class (d) ‘Com-
parison Question’.
History domain question concept classes are organized
hierarchically under the domain independent question con-
cept classes, e.g. it specifies that the history domain ques-
tion concept class ‘Concept Completion location of Mili-
taryConflict’ is a subclass of the domain independent ques-
tion type ‘Concept Completion’ which is a subclass of the
category (a) ‘Description Question’.
Each definition of the question concept class in the HDQ
Ontology specifies the relation among the concept (rela-
tion) classes specified in the history domain ontology and
the natural language patterns (NLP).
Each NLP specifies a template of a natural language
question for each question concept class and is used to
generate the natural language text of the question. For ex-
ample, the history domain question concept class ‘Con-
cept Completion location of MilitaryConflict’ associates
the concept class ‘MilitaryConflict’ (S:C1), the relation
class ‘location’ (S:R1) and the NLP “Where did
$
x take
place?” which means that, to create the natural language
question, the $x marker is replaced by the ‘label’ of a con-
cept instance of ‘MilitaryConflict’ having a ‘location’ re-
lation instance.
By defining history dependent question concept classes
based on Graesser’s taxonomy and associating history de-
pendent concept classes with natural language patterns,
the system can generate history dependent questions in
natural language. Currently, the HDQ define 28 history
domain questions.
Based on the definition, the system generates two kinds
of questions:
(A) R&C based Question: Relation and Concept based
Question. These questions are generated using the
history domain question concept classes of the his-
tory dependent question concept classes under the
question category (a) that require a concept instance
10
人工知能学会論文誌 31 巻1号SP1-F（2016 年）
with a relation instance. The answer to this type of
question is identified based on a triple described ex-
plicitly in a concept map built by the system.
(B) C based Question: Concept based question. These
questions are generated using the history domain ques-
tion concept classes of the history dependent ques-
tion concept classes under the question categories
(b), (c) and (d) that require only a concept instance.
Each of these questions asks even about information
not explicitly described.
The table at the bottom of Fig. 6 shows examples of
questions for R&C based question (first line) and C based
question (second line). In both cases, the natural language
questions are generated by filling the history domain con-
cept instance and relation instance which satisfy the con-
straints of a question concept class, into the NLP. For ex-
ample, in the case of the C based question of Fig. 6, the
NLP “How would
$
MAIN TOPIC have been different with
$
x?” is filled by replacing the $x and $MAIN TOPIC
markers by respectively the name of the concept instance
“Wilhelm II, German Emperor” and the name ofthe main
topic of study, in this case “World War I.”
In the actual learning scenario, to limit the number of
questions provided to the learner, the system identifies the
target concept (relation) instances of the questions by com-
paring the concept map built by the learner and the concept
map built by the system with the semantic information.
These targets are identified by checking the information
missing from the learner concept map.
4·4 Function of the questions in history learning
For history learning, Riley [23] pointed out that the ques-
tions can be separated into two groups depending on their
function:
(A) Shallow question: These are simple questions that
can be answered easily. If learners do not know the
information necessary to answer the question, they
can usually find it explicitly described in the docu-
ments. Shallow questions are the kind of questions
that would typically be asked in a basic knowledge
test e.g. “Where did the First Battle of the Marne
take place?” The shallow questions only lead learn-
ers to learn about basic knowledge.
(B) Deep question: These are questions difficult to an-
swer and requiring the learner to think about the learnt
information. Deep questions are the kind of ques-
tions that would typically be asked for an essay writ-
ing test e.g. “How would World War I have been dif-
ferent without Wilhelm II, German Emperor?” More
examples of deep questions appear in Table 3 in the
next chapter.
Regarding the shallow questions, the R&C based ques-
tions can be considered shallow questions because cur-
rently the relation instances (predicates) described in the
current LOD contain only basic relations for history learn-
ing. Question generated based on “Description Questions”
(Fig. 6 (a)) of Graesser’s taxonomy correspond to the
shallow questions. It supports learners in building a ba-
sic knowledge of the topic.
Regarding the deep questions, the C based questions can
be considered deep questions because the knowledge re-
quired to answer is specified in the definition of the HDQ
Ontology. Question generated based on the categories Fig.
6 (b), (c) and (d) of Graesser’s taxonomy correspond to the
deep questions.
Bransford et al. [05] pointed out the importance of giv-
ing domain-dependent meaningful question e.g. when learn-
ing history, inquiry about “who wrote this document, and
how does that affect the interpretation of events?”, whereas,
when learning physics, inquiry about the underlying phys-
ical principle at work. While learning about an historical
topic, it is necessary for learners to learn basic issues about
persons, events, etc. but only memorization about basic
topics does not always deepen their historical understand-
ing.
To deepen their historical understanding, although it is
hard to define what historical understanding is, learners
need to actively develop their knowledge by themselves
through integrating their basic knowledge and questioning
to build their own image/interpretation [12]. Furthermore,
the history professor, appearing in next chapter as evalua-
tor, emphasized the importance for learners to build their
own opinion in history learning, even about topics with-
out any definite answer. He pointed out that an important
learning activity in history is to imagine the situation sur-
rounding an event and the social forces influencing that
situation.
Of course, if the questions are not valid to deepen histor-
ical understanding, we should not provide them to learn-
ers. Therefore, we consider that evaluating the quality of
the automatically generated questions is essential.
We assume that both shallow and deep questions can be
generated by the system. The evaluation described in next
chapter aims to validate this assumption.
Content-Dependent Question Generation using LOD for History Learning in Open Learning Space
11
5. Evaluation of Quality of Questions
5·1 Evaluation method
§1 General evaluation setting
In general, it is difficult to simply compare manually
generated questions with automatically generated ones be-
cause they are not generated in the same conditions or ob-
jectives; a typical difference is that the system generates
questions according to an “individual” learner’s concept
map, whereas authors of a textbook do so according to “as-
sumed learners (specific or general learning objectives).”
In this paper, we adopted SparkNotes as a resource of
questions that is used by a huge number of learners. Then,
we asked a history professor to compare the quality of
questions according to his criteria for evaluating shallow
and deep questions. More detailed information about the
evaluation such as the topic, SparkNotes, and the evaluator
is as follows.
•Topic: WWI and WWII. Our method is specific topic-
independent although the system embeds history do-
main dependent elements. Thus, the current version
of the system can generate history domain questions
for any topic e.g. the Egyptian Civilization, the Ro-
man Empire, etc. However, we might need to add def-
initions of questions and natural language patterns in
the question ontology to adapt, even though our frame-
work can accept such extensions without any changes
of question generation system. Thus, we have to care-
fully address this issue with the combination of the set
of definitions in question ontology and historical top-
ics. Based on this recognition and to raise the reliabil-
ity of the results, we judged that it is more adequate to
choose the topics in the field to which the same set of
definitions of question ontology can be applied.
•Source of human generated questions: SparkNotes
[29]. This is a popular website, whose target users are
mainly junior high-school and high-school students that
provides learning materials, quizzes and essay ques-
tions in many domains. Around 20 million unique
users from all over the world access the website each
month. Thus, it is recognized as a meaningful website
for history learning and gives fundamental questions
to support learning. The questions for our experimen-
tal study were taken from the multiple choice quiz and
essay questions. We used all of the 58 questions about
WWI (58: WWII) in SparkNotes. Then, they were
manually separated into 38 shallow (31: WWII) and
20 (27: WWII) deep questions by identifying the 38
(31: WWII) shallow questions with an answer writ-
ten explicitly in the learning materials. The remaining
20 (27: WWII) questions not having a definite answer,
like essay questions, but contributing to deepening his-
tory understanding were categorized into deep ques-
tions.
•Evaluator: A history professor at a university with
over 20 years of history teaching experience.
For this evaluation, it is essentially not possible to eval-
uate objectively the quality of questions according to some
common objective criteria. Only subjective evaluation by
human experts, based on their respective criteria, is possi-
ble.
It is difficult to objectively generalize the evaluation re-
sults. Respective results conducted by human experts are
meaningful even if they might have contradictions. Re-
sults from different experts should not be merged because
they are built on different subjective criteria.
In our experimental study, while only one reliable ex-
pert with considerable experience of history teaching con-
ducted this evaluation, his opinion is valuable because it
is built on a strong knowledge of historical topics, history
learning strategies, and history teaching methods.
The evaluation wasconducted with the server in the fol-
lowing setting: Virtuoso 6.1.6 on Ubuntu server 3.13 to
manage SPARQL endpoints. The database was loaded
with data dumps from DBpedia 3.9 and Freebase RDF of
January 2015.
§2 How are shallow questions evaluated?
In the case of the shallow questions, the system gener-
ates them by using explicitly described semantic relation
instances representing fact (predicate) knowledge. The
size of a set of questions (R&C based Question) generated
by the system depends on the amount of semantic relation
instances represented, although the system has an advan-
tage of generating them adaptively to individual learners.
Therefore, it is important to evaluate the coverage of hu-
man questions by the set of system generated questions
to confirm whether the question generation function with
current states of LOD can be available for building/assessing
learners’ fundamental knowledge. The evaluation of the
shallow questions considered the 38 manually generated
shallow questions about WWI (31: WWII) taken from the
website SparkNotes. Each manually generated question
was paired with an automatically generated question. An
example of the process for the human generated question
“Who won the Battle of the Falkland Islands?” (Answer:
“United Kingdom”) is as follows.
•First, we manually analyzed the human generated ques-
tions to identify the concept instances involved. The
target concept instances of the question and answer
were manually identified: “Battle of the Falkland Is-
12
人工知能学会論文誌 31 巻1号SP1-F（2016 年）
Table 1 Shallow Questions Evaluation Results
Mark 1234Total
Number WWI 5 (13%) 15 (39%) 14 (37%) 4 (11%) 38
of WWII 6 (19%) 15 (48%) 5 (16%) 5 (16%) 31
couples Total 11 (16%) 30 (43%) 19 (28%) 9 (13%) 69
lands” and “United Kingdom.”
•Then, the system generated as many questions about
the identified concepts as possible. The generated ques-
tions were about all the involvedconcept instances de-
pending on the relation instance information available
e.g. “Where did the Battle of the Falkland Islands hap-
pen?”, or “Which battles involved United Kingdom?”,
etc.
•Finally, we manually selected the question with the
closest possible meaning. In that case, the selected
question was “What was the result of the Battle of the
Falkland Islands?”
In total, 38 couples of questions about WWI (31: WWII)
containing each one manually and one automatically gen-
erated question were created for the evaluation.
The evaluator was asked to give a mark from 1 to 4 for
each couple, where 1 means the knowledge required to
answer is different and 4 means the knowledge needed is
the same. We asked the evaluator to set concrete criteria
from the viewpoint of history learning.
§3 How are deep questions evaluated?
In the case of the deep questions, the knowledge that
human and system can use to generate them is different.
On one side, a teacher has deep knowledge of the topic
including the knowledge not written in a textbook as well
as the ability to read between lines. Thus, a teacher can
generate questions that require a learner to think between
lines to construct historical deep understanding including
knowledge not explicitly described.
On the other side, the system can generate only based
on explicitly represented information and ontologies that
require learners to state their opinion.
The purpose of evaluating the deep questions is to clar-
ify whether the system can generate questions that trigger
historically deep thinking by using explicitly represented
domain knowledge in LOD and specific topic independent
ontologies.
The evaluation considered the totality of the 20 (27:
WWII) manually generated deep questions and 30 (30:
WWII) deep questions automatically generated by the sys-
tem. The automatically generated questions were gener-
ated by referring to the concept instances mentioned on the
SparkNotes page “Key People and Terms” about WWI.
Among 55 (75: WWII) concept instances identified, 30
(30: WWII) concept instances with the most semantic in-
formation were selected and used as a target for the ques-
tion generation.
The manually and automatically generated questions were
then scrambled together. The evaluator was provided with
the resulting 50 (57: WWII) questions and instructed to
judge the ability to deepen learners’ understanding. Then,
the evaluator was asked categorize all the questions into
5 categories (C1-C5). Questions in category C5 are good
questions contributing to deepening the understanding of
the learners, whereas questions in category C1 do not con-
tribute significantly to deepening the understanding. The
specific criteria were left up to the evaluator.
5·2 Results
§1 Quality of shallow questions
Table 1 shows the number of couples for each mark.
The evaluator described the criteria used for attributing
grades during the evaluation as follows.
1: Both questions require different knowledge to be an-
swered e.g. “To which other prominent leader was
Kaiser Wilhelm II of Germany related?” (Human) and
“Who is related to Wilhelm II, German Emperor?” (Sys-
tem).
2: Both questions focus on different parts of the target re-
lation instance, but they require the same knowledge
to be answered, e.g. “Who won the Battle of the Falk-
land Islands?” (Human) and “What was the result of
the Battle of the FalklandIslands?” (System).
3: Both questions assess the same knowledge from dif-
ferent viewpoints (they require the same knowledge to
be answered), e.g. “Who assumed power in Germany
and led negotiations with the Allies after Wilhelm II
lost power?” (Human) and “Who succeeded Wilhelm
II, German Emperor?” (System).
4: Both questions have the same meaning e.g. “In what
city was Archduke Franz Ferdinand assassinated?” (Hu-
man) and “Where did the Assassination of Archduke
Franz Ferdinand of Austria take place?” (System).
The evaluator judged that couples marked 2, 3 and 4 re-
quire the same knowledge for junior and high school stu-
dents to be answered. Thus, we recognize couples marked
2, 3 and 4 were the same questions from the viewpoint of
requiring the same knowledge structure. In total, under the
Content-Dependent Question Generation using LOD for History Learning in Open Learning Space
13
Table 2 Deep Questions Evaluation Results
Category (Weight) C1
(1) C2
(2) C3
(3) C4
(4) C5
(5) W. Avg.
WWI Human (n=20) 15833 3.1
System (n=30) 111990 3.2
WWII Human (n=27) 610713 2.4
System (n=30) 0121116 3.0
Total Human (n=47) 7151546 2.7
System (n=60) 1 1330106 3.1
conditions of the system, 84% of the manually generated
questions about WWI and WWII could be covered by the
system.
The remaining 16% of questions were not invalid or
useless questions. For these questions, the system was not
able to generate questions that require the same knowledge
from the learners. The questions generated by the system
asked about different basic knowledge.
During the interview after the above experiments, he
suggested that even shallow questions for junior and high
school students may become deep questions for university
students with deep historical knowledge, because they in-
terpret their questions differently. We understood this is a
characteristic feature of history domain learning.
Thus, as an additional strict evaluation on the topic of
WWI, evaluator was asked to judge for each couple about
the quality of one question compared with the other. The
results of this additional evaluation showed that for 39%
(15/38) of the couples, the questions did not have a differ-
ence in quality. The system generated question was better
in 29% (11/38) of the couples, and the human generated
one was better in 32% (12/38) of the couples. These re-
sults show no significant difference between the quality of
the system and human generated shallow questions.
As a result, the question generation function has a po-
tential to generate useful questions to construct learners’
basic knowledge. Furthermore, it suggests that even shal-
low questions, in this case, R&C based question might be
useful even for learners studying history in a university.
§2 Quality of deep questions
Table 2 shows the number of questions for each cate-
gory. The result shows that it can generate questions of
the same quality as the manually generated questions in
average.
The criteria defined by the evaluatorwere:
C1) Questions asking facts.
C2) Questions asking causal relations.
C3) Others, more complex than C2 but does not require
complete integrated knowledge like C4. It requires
an understanding of the topic of the question and its
context.
C4) Questions requiring integrated knowledge of the topic
as a whole. It requires knowledge of the topic of the
questions as well as a general understanding of the
main topic important events and their context.
C5) Questions requiring a deep historical or political think-
ing. It requires having an understanding of global
history and the relations between the topic and other
historical topics.
The evaluator judged the quality of questions in cate-
gory C1 and C2 are still meaningful, whereas the quality
of the questions in category C3 or above are more suitable
in a sense of prompting learners to build their own image
of the past.
The system only uses information specific to one topic,
i.e. using one of the key concepts defined in SparkNotes,
to generate a question in this experiment. Although it de-
pends on the templates that were used by the system, the
majority of the questions are of reasonable quality (C3 or
above). Examples of questions for each category and for
both WWI and WWII can be seen in Table 3.
As a result, the question generation function has a po-
tential to generate useful questions to reinforce learners’
deep understanding of historical topics.
5·3 Discussion
The generated questions can cover most (84%) of the
shallow questions generated by humans. In addition, the
deep questions generated by the system and humans have
the same average quality. The results indicate that the
questions generated by the system can be expected to sup-
port learners, in both acquiring basic knowledge and build-
ing a deep understanding, like human generated questions.
In addition, the system can generate a huge quantity of
questions covering many historical topics. Because the
generated questions are semantically correct and are mean-
ingful for supporting learners, we consider that the HDQ
Ontology built by specifying Graesser’s taxonomy seems
14
人工知能学会論文誌 31 巻1号SP1-F（2016 年）
Table 3 Examples of deep questions
Human System
C1 What did Germany do in 1917 to hasten Russia’s
exit from the war? Was the influence of George V important during
World War I?
WWI
C2 What was the initial purpose of Britain’s invasion
of Mesopotamia? How did the fights of the First Battle of the Marne
change the course of World War I?
C3 Why did Britain need control of the Dardanelles? What were the consequences of World War I for
the people living in Mons?
C4 What was the political result of Britain’s invasion
of Gallipoli? How did the use of Submarine as mean of trans-
portation changed the course of World War I?
C5 World War I has often been described as an “un-
necessary war.” Why? Do you agree? ―
C1 What was the Soviet defense plan against Ger-
many? ―
WWII
C2 Was there any justification for Britain and
France’s policy of appeasement? What were the consequences of World War II on
Nazi Germany?
C3 Why did Allied forces invade Italy after it had al-
ready surrendered? How did the fights of the Battle of Iwo Jima
change the course of World War II?
C4 What were Germany’s mistakes in Russia and how
did they affect the outcome of the war? What kind of influence had Schutzstaffel on World
War II?
C5 Were Germany and Japan aggressions fundamen-
tally similar or fundamentally different? How would World War II have been different
without Adolf Hitler?
valid.
The main advantage of our question generation method
for the self-directed learning system is that the questions
are adapted to each learner. The system generates ques-
tions depending on the learners’ knowledge states (repre-
sented by their concept map). The adaptability is benefi-
cial to learners from the viewpoint of enhancing motivated
learning.
From the technical viewpoint, our system has the advan-
tage in a sense of generating deep questions based on the
semantic processing (LOD). However, no existing system
has been developed that can generate an answer to any his-
tory deep (essay) questions. Thus, it is important that we
confirmed the deep questions generated with our method
have a possibility of meaningfulness from the viewpoint of
deepening historical understanding, even the system can-
not give the answer.
One of the limitations of current LOD based question
generation is that current LOD as a learning material of
history is limited, i.e. the information described contains
only facts and no opinions. Opinion information is in-
cluded on Wikipedia as natural language but is not de-
scribed on the current LOD.
An interesting issue, from the viewpoint of AIresearch,
of applying the question generation function to the history
domain is that it can generate valid and meaningful ques-
tions based on the ontologies even if the answer is not ex-
plicitly represented in the LOD. For example, the answer
to the question “What kind of influence had Schutzstaffel
on World War II?”, which is a category C4 question gener-
ated by the system, is not described in the LOD. A unique
answer for this question does not exist as opposed to a
question that could be asked in the mathematics domain.
The evaluator (history professor) said “asking this type of
question is meaningful in a sense of requiring integrated
knowledge about the topic.” It means that they contribute
by encouraging thinking between lines apart from remem-
bering facts. The reason why the system can generate this
type of questions is that we could build reasonable ontolo-
gies as well as the domain of application being the history
domain.
From this point of view, to answer C5 categorized deep
questions such as “How would WWII have been different
without Adolf Hitler?”or“Did technology fundamentally
affect the outcome of the war? If so, how? If not, why
not?” they need to understand the roles and influence
of the events. Obviously, each of these questions does
not have a unique correct answer but thinking by them-
selves about these kinds of questions in their internal self-
conversation and thinking between the lines is important
to build their own image of roles and states of persons,
events and etc., even in the situation that the validity of
Content-Dependent Question Generation using LOD for History Learning in Open Learning Space
15
the answer cannot be confirmed.
The results of both evaluations demonstrate the validity
of our method for generating questions about the topics
on ‘World Wars’. However, the quality of the generated
questions about other topics is not verified. We need to
carefully address the quality of questions with the set of
definitions specified in the question ontology. The results
in other historical topics might be different: we might need
to add definitions of history domain questions and natural
language patterns in the question generation ontology to
adapt, although our framework can accept such extensions
without any changes of question generation system.
6. Concluding Remarks and Future Works
In this paper, we described a method for generating ques-
tions automatically by using LOD. The history domain on-
tology makes it possible to use the concept (relation) in-
stances from two semantic resources (DBpedia and Free-
base). The history dependent question ontology makes
possible to generate content (topic) dependent questions
using domain dependent but content (topic) independent
question concept classes. One of the advantages of the
system is that we can add definitions of history domain
questions and natural language patterns in the question on-
tology without any changes of question generation system.
The questions generated by the system can be expected to
support learners in acquiring basic knowledge and deep-
ening their understanding of history.
The evaluation showed that the system could generate
good quality questions about ‘World Wars’ by using the
current LOD. The experimental results showed that the
questions generated by the system can cover the major-
ity of the questions generated by humans. In addition, the
questions enhancing history thinking generated by the sys-
tem and by human were of the same average quality. By
considering the system can generate much more questions
not appearing on SparkNotes and its adaptability, the re-
sults described in this paper seems quite meaningful in the
situation of individual learners’ support. However, we rec-
ognized there is no unique criterion to evaluate the quality
of history questions. More investigation has to be done by
other history professors.
In future work, by developing a richer ontology includ-
ing multiple concepts and relations, and with richer se-
mantics being embedded in LOD, the system could be-
come able to generate more meaningful questions from
richer relation instances information. Furthermore, we
need to carefully address the quality of questions about
other topics with a combination of a set of definitions spec-
ified in question ontology and the topic.
In this paper, we build a domain independent general
framework for question generation using LOD, although
we confirmed that the quality of questions generated by
the method only on the topic of World Wars. The impor-
tant point to realize the generality of the system is we sep-
arate domain independent and history dependent concepts
of the question generation ontology. Because the mean-
ingful questions to enhance understanding depend on re-
spective learning domains, we should systemize domain
dependent question ontology to adapt to other domains. In
other words, the system has a possibility to adapt to other
domains by changing the history dependent question on-
tology to another domain dependent one.
Furthermore, we think two requirements need to be sat-
isfied for the question generation to adapt to other do-
mains:
1. Meaningful information needs to be available on the
LOD to be able to support learning about topics in the
targeted domain.
2. The targeted domain ontology must be specified. The
current ontology only specifies concept and relation
classes for the history domain.
The kind of learning domains that can satisfy above re-
quirements should be clarified in the future papers.
Finally, the way the function is embedded in the sys-
tem and its utility from the viewpoint of learning support
should be carefully addressed through our next works, e.g.
how we embed the question generation function into the
system, how and when the system shows shallow and deep
questions, how many questions the system shows, whether
the system can give positive influence to learners, and so
on. We expect to conduct several experiments to address
the above points in future works.
Acknowledgments
We would like to offer our special thanks to Professor
Akifumi Sumitomo for his meaningful comments.
This work was supported by JSPS KAKENHI Grant
Numbers 24300288, 15H02934.
♦References ♦
[01] Agarwal, M., Shah, R. and Mannem, P. (2011). Automatic ques-
tion generation using discourse cues. In Proceedings of the 6th Work-
shop on Innovative Use of NLP for Building Educational Applica-
tions, p. 19. Association for Computational Linguistics.
[02] Biswas, G., Roscoe, R., Jeong, H. and Sulcer, B. (2009). Pro-
moting self-regulated learning skills in agent-based learning environ-
ments. In Proceedings of the 17th International Conference onCom-
puters in Education, pp. 67-74.
[03] Bizer, C., Lehmann, J., Kobilarov, G., Auer, S., Becker, C., Cy-
ganiak, R. and Hellmann, S. (2009). DBpedia-A crystallization point
16
人工知能学会論文誌 31 巻1号SP1-F（2016 年）
for the Web of Data. InWeb Semantics: Science, Services and Agents
on the World Wide Web, Vol.7, No.3, pp. 154-165.
[04] Bollacker, K., Evans, C., Paritosh, P., Sturge, T. and Taylor, J.
(2008). Freebase: a collaboratively created graph database for struc-
turing human knowledge. In Proceedings of the 2008 ACM SIGMOD
International Conference on Management of Data, pp. 1247-1250.
[05] Bransford, J. D., Brown, A. and Cocking, R. (1999). How Peo-
ple Learn: Mind, Brain, Experience, and School. Washington, DC:
National Research Council.
[06] De Jong, T. (2006). Technological advancesin inquiry learning.
In Science, Vol.312, No.5773, pp. 532-533.
[07] Graesser, A., Ozuru, Y. and Sullins, J. (2010). What is a good
question?. In Bringing Reading Research to Life. Guilford Press.
[08] Heilman, M. and Smith, N.A. (2009). Question Generation via
Overgeneration Transformations and Ranking, No. CMU-LTI-09-
013. Carnegie-Mellon University, Pittsburgh.
[09] Hirashima, T., Yamasaki, K., Fukuda, H. and Funaoi, H. (2011).
Kit-build concept map for automatic diagnosis. In Artificial Intelli-
gence in Education, pp. 466-468. Springer Berlin Heidelberg.
[10] Hmelo-Silver, C. E., Duncan, R. G. and Chinn, C. A. (2007).
Scaffolding and achievement in problem-based and inquiry learning:
A response to Kirschner, Sweller, and Clark (2006). In Educational
Psychologist, Vol.42, No.2, pp. 99-107.
[11] Husbands, C. (1996). What is History Teaching?: Language,
Ideas and Meaning in Learning about the Past. Berkshire: Open Uni-
versity Press.
[12] Husbands, C., Kitson, A. and Pendry, A. (2003). Understand-
ing History Teaching: Teaching and Learning about the Past in Sec-
ondary Schools. McGraw-Hill International.
[13] Jouault, C. and Seta, K. (2013). Adaptive self-directed learning
support by question generation in a semantic open learning space.
In International Journal of Knowledge and Web Intelligence, Vol.4,
No.4, pp. 349-363.
[14] Jouault, C. and Seta, K. (2013). Wikipedia-based concept-map
building and question generation. In The Journal of Information and
Systems in Education, Vol.12,No.1, pp. 50-55.
[15] Jouault, C. and Seta, K. (2014). Content-dependent question gen-
eration for history learning in semantic open learning space. In Intel-
ligent Tutoring Systems, pp. 300-305. Springer International Publish-
ing.
[16] Kashihara, A. and Taira, K. (2009). Developingnavigation plan-
ning skill with learner-adaptable scaffolding. In Proceedings of the
2009 Conference on Artificial Intelligence in Education: Building
Learning Systems that Care: From Knowledge Representationto Af-
fective Modelling, pp. 433-440. IOS Press.
[17] Le, N.T. and Pinkwart, N. (2014). Question Generation Using
WordNet. In Proceedings of the 22nd International Conference on
Computers in Education.
[18] Li, H., Duan, Y., Clewley, D. N., Morgan, B., Graesser, A. C.,
Shaffer, D. W. and Saucerman, J. (2014). Question asking during
collaborative problem solving in an online game environment. In In-
telligent Tutoring Systems, pp. 617-618. Springer International Pub-
lishing.
[19] Lindberg, D., Popowich, F., Nesbit, J. and Winne, P. (2013).
Generating natural language questions to support learning on-line.
In Proceedings of the 14th European Workshop NLG, pp. 105-114.
Association for Computational Linguistics.
[20] Mazidi, K. and Nielsen, R. D. (2014). Pedagogical evaluation of
automatically generated questions. In Intelligent Tutoring Systems,
pp. 294-299. Springer International Publishing.
[21] Nesbit, J.C. and Adesope, O. O. (2006) Learning with concept
and knowledge maps: A meta-analysis. In Review of Educational
Research, Vol. 76, No.3, pp. 413-448.
[22] Otero, J. (2009). Question generation and anomaly detection in
texts. Handbook of Metacognition in Education, pp. 47-59.
[23] Riley, M. (2000). Into the Key Stage 3 history garden: choosing
and planting your enquiry questions. In Teaching History. London:
Historical Association.
[24] Roth, W. M. (1996). Teacher questioning in an open-inquiry
learning environment: Interactions of context, content, and student
responses. Journal of Research in Science Teaching, Vol.33, No.7,
pp. 709-736.
[25] Seta, K., Noguchi, D. and Ikeda, M. (2011). Presentation-based
collaborative learning support system to facilitate meta-cognitively
aware learning communication. In The Journal of Information and
Systems in Education, Vol.9, No.1, pp.3-14.
[26] Seta, K. and Ikeda, M. (2011). Presentation based meta-learning
environment by facilitating thinking between lines: A model based
approach. In ToyohideWatanabe and Lakhmi C. Jain (Eds.): Innova-
tions in Intelligent Machines-2-Intelligent Paradigms and Applica-
tions, Studies in Computational Intelligence, Vol. 376, pp. 143-166.
Springer.
[27] Slotta, J. D. (2004). The web-based inquiry science environment
(WISE): Scaffolding knowledge integration in the science classroom.
In Internet Environments for Science Education, pp. 203-232.
[28] Smart, L. (1996). Using I.T. in Primary School History. London:
Cassell.
[29] SparkNotes Editors (accessed December 12, 2014). Spar-
kNote on World War I (1914-1919). SparkNotes LLC. 2005.
http://www.sparknotes.com/history/european/ww1/
[30] Stow, W. and Haydn, T. (2000). 7 Issues in the teaching of
chronology. In Issues in History Teaching, p. 83. Routledge.
[31] Thuering, M., Hannemann, J., and Haake, J. M. (1995). Hyper-
media and cognition: Designing for comprehension, Communication
of the ACM, Vol.38, No.8, pp. 57-66.
[32] Wolfe, J.H. (1976). Automatic question generation from text.
ACM SIGCUE Outlook 10(SI), pp. 104-112.
[33] Wyse, B. and Piwek, P. (2009). Generating questions from open-
learn study units. In AIED 2009 Workshop Proceedings Volume 1:
The 2nd Workshopon Question Generation.
[34] Zimmerman, B. J. and Schunk, D. H. (Eds.). (2001). Self-
Regulated Learning and Academic Achievement: Theoretical Per-
spectives. Routledge.
〔担当委員：川村 隆浩〕
Received February 09, 2015.
Author’s Profile
Corentin Jouault
Corentin Jouault received the M.E. in Computer Science
from EISTI, France in 2010. He is currently a Ph.D. can-
didate at the Graduate School of Science, Osaka Prefecture
University. His research interests include ontological engi-
neering, semantic web, artificial intelligence, intelligent tu-
toring systems and software engineering. He is a member of
APSCE.
Kazuhisa Seta (Member)
Kazuhisa Seta received a B.E. and M.E. from Ryukoku Uni-
versity in 1993 and 1995, respectively. He received a Ph.D.
from Osaka University in 1998. He is currently a profes-
sor in the College of Sustainable System Sciences and the
Graduate School of Science, Osaka Prefecture University.
His research interests include software engineering, intel-
ligent tutoring systems, human resource management, and
ontological engineering. He received a Best Paper Award
from the Japanese Society for Information and Systems in Education in 2012 and
2015. He is a member of IEICE, IPSJ, JSiSE, JCSS, APSCE, and IAIED.
Yuki Hayashi (Member)
YukiHayashi received his B.E., M.E., and Ph.D. from Nagoya
University in 2007, 2009, and 2012, respectively. From
2012 to 2014, he was an assistant professor in Seikei Uni-
versity. He is currently an assistant professor in the College
of Sustainable System Sciences, Osaka Prefecture Univer-
sity. His research interests include computer-supported col-
laborative learning and human-computer interaction. He is
a member of IPSJ, JSiSE, and HIS.
