Figure 4 - uploaded by Tao Li
Content may be subject to copyright.
Source publication
Assessing the difficulty of an exercise question is a subjective process and is usually done by the instructor based on experience. An accurate assessment of the difficulty of exercise and exam questions is important and will help to better allocate credits to assignments and exams. Our contribution is in defining a relatively objective approach to...
Context in source publication
Context 1
... the credit value in fact denotes the desired question difficulty. The difficulty of the automatically generated question may exceed the desired difficulty by a small per- centage. A good question generator should possess these three characteristics: domain consistency, correctness, and completeness of coverage. Domain consistent algorithms generate questions that produce results that are mostly contained in the specified empiri- cal value intervals. If this is the case for every node, we say the value domains are consistent. In other words, if we randomly choose, for those nodes in the input condi- tions, the values in the corresponding inter- val, we expect that the result value to be computed also lies in the specified interval of the destination node. Domain consistency depends on the experience of the system authors. A correct question generation algorithm must not generate questions which are not solvable based on the given concept graph. This is important because students often cannot determine whether or not a question is solvable and they may waste much time trying to solve an impossible problem. When the algorithm is able to generate questions that exercise every function of the DAG and relate every concept in the hierar- chy (in many questions), we say it has a complete coverage. Our automatic question generator will not generate any unsolvable question. In addi- tion, it provides domain consistency and completeness. This is a very important point and the users will not waste time to tackle an unsolvable problem and will not suffer from the frustration of not being able to solve a problem after devoting a significant amount of time. In addition, it also guaran- tees that the question generator will provide a wide variety of questions for the users. The screen shot of an automatically generated question is shown in Figure 3. One of the major hurdles in building a suc- cessful tutoring system is the diagnosis of mistakes made by students. It is extremely difficult to build a comprehensive knowledge base that can model unpredictable student behavior. A typical student model does not cover many unexpected behavior. In our opinion, it is too difficult or nearly impossible to build a student model that covers all the possible student behavior and relevant knowledge in problem solving. It is more convenient to provide the students with a set of tools (such as definitions and equations) that they can use in solving prob- lems of a particular subject. We call this sys- tem guided problem solving and diagnosis. Although this may occasionally limit the stu- dent’s creativity, the pay-off is significant the system can diagnose more mistakes a student makes during problem solving and provide more sensible advice to the student. Our guided problem solving algorithm util- izes the data structure generated by the automatic question generator. It is essen- tially the reverse of the question generation process. The guided learning process is closely coupled with the graphics interface. After a question is automatically generated, we keep the data structure and the OPEN queue. If the user chooses to enter guided learning mode, the system initializes the user interface so that the equations for the nodes on the OPEN queue are visible and other equations are not visible. This helps to narrow down the search range. The user may select one equation and put an answer in there. If the result of calcula- tion is correct, the system will fill the intermediate result in a table. This helps the user to remeber intermediate values. The node is then eliminated from the OPEN queue and the corresponding equation, if not used by any other node in OPEN, becomes invisible. When the user enters an incorrect interme- diate result, the system will inform the user about the mistake. This process continues until all the nodes are removed from the OPEN queue. At this time, a correct final result is arrived at. A screen shot of a step in guided problem is shown in Figure 4. An objective method for assessing the diffi- culty of automatically generated questions is presented here. This method has been ap- plied to implement a novel intelligent tutor- ing system for computer architecture learning. Our system is able to automatically gen- erate a large variety of questions from the knowledge base. This approach is applicable to tutoring many subjects in science and engineering. We designed robust algorithms for automatic question generation in statically structured systems. We also designed a system guided learning approach that is closely attached to the automatic question generator. Part of the knowledge structure has been imple- mented. The system with a partial knowl- edge is working and it demonstrates the fea- sibility of our approach. We are formulating an approach to assess the difficulty of questions that require dy- namical application of many operators, and we will apply the approach to solving difficult questions in geometry. The authors wish to acknowledge the effort of our students for implementing the com- puter architecture tutoring system. Frasson, C., G. Gauthier and A. Lesgold (eds.) (1996). Intelligent Tutoring Sys- tems , LNCS-1086, 3 rd International Con- ference on Intelligent Tutoring Systems (ITS’96), Montreal, Canada, ...
Citations
... Approaches to estimation of difficulty level are proposed in literature in the contexts of intelligent tutoring systems (ITS), computer adaptive tests (CAT) and class room assessments. Difficulty is estimated dynamically for ITS by considering the performance of individual students in an item (Jiu and Park, 2004;Hatzilygeroudis et al., 2006;Li and Sambasivam, 2003). Difficulty level of an item is adjusted in CAT to the students ability level (Karahoca, Karahoca, and Ince, 2009;Kunichika et al,. ...
... An item becomes more difficult as more conditions have to be satisfied to reach a potential solution. More conditions associated with an item results in a more difficult item (Li and Sambasivam, 2003;Fisher-Hoch and Hughes, 1996;Lumley et al., 2012;Katz, Lipps, and Trafton, 2002). ...
The quality of assessment determines the quality of learning, and is characterized by validity, reliability and difficulty. Mastery of learning is generally represented by the difficulty levels of assessment items. A very large number of variables are identified in the literature to measure the difficulty level. These variables, which are not completely independent of one another, are categorized into learner dependent, learner independent, generic, non-generic and score based. This research proposes a model for predicting the difficulty level of assessment items in engineering courses using learner independent and generic variables. An ordinal regression model is developed for predicting the difficulty level, and uses six variables including three stimuli variables (item presentation, usage of technical notations and number of resources), two content related variables (number of concepts and procedures) and one task variable (number of conditions). Experimental results from three engineering courses provide around 80% accuracy in classification of items using the proposed model.
... Generating data and questions automatically that satisfy a given set of constraints has been actively investigated in diverse contexts. [10][11][12][13][14] A taxonomy of questions for question generation is presented in. 16 In the following, we provide a brief description of approaches to automated question generation. ...
Structured Query Language (SQL) is an ANSI and ISO standard declarative query language for querying and manipulating relational databases. It is easy to write SQL queries but very difficult to validate them. Often students conclude that a SQL query is correct simply because the query compiles, executes, and fetches data. Therefore, it is crucial that SQL assessment tasks are carefully designed and implemented to ensure a deep learning experience for students. In this paper, we propose an approach to automatically generate SQL queries for assessing students' SQL learning. SQL concepts are modeled using RDFS. The user can select SQL concepts to be included in an assessment and our approach will generate appropriate queries. The proposed approach is generic and is database metadata driven. A Web-based prototype system is developed to illustrate the effectiveness of the proposed approach.
... Authentic contexts [33] Exploration contexts [34] Question generation [35] User modeling [36] Organization of gestalt [37] Pleasure [38] Teachers ...
The plethora of different subfields in intelligent tutoring systems (ITS) are often difficult to integrate theoretically when
analyzing how to design an intelligent tutor. Important principles of design are claimed by many subfields, including but
not limited to: design, human-computer interaction, perceptual psychology, cognitive psychology, affective and motivation
psychology, statistics, artificial intelligence, cognitive neuroscience, constructivist and situated cognition theories. Because
these theories and methods sometimes address the same grain size and sometimes different grain sizes they may or may not conflict
or be compatible and this has implications for ITS design. These issues of theoretical synthesis also have implications for
the experimentation that is used by our various subfields to establish principles. Because our proposal allows the combination
of multiple perspectives, it becomes apparent that the current “forward selection” method of theoretical progress might be
limited. An alternative “backward elimination” experimental method is explained. Finally, we provide examples to illustrate
how to build the bridges we propose.
... The systems which involve the tutor in the task of creating a set of questions to be stored in a repository prove to be more reliable and consequently are used more for official exams, in order to obtain an objective assessment. Those systems, such as the ones described in (Li & Sambasivam, 2003) and in (Lister & Jerram, 2001), sometimes use an XML test configuration file to define some rules for the question selection. In question repository based systems; the challenge is to give a good organization to the repository, to avoid question replication, and to use a good question selection procedure in order to assess learners' skills on the desired subjects. ...
... As for a means for sequencing and control access to the tests, none of the tools analyzed has a flexible system. The system described in (Li & Sambasivam., 2003) permits the learner to sit an exam many times, until a minimum acceptable score is achieved. In (McGough et al., 2001) the questions are grouped into sets, and the strategy to pass a set, and consequently access to the next, is to give the correct response to 3 answers in a row for that set. ...
On-Line Testing is that sector of e-learning aimed at assessing learner's knowledge through e-learning means. In on-line testing, due to the necessity of evaluating a big mass of learners in strict times, the means for knowledge evaluation had to evolve to satisfy the new necessities: objective tests, more rapidly assessable, started gaining more credence in the determination of learners' results. In this paper, we present an On-Line Testing system, named eWorkbook, which can be used for evaluating learner's knowledge by creating (the tutor) and taking (the learner) on-line tests based on multiple choice question type. Its use is suitable within the academic environment in a blended learning approach, by providing tutors with an additional assessment tool, and learners with a distance self-assessment means. Among other features, eWorkbook can record and visualize, in a suitable graphical format, learner's interactions with the system interface during the test session. This is valuable information for understanding the learner's behaviour when taking a test. In particular, the graphical analysis of the test outcomes has helped us in the discovery of several strategies employed by the learners to perform the test. In the paper, the main characteristics of the system are presented together with a rationale behind them and an outline of the architectural design of the system.
... The systems which involve the tutor in the task of creating a set of questions to be stored in a database , prove to be more reliable and consequently are used more for official exams, in order to obtain an objective assessment. Those systems, such as the ones described in (Li & Sambasivam, 2003) and in (Lister & Jerram, 2001), sometimes use an XML test configuration file to define some rules for the question selection. In question database based systems, the challenge is to give a good organization to the database, to avoid question replication, and to use a good question selection procedure in order to assess learners' skills on the desired subjects. ...
... As for a means for sequencing and control access to the tests, none of the tools analyzed has a flexible system. The system described in (Li & Sambasivam., 2003) permits the learner to sit an exam many times, until a minimum acceptable score is achieved. In (McGough et al., 2001) the questions are grouped into sets, and the strategy to pass a set, and consequently access to the next, is to give the correct response to 3 answers in a row for that set. ...
Computer aided assessment (CAA) tools are more and more widely adopted in academic environments mixed to other assessment means. In this article, we present a CAA Web application, named eWorkbook, which can be used for evaluating learner's knowledge by creating (the tutor) and taking (the learner) on-line tests based on multiple choice, multiple response and true/false question types. Its use is suitable within the academic environment in a blended learning approach, by providing tutors with an additional assessment tool, and learners with a distance self-assessment means. In the article, the main characteristics of the tool are presented together with a rationale behind them and an outline of the architectural design of the system.
... In this backdrop, in this paper we present an interesting research which now attempts qualitative assessment of problem with a particular objective towards facilitating automatic test composition. It is interesting to note, that there have been previous attempts to quantify the complexity of problems [2] [3] [4] though not from automatic design perspective. Some tried to figure out cognition based solution to the " question complexity " problem and understanding student view of question complexity [1] [2]. ...
Testing and evaluation is an integral part of the learning process. Educators have often tried to device methods for design of test-ware. Intelligent design and compilation of test-ware is a very interesting problem with immense applications. This research aims at automating the process of intelligent design of test-ware by providing qualitative assessment of questions. In this attempt, we provide some synthetic parameters for the evaluation of question in its concept space. The parameters are tested in some real world scenarios and intuitive inferences are deduced predicting the performance of the parameters. It is observed that the difficulty of a question is often a function of the concepts it tests. Concept knowledge can be represented in the form of linked concepts in semantic nets, the links representing the relationships between the concepts. If this directed graph is known, complexity of a question can be computed by synthetic means.
... Intelligent Tutoring Systems can statistically model understanding the concepts in the learning domain. Several methods have been proposed to determine the hardness of questions (Khan et al., 2003, Kunichika et al., 2002, Li & Sambasivam, 2003). Hwang (2003) introduced the " Concept Effect Graphs " where the subject materials can be viewed as a tree diagram comprising chapters, sections, sub-sections and key concepts to be learned. ...
... Although some studies are available in literature, we suggest a new paradigm to determine the level of hardness for a question (Khan et al., 2003, Kunichika et al., 2002, Li & Sambasivam, 2003). As seen in Question Difficulty Rate Table, any question has a value in interval (0, 1] initially. ...
The object of this study is to model the level of a question difficulty by a differential equation at a pre-specified domain knowledge, to be used in an educational support system. For this purpose, we have developed an intelligent tutoring system for mathematics education. Intelligent Tutoring Systems are computer systems designed for improvement of learning and teaching processes in the domain knowledge. The developed system, which is called as MathITS, is based on conceptual map modeling.The Mathematica Kernel is used as an expert system and knowledge representation is based on LaTeX notation in MathITS.
... Some researchers produced results (Soldatova & Mizoguchi, 2003; Kunichika, Katayama, Hirashima, & Takeuchi, 2003) using approaches not based on conceptual structures. In a previous paper (Li & Sambasivam, 2003 ), we presented research results on automatic question generation and difficulty assessment for intelligent tutoring. That approach was successfully applied to the computer architecture course using a quantitative hierarchy. ...
In our previous research, we investigated the automatic generation of questions with single vari- able and the application to computer architecture teaching. In the current research, we extend the previous approach to generating questions with multiple variables on Directed Acyclic Graph (DAG) knowledge structures. Questions generated with the new algorithm are more complex and require more mathematical skills to solve. The algorithms can be applied to any discipline for which the conceptual and analytical knowledge can be represented by a DAG.
A course in Engineering programs is designed and conducted to facilitate students to acquire a set of competencies to meet a subset of Program Outcomes(Graduate Attributes) identified by the accreditation agency. Formative and summative assessments, if in alignment with the competencies, enable the instructor to guide the students to learn well, and to measure the level of attainment of competencies. The difficulty levels of the assessment items are a measure of mastery of the competencies. Well defined difficulty rubrics provide guidelines to the instructor to prepare assessment items with stated difficulty levels. Several difficulty rubrics which are learner dependent or independent, subject specific or non-specific and score-based or score-independent to characterize the difficulty level are reported in the literature. This paper presents a set of learner-independent and score-independent difficulty rubrics. The parameters that characterize these rubrics are arranged in a hierarchy. The nodal parameters are content, task and structure (stimulus) of assessment items. The leaf and atomic parameters are identified in the framework of Anderson-Bloom taxonomy. The parameters for a given assessment item are identified and difficulty is determined by applying weight based and fuzzy logic rule based methods. The efficacy of the proposed item difficulty rubrics is validated by designing different difficulty level items for the course Basics of Digital Systems, and validating the difficulty of the designed items by the subject matter experts.
