No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Using Pattern Matching to Assess Gameplay
Abstract
In this chapter we describe Analysis of Patterns in Time (APT) and how it can be used to analyze gameplay choices to provide evidence of a play-learner’s understanding of concepts modeled in a game. APT is an empirical approach to observing and coding phenomena as mutually exclusive and exhaustive categories within classifications. These data form a temporal map of joint and sequential patterns. We examine the case of the online Diffusion Simulation Game. An algorithm calculates scores for gameplay data patterns and compares them with scores for patterns based on optimal strategies derived from the game’s conceptual model. We discuss the results of using APT for analysis of game sessions for three play-learners. We describe how APT can be included as part of a serious game to conduct formative assessment and determine appropriate hints, coaching, or other forms of scaffolding during gameplay. We conclude by discussing APT methods for summative assessment.
... The praxiological study described in this article is a form of summative evaluation (Scriven, 1967;Worthen & Sanders, 1987). We used a method to verify praxiological theory called Analysis of Patterns in Time (APT; Frick, 1990;Frick & Reigeluth, 1992;Frick et al., 2022;Myers & Frick, 2015). ...
... These are linear, additive models based on an algebraic function for a line in a Cartesian coordinate system (see Kirk, 1999Kirk, , 2013Tabachnick & Fidell, 2018). However, Frick (1983Frick ( , 1990 clearly demonstrated empirically and proved mathematically that stochastic educational relationships cannot be verified as precisely by the LMA when compared to APT. Myers and Frick (2015) characterized the difference as follows: The LMA relates measures, whereas APT measures relations. This is not wordplay, but a fundamental difference in approach to measurement and analysis of stochastic relations. ...
... In the LMA, the strength of a relationship is often characterized by percent of variance accounted for in the independent variable by one or more dependent variables (e.g., see Kirk, 1999Kirk, , 2013Tabachnick & Fidell, 2018). The LMA is appropriate for verification of deterministic relations among separate variables or factors (Frick, 1983;Myers & Frick, 2015). ...
In this naturalistic design-research study, we tracked 172,417 learning journeys of students who were interacting with an online resource, the Indiana University Plagiarism Tutorials and Tests (IPTAT) at https://plagiarism.iu.edu. IPTAT was designed using First Principles of Instruction (FPI; Merrill in Educ Technol Res Dev 50:43–59, 2002, https://doi.org/10.1007/BF02505024; First principles of instruction: identifying and designing effective, efficient, and engaging instruction, Wiley, New York, 2013; M. David Merrill’s First Principles of Instruction, Association for Educational Communications and Technology, Washington, 2020). Students who used IPTAT were mostly from university and advanced high school courses taught in 186 countries and territories. Instructors expected their students to pass one of trillions of difficult Certification Tests (CT) provided through IPTAT. Each CT assessed student ability to classify samples of writing as word-for-word plagiarism, paraphrasing plagiarism, or no plagiarism—when given original source materials. In 51,646 successful learning journeys, students who were initially nonmasters and who later achieved mastery had viewed, on average, 89 IPTAT tutorial webpages designed with FPI. In the 23,307 unsuccessful learning journeys, students who were nonmasters and who had not (yet) achieved mastery had viewed an average of 52 tutorial webpages designed with FPI. Analysis of Patterns in Time (Frick in American Educational Research Journal 27:180–204, 1990) and Bayesian analysis revealed that students were nearly 4 times more likely to pass a CT when they selected one or more parts of IPTAT instruction designed with FPI. These results support the instrumental value of First Principles of Instruction for design of online learning in a massive, open, online course (MOOC). These findings further demonstrate the value of an innovative approach to modern learning analytics, Analysis of Patterns in Time, when coupled with Bayesian reasoning.
... This study will be using a research method called Analysis of Patterns in Time (APT-Frick 1990;Frick et al. 2008;Myers and Frick 2015) in order to study instructional effectiveness. Students are in control of which parts of the IPTAT they undertake and complete. ...
... In MAPSAT, relations themselves are empirically observed and coded. MAPSAT was invented decades ago, and is well suited as a methodology for modern learning analytics, as well as for many other kinds of research (Frick 1990;Myers and Frick 2015). For example, Google Analytics utilizes a variation of APT methods in its ''behavior flow'' reports; however, temporal maps cannot be queried as in full APT. ...
... We are further developing software that will track each student's use and store it as a temporal map. See Myers and Frick (2015) for examples of how we have used temporal maps with APT in studying play-learner usage of the online Diffusion Simulation Game. ...
We illustrate a very recent research study that demonstrates the value of Massive Open Online Courses (MOOCs) as vehicles for research. We describe the development of the Indiana University Plagiarism Tutorials and Tests (IPTAT). Our new design has been guided by First Principles of Instruction: authentic problems, activation, demonstration, application, and integration. We further discuss our data collection mechanisms and early usage of this new mini-MOOC. In the first study, we investigated a built-in assessment feature for students to evaluate instructional quality and user experience. To do this, we adapted scales from the Teaching and Learning Quality instrument. As a follow-up study, we plan to further investigate patterns of usage of the IPTAT by students through creation of individual temporal maps. We plan to use Analysis of Patterns in Time, a method that provides learning analytics.
... Some of the skills that can be assessed based on gameplay data are teamwork ability [19,21,54], language proficiency [19,54,56], financial investment skills [19], math fluency [24,51,54,56], ICT skills [54,57], creative problem solving [45,51], spatial navigation [58], fine motor skills [51], metacognition and systems thinking [51], memory retention [10,45,59], cultural knowledge [41], and the understanding of specific science concepts, such as Newtonian mechanics [11,60]. Approaches for game-based assessment can also allow to track a user's cognitive development and learning trajectories over time [51,61,62,63] and to examine specific gaps in knowledge [46,63] or learning di culties, such as reading problems and dyscalculia [61,64]. ...
... Some of the skills that can be assessed based on gameplay data are teamwork ability [19,21,54], language proficiency [19,54,56], financial investment skills [19], math fluency [24,51,54,56], ICT skills [54,57], creative problem solving [45,51], spatial navigation [58], fine motor skills [51], metacognition and systems thinking [51], memory retention [10,45,59], cultural knowledge [41], and the understanding of specific science concepts, such as Newtonian mechanics [11,60]. Approaches for game-based assessment can also allow to track a user's cognitive development and learning trajectories over time [51,61,62,63] and to examine specific gaps in knowledge [46,63] or learning di culties, such as reading problems and dyscalculia [61,64]. ...
With many million users across all age groups and income levels, video games have become the world’s leading entertainment industry. Behind the fun experience they provide, it goes largely unnoticed that modern game devices pose a serious threat to consumer privacy. To illustrate the industry’s potential for illegitimate surveillance and user profiling, this article offers a classification of data types commonly gathered by video games. Drawing from patents and literature of diverse disciplines, it is also discussed how patterns and correlations in collected gameplay data may leak additional information in ways not easily understood or anticipated by the user. This includes inferences about a user’s biometric identity, age, gender, emotions, skills, interests, consumption habits, and personality traits. Based on these findings, it is argued that video games need to be brought into the focus of privacy research and discourse. Considering the granularity and enormous scale of the data collection taking place, this industry deserves the same level of scrutiny as other digital services, such as search engines, dating apps, or social media platforms. The knowledge compiled in this article can serve as a basis for privacy impact assessments, consumer education, and further research into the societal impact of video games.
... Some of the skills that can be assessed based on gameplay data are teamwork ability [19,21,54], language proficiency [19,54,56], financial investment skills [19], math fluency [24,51,54,56], ICT skills [54,57], creative problem solving [45,51], spatial navigation [58], fine motor skills [51], metacognition and systems thinking [51], memory retention [10,45,59], cultural knowledge [41], and the understanding of specific science concepts, such as Newtonian mechanics [11,60]. Approaches for game-based assessment can also allow to track a user's cognitive development and learning trajectories over time [51,61,62,63] and to examine specific gaps in knowledge [46,63] or learning difficulties, such as reading problems and dyscalculia [61,64]. ...
... Some of the skills that can be assessed based on gameplay data are teamwork ability [19,21,54], language proficiency [19,54,56], financial investment skills [19], math fluency [24,51,54,56], ICT skills [54,57], creative problem solving [45,51], spatial navigation [58], fine motor skills [51], metacognition and systems thinking [51], memory retention [10,45,59], cultural knowledge [41], and the understanding of specific science concepts, such as Newtonian mechanics [11,60]. Approaches for game-based assessment can also allow to track a user's cognitive development and learning trajectories over time [51,61,62,63] and to examine specific gaps in knowledge [46,63] or learning difficulties, such as reading problems and dyscalculia [61,64]. ...
With many million users across all age groups and income levels, video games have become the world's leading entertainment industry. Behind the fun experience they provide, it goes largely unnoticed that modern game devices pose a serious threat to consumer privacy. To illustrate the industry's potential for illegitimate surveillance and user profiling, this paper offers a classification of data types commonly gathered by video games. Drawing from patents and literature of diverse disciplines, we also discuss how patterns and correlations in collected gameplay data may leak additional information in ways not easily understood or anticipated by the user. This includes inferences about a user's biometric identity, age and gender, emotions, skills, interests, consumption habits, and personality traits. Based on these findings, we argue that video games need to be brought into the focus of privacy research and discourse. Considering the granularity and enormous scale of the data collection taking place, this industry deserves the same level of scrutiny as other digital services, such as search engines , dating apps, or social media platforms. The knowledge compiled in this paper can serve as a basis for privacy impact assessments, consumer education, and further research into the societal impact of video games.
... ■ GA4 when supplemented with Excel can do some kinds of APT as envisioned originally by Frick (1983Frick ( , 1990) and Myers and Frick (2015). ...
Slides used in the presentation I made to the 4th Annual Learning Analytics Summit. I cover about 50 years of my quest to do learning analytics. And I do this rather quickly, in about 25 minutes (35 slides). The blue underline links in the slides go to various resources that you can explore in further depth.
... APT is a proven methodology that differs from traditional qualitative and quantitative approaches to measurement (Frick, 1983(Frick, , 1990Frick & Dagli, 2016;Myers & Frick, 2015). ...
If you were an investor, and you could choose between a company that is not likely to use effective business strategies and a company that is three to five times more likely to use them, where would you put your money? Likewise, if you could use methods of education that were three to five times more likely to help students succeed in their learning journeys, would you?
We document the extraordinary effectiveness of First Principles of Instruction for promoting online learning. We used these principles to design the online Indiana University Plagiarism Tutorials and Tests. Analysis of Patterns in Time was the primary research methodology for evaluating this MOOC in 2019 and 2020. We used APT to segment nearly 1.87 million temporal maps and to match learning patterns by leveraging Google Analytics tracking and reporting tools. We then created spreadsheet formulas to compute APT likelihood ratios from the GA results. We found that successful students worldwide were nearly four times as likely to select instruction designed with First Principles, when compared to unsuccessful learners.
Analysis of Patterns in Time can be used as a practical way to evaluate instructional effectiveness. Applying APT to do learning analytics means big data can be harnessed to evaluate online teaching and learning. APT is a powerful method for finding meaningful patterns in massive datasets.
Innovative Learning Analytics for Evaluating Instruction covers the application of a forward-thinking research methodology that uses big data to evaluate the effectiveness of online instruction. Analysis of Patterns in Time (APT) is a practical analytic approach that finds meaningful patterns in massive data sets, capturing temporal maps of students’ learning journeys by combining qualitative and quantitative methods. Offering conceptual and research overviews, design principles, historical examples, and more, this book demonstrates how APT can yield strong, easily generalizable empirical evidence through big data; help students succeed in their learning journeys; and document the extraordinary effectiveness of First Principles of Instruction. It is an ideal resource for faculty and professionals in instructional design, learning engineering, online learning, program evaluation, and research methods.
In this dissertation (2012), analysis of patterns in time (APT) was used to compare gameplay results from the Diffusion Simulation Game (DSG) with predictions based on diffusion of innovations theory (DOI). A program was written to automatically play the DSG by analyzing the game state during each turn, seeking patterns of game component attributes that matched optimal strategies based on DOI theory. When the use of optimal strategies did not result in the desired number of successful games, here defined as the threshold of confidence for model verification, further investigation revealed flaws in the computational model. These flaws were incrementally corrected and subsequent gameplay results were analyzed until the threshold of confidence was achieved.
In addition to analysis of patterns in time for model verification (APTMV), other verification methods used included code walkthrough, execution tracing, desk checking, syntax checking, and statistical analysis. The APTMV method was found to be complementary to these other methods, providing quantified evidence of the computational model's degree of accuracy and pinpointing flaws that could be corrected to improve fidelity. The APTMV approach to verification and improvement of computational models is described and compared with other methods, and improvements to the process are proposed.
Analysis of patterns in time (APT) is a method for gathering information about observable phenomena such thatprobabilities of temporal patterns of events can be estimated empirically. If appropriate sampling strategies are employed, temporal patterns can be predicted from APT results. As an example of the fruitfulness of APT, it was discovered in a classroom observational study that elementary students were on task 97% of the time if some form of direct instruction was occurring also, whereas they were on task only 57% of the time during nondirect instruction. As a second example, APT results were used as a rule base for an expert system in adaptive computer-based testing. When two different computer tests were studied, average samples of 9 and 13 test items were required to make mastery and nonmastery decisions when items were selected at random. These decisions were, respectively, 94% and 98% accurate compared to those reachedfrom two much larger test item pools. Finally, APT is compared to the linear models approach and event history analysis. The major difference is that in APT there is no mathematical model assumed to characterize relations among variables. In APT the model is the temporal pattern being investigated.
Focuses on prescriptions for designing the instructional overlay of computer-based simulations, which serves to optimize learning and motivation. The instructional functions of simulations and the instructional features that should be used to achieve these functions are described. The design of computer-based simulations is presented in the form of a general model that offers prescriptions for the design of the introduction, acquisition, application, and assessment stages of simulations and for dealing with the issue of control (system or learner). Variations on the general model are based on the nature of the behavior (using procedures, process principles, or causal principles), complexity of the content, form of learner participation, form of changes being simulated (physical or nonphysical), and motivational requirements. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
This article discusses the model of the diffusion of innovations and describes this model’s key variables. Diffusion is the process by which an innovation is communicated through certain channels over time among the members of a social system. An innovation is an idea, practice, or object that is perceived as new by an individual or other unit of adoption.
Probabilistic graphical models and decision graphs are powerful modeling tools for reasoning and decision making under uncertainty. As modeling languages they allow a natural specification of problem domains with inherent uncertainty, and from a computational perspective they support efficient algorithms for automatic construction and query answering. This includes belief updating, finding the most probable explanation for the observed evidence, detecting conflicts in the evidence entered into the network, determining optimal strategies, analyzing for relevance, and performing sensitivity analysis. The book introduces probabilistic graphical models and decision graphs, including Bayesian networks and influence diagrams. The reader is introduced to the two types of frameworks through examples and exercises, which also instruct the reader on how to build these models. The book is a new edition of Bayesian Networks and Decision Graphs by Finn V. Jensen. The new edition is structured into two parts. The first part focuses on probabilistic graphical models. Compared with the previous book, the new edition also includes a thorough description of recent extensions to the Bayesian network modeling language, advances in exact and approximate belief updating algorithms, and methods for learning both the structure and the parameters of a Bayesian network. The second part deals with decision graphs, and in addition to the frameworks described in the previous edition, it also introduces Markov decision processes and partially ordered decision problems. The authors also provide a well-founded practical introduction to Bayesian networks, object-oriented Bayesian networks, decision trees, influence diagrams (and variants hereof), and Markov decision processes. give practical advice on the construction of Bayesian networks, decision trees, and influence diagrams from domain knowledge. give several examples and exercises exploiting computer systems for dealing with Bayesian networks and decision graphs. present a thorough introduction to state-of-the-art solution and analysis algorithms. The book is intended as a textbook, but it can also be used for self-study and as a reference book.
After more than four decades, development of artificially intelligent tutoring systems has been constrained by two interrelated problems: knowledge representation and natural language understanding. G. S. Maccia's epistemology of intelligent natural systems implies that computer systems will need to develop qualitative intelligence before these problems can be solved. Recent research on how human nervous systems develop provides evidence for the significance of qualitative intelligence. Qualitative intelligence is required for understanding of culturally bound meanings of signs used in communication among intelligent natural systems. S. I. Greenspan provides neurological and clinical evidence that emotion and sensation are vital to the growth of mind-capabilities that computer systems do not currently possess. Therefore, we must view computers in education as media through which a multitude of teachers can convey their messages. This does not mean that the role of classroom teachers is diminished. Teachers and students can be empowered by these additional learning resources.
Getting an innovation adopted is difficult; a common problem is increasing the rate of its diffusion. Diffusion is the communication of an innovation through certain channels over time among members of a social system. It is a communication whose messages are concerned with new ideas; it is a process where participants create and share information to achieve a mutual understanding. Initial chapters of the book discuss the history of diffusion research, some major criticisms of diffusion research, and the meta-research procedures used in the book. This text is the third edition of this well-respected work. The first edition was published in 1962, and the fifth edition in 2003. The book's theoretical framework relies on the concepts of information and uncertainty. Uncertainty is the degree to which alternatives are perceived with respect to an event and the relative probabilities of these alternatives; uncertainty implies a lack of predictability and motivates an individual to seek information. A technological innovation embodies information, thus reducing uncertainty. Information affects uncertainty in a situation where a choice exists among alternatives; information about a technological innovation can be software information or innovation-evaluation information. An innovation is an idea, practice, or object that is perceived as new by an individual or an other unit of adoption; innovation presents an individual or organization with a new alternative(s) or new means of solving problems. Whether new alternatives are superior is not precisely known by problem solvers. Thus people seek new information. Information about new ideas is exchanged through a process of convergence involving interpersonal networks. Thus, diffusion of innovations is a social process that communicates perceived information about a new idea; it produces an alteration in the structure and function of a social system, producing social consequences. Diffusion has four elements: (1) an innovation that is perceived as new, (2) communication channels, (3) time, and (4) a social system (members jointly solving to accomplish a common goal). Diffusion systems can be centralized or decentralized. The innovation-development process has five steps passing from recognition of a need, through R&D, commercialization, diffusions and adoption, to consequences. Time enters the diffusion process in three ways: (1) innovation-decision process, (2) innovativeness, and (3) rate of the innovation's adoption. The innovation-decision process is an information-seeking and information-processing activity that motivates an individual to reduce uncertainty about the (dis)advantages of the innovation. There are five steps in the process: (1) knowledge for an adoption/rejection/implementation decision; (2) persuasion to form an attitude, (3) decision, (4) implementation, and (5) confirmation (reinforcement or rejection). Innovations can also be re-invented (changed or modified) by the user. The innovation-decision period is the time required to pass through the innovation-decision process. Rates of adoption of an innovation depend on (and can be predicted by) how its characteristics are perceived in terms of relative advantage, compatibility, complexity, trialability, and observability. The diffusion effect is the increasing, cumulative pressure from interpersonal networks to adopt (or reject) an innovation. Overadoption is an innovation's adoption when experts suggest its rejection. Diffusion networks convey innovation-evaluation information to decrease uncertainty about an idea's use. The heart of the diffusion process is the modeling and imitation by potential adopters of their network partners who have adopted already. Change agents influence innovation decisions in a direction deemed desirable. Opinion leadership is the degree individuals influence others' attitudes
An introduction to the diversity of methods and areas of application of mathematics in psychology. Harvard Book List (edited) 1971 #88 (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Nonmetric temporal path analysis (NTPA): An alternative to the linear models approach for verification of stochastic educational relations
- T Frick
Frick, T. (1983). Nonmetric temporal path analysis (NTPA): An alternative to the linear
models approach for verification of stochastic educational relations. Unpublished doctoral
dissertation, Indiana University, Bloomington, IN USA. Retrieved from
http://www.indiana.edu/~tedfrick/ntpa/
A Web simulation on educational change: Challenges and solutions for development. Paper presented at the meeting of the Association for Educational Communication and Technology
- T Frick
- K.-J Kim
- B Ludwig
- R Huang
Frick, T., Kim, K.-J., Ludwig, B. and Huang, R. (2003). A Web simulation on educational
change: Challenges and solutions for development. Paper presented at the meeting of the
Association for Educational Communication and Technology, Anaheim, CA. Retrieved
from http://www.indiana.edu/~tedfrick/aect2003/frick_kim_ludwig_huang.pdf
Development of educational theory derived from three educational theory models (final report, project no. 5-0638)
- E S Maccia
- G Maccia
Maccia, E. S. & Maccia, G. (1966). Development of educational theory derived from three
educational theory models. Washington, DC: Final Report, Project No. 5-0638, U.S.
Department of Health, Education, and Welfare.
New ways to measure systemic change: Map & analyze patterns & structures across time (MAPSAT) Featured research paper presented at the annual conference of the Association for Retrieved from https
- T Frick
- R Myers
- K Thompson
- S York
Measuring effectiveness of instructional games and simulations: Pattern analysis of play in the Diffusion Simulation Game
- R D Myers
- T W Frick
Myers, R. D., & Frick, T. W. (in preparation). Measuring effectiveness of instructional games
and simulations:
Pattern analysis of play in the Diffusion Simulation Game.
ATIS graph theory. Columbus, OH: System Predictive Technologies
- K Thompson
Thompson, K. (2008). ATIS graph theory. Columbus, OH: System Predictive Technologies.
Retrieved from http://www.indiana.edu/~aptfrick/reports/11ATISgraphtheory.pdf
Educational evaluation: Alternative approaches and practical guidelines
- B Worthen
- J Sanders
Worthen, B., & Sanders, J. (1987). Educational evaluation: Alternative approaches and
practical guidelines. New York, NY: Longman.
The methodology of evaluation
- E M Rogers
Rogers, E. M. (2003). Diffusion of innovations. (5th ed.). New York, NY: Simon & Schuster.
Scriven, M. (1967). The methodology of evaluation. In AERA monograph series on
curriculum evaluation, No. 1 (pp. 39-89). Chicago, IL: Rand McNally.
Analyzing interaction patterns to verify a simulation/game model Unpublished doctoral dissertation Retrieved from http Measuring effectiveness of instructional games and simulations: Pattern analysis of play in the Diffusion Simulation Game
- R D Myers
Myers, R. D. (2012). Analyzing interaction patterns to verify a simulation/game model.
Unpublished doctoral dissertation, Indiana University, Bloomington, IN USA. Retrieved
from http://webgrok.com/papers/RodneyMyers_DissertationFinal_Approved.pdf
Myers, R. D., & Frick, T. W. (in preparation). Measuring effectiveness of instructional games
and simulations:
Pattern analysis of play in the Diffusion Simulation Game.
Featured research paper presented at the annual conference of the Association for Educational Communications & Technology
- T Frick
- R Myers
- K Thompson
- S York
Frick, T., Myers, R., Thompson, K., & York, S. (2008). New ways to measure systemic
change: Map & Analyze Patterns & Structures Across Time (MAPSAT). Featured research
paper presented at the annual conference of the Association for Educational
Communications
&
Technology,
Orlando,
FL.
Retrieved
from
https://www.indiana.edu/~tedfrick/MAPSATAECTOrlando2008.pdf