Fig 6 - uploaded by Aidan Delaney
Content may be subject to copyright.
Source publication
The least cognitively demanding way to create a diagram is to draw it with a pen. Yet there is also a need for more formal visualizations, that is, diagrams created using both traditional keyboard andmouse interaction. Our objective is to allow the creation of diagrams using traditional and stylus-based input. Having two diagram creation interfaces...
Context in source publication
Context 1
... recreate an actual (formal) straight line in the formal view between these two points. There may be some fine tuning required, depending on the visual language. For instance, if the line is intended to connect two items together then some adjustment to the length of the line or position of the end points may be required. This is illustrated in Fig. 6, where a line has been sketched between two rectangles, in an incomplete class diagram, that will form part of an arrow between them, representing an association. The end points of the line do not perfectly meet the rectangles. One end of the line needs to be trimmed and the other extended. The result is shown in the formal diagram. ...
Citations
... Applying linear classifiers to these gesture sets, Rubine reports recognition rates of over 96%, even for relatively small training set sizes of 15 samples per class. Rubine's feature set has been successfully applied in pen-based intelligent user interfaces (Stapleton et al., 2015;Williford et al., 2020), multitouch gesture recognition (Cirelli and Nakamura, 2014;Rekik et al., 2014) and even eye-tracking analysis (Çagla Çg and Metin Sezgin, 2015;Alamudun et al., 2017). ...
Digital pen features model characteristics of sketches and user behavior, and can be used for various supervised machine learning (ML) applications, such as multi-stroke sketch recognition and user modeling. In this work, we use a state-of-the-art set of more than 170 digital pen features, which we implement and make publicly available. The feature set is evaluated in the use case of analyzing paper-pencil-based neurocognitive assessments in the medical domain. Most cognitive assessments, for dementia screening for example, are conducted with a pen on normal paper. We record these tests with a digital pen as part of a new interactive cognitive assessment tool with automatic analysis of pen input. The physician can, first, observe the sketching process in real-time on a mobile tablet, e.g., in telemedicine settings or to follow Covid-19 distancing regulations. Second, the results of an automatic test analysis are presented to the physician in real-time, thereby reducing manual scoring effort and producing objective reports. As part of our evaluation we examine how accurately different feature-based, supervised ML models can automatically score cognitive tests, with and without semantic content analysis. A series of ML-based sketch recognition experiments is conducted, evaluating 10 modern off-the-shelf ML classifiers (i.e., SVMs, Deep Learning, etc.) on a sketch data set which we recorded with 40 subjects from a geriatrics daycare clinic. In addition, an automated ML approach (AutoML) is explored for fine-tuning and optimizing classification performance on the data set, achieving superior recognition accuracies. Using standard ML techniques our feature set outperforms all previous approaches on the cognitive tests considered, i.e., the Clock Drawing Test, the Rey-Osterrieth Complex Figure Test, and the Trail Making Test, by automatically scoring cognitive tests with up to 87.5% accuracy in a binary classification task.
... They proposed to perform a discrete check on a small number of edges to get the partial contour edge and then use connectivity to find all the contours. However, although this algorithm reduced the number of edges to be checked, it still required time-consuming floating-point multiplication to judge contour edges (Stapleton et al. 2015). Bagnall et al. used Catmull-Rom spline to parameterize contour feature edges and then used texture mapping method to draw. ...
For the purpose of applying information technology to the creation of ink style painting, the algorithm of ink painting rendering based on the deep learning framework and convolutional neural network model is designed and improved. Firstly, the ink style rendering program is written in Python. Secondly, VGG under Caffe architecture and Illustration 2Vec models are transplanted to TensorFlow architecture, and the image is rendered in ink style based on deep learning framework and convolutional neural network model. Finally, based on Node.js, the server-side program for image ink style rendering is built. Among them, Express is adopted as the Web-side framework, and the front-end page effect is completed. The results show that the ink rendering logic program is applicable, and the expected purpose is achieved.
... SetPad enabled users to explore discrete math problems by portraying set expressions using pen-based input [12]. SketchSet supported users in creating and editing Euler diagrams via sketch-based interactions or mouse editing [53,58]. SketchViz enabled dynamic domain comprehension and information reconstruction tasks [6], and was shown to provide a user experience similar to using pen and paper, while still offering features not supported by pen and paper or other existing tools (e.g., direct and bimanual manipulation) [8]. ...
Prewriting is the process of generating and organizing ideas before drafting a document. Although often overlooked by novice writers and writing tool developers, prewriting is a critical process that improves the quality of a final document. To better understand current prewriting practices, we first conducted interviews with writing learners and experts. Based on the learners' needs and experts' recommendations, we then designed and developed InkPlanner, a novel pen and touch visualization tool that allows writers to utilize visual diagramming for ideation during prewriting. InkPlanner further allows writers to sort their ideas into a logical and sequential narrative by using a novel widget — NarrativeLine. Using a NarrativeLine, InkPlanner can automatically generate a document outline to guide later drafting exercises. Inkplanner is powered by machine-generated semantic and structural suggestions that are curated from various texts. To qualitatively review the tool and understand how writers use InkPlanner for prewriting, two writing experts were interviewed and a user study was conducted with university students. The results demonstrated that InkPlanner encouraged writers to generate more diverse ideas and also enabled them to think more strategically about how to organize their ideas for later drafting.
... These relationships are visualized by labeled arrows between the sketches. Stapleton et al. (2015) combine traditional keyboard and mouse input with freehand pen input. They present a tool that allows the creation of diagrams and sketches in one of two modes: Users can either create diagrams via a formal diagram editor interface, or they can sketch them by using a stylus. ...
Most engineers and designers prefer to use large drawing boards such as whiteboards or flip charts for the initial collaborative sketching of a system’s models. Large interactive displays have recently begun to replace these physical drawing boards, blurring the line between freehand sketching and toolkit-aided modeling. While digital boards offer more flexibility in drawing and navigating models, they must also provide appropriate cognitive support for frequent shifts of focus and navigation between related artifacts. Furthermore, automated assistance in uncovering potential inconsistencies and contradictions between model sketches would be beneficial so that users do not get lost amid their sketches. In this paper, we discuss an approach to create relationships between the elements of informal hand-drawn sketches on large interactive displays by combining fuzzy search with classic information retrieval techniques. The identification and maintenance of relationships is particularly challenging because we are working with hand-drawn and hand-lettered model sketches rather than the syntactically clean models created with digital modeling toolkits. We evaluated our approach by analyzing 89 model sketches from 16 industry projects and found that it identifies relations between sketched model elements with high precision and recall.
... In particular, if all of the marked points belong to int(A) (i.e., M = ∅ ext ) then the zone z is covered by A and so the old zone z is removed from the diagram (lines 31-32). Moreover, if at least one marked point belongs to int(A) then the zone z is either split or covered and so a new zone is generated and added to the diagram (lines [35][36][37][38]. ...
... In [40], an application is presented that interprets an Euler Diagram sketched with a pen or a mouse, and calculates the abstract diagram (this application has been generalised in [35] to Euler Diagrams augmented with graphs). The authors claim complexity that is asymptotically similar to ours, but this claim is not substantiated, with the paper not providing details of how, given two regions r a and r b , the system actually computes whether r r ∩ a b is empty or not. ...
Euler diagrams are an accessible and effective visualisation of data involving simple set-theoretic relationships. Efficient algorithms to quickly compute the abstract regions of an Euler diagram upon curve addition and removal have previously been developed (the single marked point approach, SMPA), but a strict set of drawing conventions (called well-formedness conditions) were enforced, meaning that some abstract diagrams are not representable as concrete diagrams. We present a new methodology (the multiple marked point approach, MMPA) enabling online region computation for Euler diagrams under the relaxation of the drawing convention that zones must be connected regions. Furthermore, we indicate how to extend the methods to deal with the relaxation of any of the drawing conventions, with the use of concurrent line segments case being of particular importance. We provide complexity analysis and compare the MMPA with the SMPA. We show that these methods are theoretically no worse than other comparators, whilst our methods apply to any case, and are likely to be faster in practice due to their online nature. The machinery developed for the concurrency case could be of use in Euler diagram drawing techniques (in the context of the Euler Graph), and in computer graphics (e.g. the development of an advanced variation of a winged edge data structure that deals with concurrency). The algorithms are presented for generic curves; specialisations such as utilising fixed geometric shapes for curves may occur in applications which can enhance capabilities for fast computations of the algorithms' input structures. We provide an implementation of these algorithms, utilising ellipses, and provide time-based experimental data for benchmarking purposes.
In this paper, we introduce Speedith which is an interactive diagrammatic theorem prover for the well-known language of spider diagrams. Speedith provides a way to input spider diagrams, transform them via the diagrammatic inference rules, and prove diagrammatic theorems. Speedith’s inference rules are sound and complete, extending previous research by including all the classical logic connectives. In addition to being a stand-alone proof system, Speedith is also designed as a program that plugs into existing general purpose theorem provers. This allows for other systems to access diagrammatic reasoning via Speedith, as well as a formal verification of diagrammatic proof steps within standard sentential proof assistants. We describe the general structure of Speedith, the diagrammatic language, the automatic mechanism that draws the diagrams when inference rules are applied on them, and how formal diagrammatic proofs are constructed.
This paper presents an algorithm for proving plane geometry theorems stated by text and diagram in a complementary way. The problem of proving plane geometry theorems involves two challenging subtasks, being theorem understanding and theorem proving. This paper proposes to consider theorem understanding as a problem of extracting relations from text and diagram. A syntax-semantics (S2) model method is proposed to extract the geometric relations from theorem text, and a diagram mining method is proposed to extract geometry relations from diagram. Then, a procedure is developed to obtain a set of relations that is consistent with the given theorem with high confidence. Finally, theorem proving is conducted by using the existing proving methods which take the extracted geometric relations as input. The experimental results show that the proposed theorem proving algorithm can prove 86% of plane geometry theorems in the test dataset of 200 theorems, which is all the theorems in the popular textbook. The proposed algorithm outperforms the existing algorithms mainly because it can extract relations not only from text but also from diagram.
Today, modeling and subsequent digital model representations are essential constituents in collaborative endeavors on organizational change. Once created, models need to be digitized for shared stakeholder understanding and further processing. Whenever paper serves as carrier medium, it is likely to disrupt further processing, elicitation, and modeling. While digital environments support transformation processes in collaborative modeling from its very beginning, the necessary technical infrastructure still might hamper situated capturing of models. Hence, this contribution aims to reduce the need for sophisticated technical components by enabling stakeholders to capture their paper-based models in a situation-sensitive way. We present a system that enables capturing paper-based models with mundane technical means by end users under uncontrolled conditions. We describe the components developed for recognition of these models and embed it in a mixed-modality workflow supported by a tabletop interfacing a web platform for further processing. As our empirical evidence demonstrates, this approach enables both situated and error-tolerant capturing of hand-drawn conceptual models by individual users. Moreover, it can be integrated with more sophisticated IT-based modeling tools for further digital processing.
