Figure 5 - uploaded by Amanda Swearngin
Content may be subject to copyright.
A calculator interface with incomplete keyboard support (a-calculator.com), enhanced to provide a keyboard shortcut for each command, as enabled by Genie's analyses.
Source publication
Most web applications are designed as one-size-fits-all, despite considerable variation in people's expertise, physical abilities, and other factors that impact interaction. For example, some web applications require the use of a mouse, precluding use by many people with severe motor disabilities. Other applications require laborious manual input t...
Contexts in source publication
Context 1
... system prioritizes imperative phrases and verbs to use as these labels, but if none can be found will fall back to remaining metadata. Developers can use the remaining metadata to provide a more detailed description of the command to be shown in an interface, such as the one shown in Figure 5. This remaining metadata is labeled as description metadata. ...
Context 2
... the two outputs of this process are a command label and description metadata. In Figure 5, the multiplication command has a command label Multiply and description metadata of Operator button, and Ops. ...
Context 3
... websites only support mouse input, or selectively implement support for a small set of keyboard accelerators. For example, the calculator shown in Figure 5 is a basic calculator with simple number, operator (e.g., +, *), and memory functions (e.g., MR, MC). It has keyboard shortcuts for numbers, but not for operators or memory functions, rendering the calculator useless without a mouse. ...
Context 4
... used Genie to build the interface shown in Figure 5. This interface displays a list of commands and a corresponding automatically generated keyboard shortcut for each command. ...
Citations
... An alternative is to repurpose and augment existing applications. For example, existing UIs can be retargeted to support new modalities [46] or support responsive resizing [21,52]. Interaction proxies apply runtime modification to existing mobile apps by "repairing" inaccessible or difficult-to-use UI elements [57]. ...
Touch is the primary way that users interact with smartphones. However, building mobile user interfaces where touch interactions work well for all users is a difficult problem, because users have different abilities and preferences. We propose a system, Reflow, which automatically applies small, personalized UI adaptations, called refinements -- to mobile app screens to improve touch efficiency. Reflow uses a pixel-based strategy to work with existing applications, and improves touch efficiency while minimally disrupting the design intent of the original application. Our system optimizes a UI by (i) extracting its layout from its screenshot, (ii) refining its layout, and (iii) re-rendering the UI to reflect these modifications. We conducted a user study with 10 participants and a heuristic evaluation with 6 experts and found that applications optimized by Reflow led to, on average, 9% faster selection time with minimal layout disruption. The results demonstrate that Reflow's refinements useful UI adaptations to improve touch interactions.
... The use of computational approaches for understanding, generating, and adapting user interfaces has been an important long-time research topic in the CHI community [3,33,35]. In recent years, the need for adapting interfaces to support different modalities, usage scenarios, and form factors of devices [7,12,31,32], the goal of understanding screen semantics for accessibility purposes [23,36], the use of graphical interfaces as a resource for understanding their underlying tasks [19,21,22], the computational generative approach for assisting the design of user interfaces [9,14], and the popularity of intelligent interface agents that understand and manipulate user interfaces based on users' behaviors [16,29] have significantly expanded the horizon of this area, opening up new research opportunities while drawing significant attention and community interest to this topic. ...
... To better understand and customize user interfaces, reverse engineering has been used as a way to perform GUI customization. Prior reverse engineering approaches can identify interface elements and allow the user to add interactive enhancements to a GUI (e.g., Façades [30] and Prefab [5]), infer constraints to describe a layout from UI exemplars (e.g., InferUI [2]), recover higher-level constraint-based specifications [25], enable users to engage with web applications via different input modalities (e.g., Genie [31]), generate potential constraint-based layout alternatives (e.g., Scout [32]), and enable platform-and framework-independent customization for both GUI layouts and web layouts (e.g., ReverseORC [12]). An important objective on this research topic that we wish to discuss in this work is to better understand both the layout structure and the design process of GUI designers, and further develop efficient approaches to design adaptive user interfaces. ...
Computational approaches for user interfaces have been used in adapting interfaces for different modalities, usage scenarios and device form factors, understanding screen semantics for accessibility , task-automation, information extraction, and in assisting interface design. Recent advances in machine learning (ML) have drawn considerable attention across HCI and related fields such as computer vision and natural language processing, leading to new ML-based user interface approaches. Similarly, significant progress has been made with more traditional optimization-and planning-based approaches to accommodate the need for adapting UIs for screens with different sizes, orientations and aspect ratios, and in emerging domains such as VR/AR and 3D interfaces. The proposed workshop seeks to bring together researchers interested in all kinds of computational approaches for user interfaces across different sectors as a community, including those who develop algorithms and models and those who build applications, to discuss common issues including the need for resources, opportunities for new applications , design implications for human-AI interaction in this domain, and practical challenges such as user privacy.
... By using automated RE tools, specifications of GUI elements, layouts, and application behaviours can be extracted and modified independently of their underlying implementations and platforms. Previous works on reverse engineering UIs focus on detecting components in the interface [8,42,64], migrating UIs from one platform to another [17,37,41,43,44,56,57], and/or performing input and output redirection [9,10,64,65]. Previous works have shown that RE tools can reconstruct UI layouts that look similar to the originals, and can then generate implementations of the UI for other UI platforms and toolkits. ...
... Sikuli [6,71] supported UI testing by writing visual test scripts. Genie [65] reverse engineered underlying commands to enable users to engage with web applications via different input modalities. Hurst et al. [27] presented improved target boundary detection based on the combination of an accessibility API and pixel-based methods. ...
Reverse engineering (RE) of user interfaces (UIs) plays an important role in software evolution. However, the large diversity of UI technologies and the need for UIs to be resizable make this challenging. We propose ReverseORC, a novel RE approach able to discover diverse layout types and their dynamic resizing behaviours independently of their implementation, and to specify them by using OR constraints. Unlike previous RE approaches, ReverseORC infers flexible layout constraint specifications by sampling UIs at different sizes and analyzing the differences between them. It can create specifications that replicate even some non-standard layout managers with complex dynamic layout behaviours. We demonstrate that ReverseORC works across different platforms with very different layout approaches, e.g., for GUIs as well as for the Web. Furthermore, it can be used to detect and fix problems in legacy UIs, extend UIs with enhanced layout behaviours, and support the creation of flexible UI layouts.
... Interacting with an Internet of Things can also benefit from a combination of freehand pointing and voice commands, as demonstrated in the Minuet system [14]. As Web-based UI becomes increasingly pervasive, it becomes important to offer flexible moddalities at the users' disposal, e.g., using input targeting [35]; we expect our tool to be extensible to address multimodal interaction in these above application scenarios. ...
Supporting voice commands in applications presents significant benefits to users. However, adding such support to existing GUI-based web apps is effort-consuming with a high learning barrier, as shown in our formative study, due to the lack of unified support for creating multimodal interfaces. We present Geno---a developer tool for adding the voice input modality to existing web apps without requiring significant NLP expertise. Geno provides a high-level workflow for developers to specify functionalities to be supported by voice (intents), create language models for detecting intents and the relevant information (parameters) from user utterances, and fulfill the intents by either programmatically invoking the corresponding functions or replaying GUI actions on the web app. Geno further supports multimodal references to GUI context in voice commands (e.g. "move this [event] to next week" while pointing at an event with the cursor). In a study, developers with little NLP expertise were able to add multimodal voice command support for two existing web apps using Geno.
