Fig 2 - uploaded by Etienne Gernez
Content may be subject to copyright.
Overview of the five types of application components currently in use in the UI architecture We defined five main application component types, App display, Widget display, Annotation, Ocean overlay and AR map. The components offer distinct methods of information display and various systems may take advantage of their inherent affordances. Table I shows how the functions of an artic vessel can be mediated to officers using the various formats. Later in this section we show the current versions of a number of the components that we have tested in our lab and onboard ships.
Source publication
Augmented reality (AR) technologies support navigators by overlaying the perceived world with virtual information collected from the ship bridge systems. However, the variety of operational scenarios, types of ships and bridge equipment from different vendors requires an integration system that enables multiple maritime applications to employ AR as...
Context in source publication
Citations
... This current work has been carried out under the OpenAR 1 research project that is seeking to expand the OpenBridge Design Guideline by incorporating support for AR design [23]. ...
Augmented reality (AR) technology has emerged as a promising solution that can potentially reduce head-down time and increase situational awareness during navigation operations. It is also useful for remote operation centers where video feeds from remote ships can be "augmented" with data and information. In this article, we introduce a user interface design concept that supports ship navigation by showing data about points of interest in AR. This approach enables users to view and interact with relevant data in the maritime environment by bridging the gap between digital information and real-world features. The proposed concept can provide operational data from various maritime systems, such as radar, GPS, AIS, or camera systems, empowering users with a wealth of information about their surroundings. Developed through an iterative user-centered design process, it was built as an extension to the OpenBridge design system, an open-source platform facilitating consistent design in maritime workplaces. Furthermore, we use this concept to propose a design framework that paves the way for establishing new standards for AR user interface design in the maritime domain.
... The pilot can also maintain environmental awareness at night [6]. Nordby et al. presented a more detailed design idea for the information generation and display method and interactive interface of visual navigation aids [7]. In their work, the displayable area in smart glasses was divided into five areas: sky, water antenna, mask, free movement and water surface, and corresponding information generation and display were performed, respectively. ...
Visual aids for navigation systems are a key function of intelligent onboard navigation units for ships. These systems visualize the planning decisions of an intelligent navigation unit for computer-aided visual navigation to reduce the cognitive workload of the driving control personnel of an intelligent ship. In this study, we propose a dynamic data driven method to simulate the motion of ships on the water. We constructed a validation dataset for ship motion simulation toward visual-aided navigation considering the combined effects of wind, waves and currents. According to the validation results on this dataset, the proposed simulation method exhibited good accuracy and real-time performance. Under the given control and environmental input series, the mean errors of simulated heading and simulated trajectory were 5.7°/h and 282.6 m/h, respectively; the simulation time was 475 ms for each hour of physical duration. Overall, the proposed simulation method satisfies the need for visual navigation aids and could improve the situational awareness of the driving control personnel of the intelligent ship.
... At the same time, a structure for configuring the information contents from the modules is prepared, which enables a uniform representation of the information contents for different maintenance use cases in the maritime industry. Nordby et al. (2020) proposed a user interface (UI) architecture upon the OpenBridge Design System that describes how AR can function as an open, shared platform across different bridge systems by supporting the integration of generic maritime applications in AR. ...
Concern for the synthesis of the overall design process for marine structures, and its integration with production, maintenance, repair, service and decommissioning. Particular attention shall be given to the roles and requirements of computer-based design and production, and to the utilization of information technology
... AR superimposes information on top of a person's view in any environment. AR technology is becoming increasingly viable for maritime use, although still considered in the early stage of development (Frydenberg et al., 2021;Nordby et al., 2020). AR is interesting for maritime applications because it allows the user to maintain a heads-up position which is an integral component of maritime navigation. ...
... To move forward, there is a need to re-evaluate the operational environment and find solutions that allow for a more harmonized approach to safety and efficiency. One innovative solution that has been developed to address the lack of standardization in maritime workspaces is the OpenBridge project (Nordby et al., 2020;Nordby et al., 2019). OpenBridge is a Norwegian based open-source user-friendly solution which can be implemented into existing equipment, resulting in cost effective, immediate improvements in the organizational environment. ...
... Almost twenty years later Papers II and III have shown that mariners are hesitant to accept new decision support functions while existing systems remain non-integrated . The OpenBridge project is an example of how it is possible to design consistent user interfaces across multi-vendor systems, which include userinterface guidelines that are available open-source (Nordby et al., 2020). The OpenBridge navigation systems were used in Papers IV and V and participants were all in agreement that consistent user interface design is a necessary next step towards overall ship safety Frydenberg et al., 2021). ...
The maritime industry is undergoing a transformation driven by digitalization and connectivity. There is speculation that in the next two decades the maritime industry will witness changes far exceeding those experienced over the past 100 years. While change is inevitable in the maritime domain, technological developments do not guarantee navigational safety, efficiency, or improved seaway traffic management. The International Maritime Organization (IMO) has adopted the Maritime Autonomous Surface Ships (MASS) concept to define autonomy on a scale from Degrees 1 through 4. Investigations into the impact of MASS on various aspects of the maritime sociotechnical system is currently ongoing by academic and industry stakeholders. However, the early adoption of MASS (Degree 1), which is classified as a crewed ship with decision support, remains largely unexplored. Decision support systems are intended to support operator decision-making and improve operator performance. In practice they can cause unintended changes throughout other elements of the maritime sociotechnical system. In the maritime industry, the human is seldom put first in technology design which paradoxically introduces human-automation challenges related to technology acceptance, use, trust, reliance, and risk. The co-existence of humans and automation, as it pertains to navigation and navigational assistance, is explored throughout this thesis.
The aims of this thesis are (1) to understand how decision support will impact navigation and navigational assistance from the operator’s perspective and (2) to explore a framework to help reduce the gaps between the design and use of decision support technologies. This thesis advocates for a human-centric approach to automation design and development while exploring the broader impacts upon the maritime sociotechnical system. This work considers three different projects and four individual data collection efforts during 2017-2022. This research took place in Gothenburg, Sweden, and Warsash, UK and includes data from 65 Bridge Officers (navigators) and 16 Vessel Traffic Service (VTS) operators. Two testbeds were used to conduct the research in several full mission bridge simulators, and a virtual reality environment. A mixed methods approach, with a heavier focus on qualitative data, was adopted to understand the research problem. Methodological tools included literature reviews, observations, questionnaires, ship maneuvering data, collective interviews, think-aloud protocol, and consultation with subject matter experts. The data analysis included thematic analysis, subject matter expert consultation, and descriptive statistics.
The results show that operators perceive that decision support will impact their work, but not necessarily as expected. The operators’ positive and negative perceptions are discussed within the frameworks of human-automation interaction, decision-making, and systems thinking. The results point towards gaps in work as it is intended to be done and work as it is done in the user’s context. A user-driven design framework is proposed which allows for a systematic, flexible, and iterative design process capable of testing new technologies while involving all stakeholders. These results have led to the identification of several research gaps in relation to the overall preparedness of the shipping industry to manage the evolution toward smarter ships. This thesis will discuss these findings and advocate for human-centered automation within the quickly evolving maritime industry.
... The spatial, physical, and temporal possibilities of this technology challenge the current practice within the field of interaction design, which up until now has been mostly screen-based [3]. In this early phase of building knowledge of how to design for AR, our aim is to develop design examples of AR applications in order to build useful design frameworks for AR which can potentially support practicing interaction designers [4]. The background motivation for designing a software application for icebreaker assistance and convoy mode derives from our design-driven field research at sea [5], where our interviews and observations with navigators demonstrated that the premises for icebreaker assistance had considerable potential for the improvement of safety for both the icebreaking vessel and the assisted vessels during these critical operations. ...
... In recent years, the development of AR technologies has been rapid and marketdriven, especially after the launch of head-mounted AR technologies, such as Microsoft Hololens [22]. The potential for AR in the maritime domain shows a promising outlook [23], and previous research has uncovered a range of suitable scenarios for its use on ship's bridges [3,4]. Navigator's SA and performance can be enhanced through the use of AR by functioning as an information support for navigating, regulating information overload, reducing integration work, and connecting digital and analogue information [3,[23][24][25][26]. ...
... Navigator's SA and performance can be enhanced through the use of AR by functioning as an information support for navigating, regulating information overload, reducing integration work, and connecting digital and analogue information [3,[23][24][25][26]. Nevertheless, research on how to involve users' needs in the design process of AR technology for the maritime domain remains limited [4]. Therefore, we argue the need to develop tools with relevant and specific data and flexible UIs that support navigators with precise information on icebreaker assistance operations, such as during convoy operations. ...
A vessel convoy is a complex and high-risk operation completed during icebreaking operations in the Arctic. Icebreaker navigators need to continuously communicate with their crew while monitoring information such as speed, heading, and distance between vessels in the convoy. This paper presents an augmented reality user interface concept, which aims to support navigators by improving oversight and safety during convoy operations. The concept demonstrates how augmented reality can help to realize a situated user interface that adapts to user’s physical and operational contexts. The concept was developed through a human-centered design process and tested through a virtual reality simulator in a usability study involving seven mariners. The results suggest that augmented reality has the potential to improve the safety of convoy operations by integrating distributed information with heads-up access to operation-critical information. However, the user interface concept is still novel, and further work is needed to develop the concept and safely integrate augmented reality into maritime operations.
... One of the challenges in maritime research is that maritime user interfaces for ships' bridges are highly dependent on the context in which they are used, and a rich maritime context is difficult to recreate in the early stages of user-centered design processes [25,39]. The SEDNA project has removed the constraints of regulation and existing technological solutions, and has instead applied user-centered, design thinking throughout the project's life cycle to develop, test, and apply novel concepts for navigation using VR. ...
Maritime user interfaces for ships’ bridges are highly dependent on the context in which they are used, and rich maritime context is difficult to recreate in the early stages of user-centered design processes. Operations in Arctic waters where crews are faced with extreme environmental conditions, technology limitations and a lack of accurate navigational information further increase this challenge. There is a lack of research supporting the user-centered design of workplaces for hazardous Arctic operations. To meet this challenge, this paper reports on the process of developing virtual reality-reconstructed operational scenarios to connect stakeholders, end-users, designers, and human factors specialists in a joint process. This paper explores how virtual reality-reconstructed operational scenarios can be used as a tool both for concept development and user testing. Three operational scenarios were developed, implemented in a full mission bridge simulator, recreated in virtual reality (VR), and finally tested on navigators (end-users). Qualitative data were captured throughout the design process and user-testing, resulting in a thematic analysis that identified common themes reflecting the experiences gained throughout this process. In conclusion, we argue that operational scenarios, rendered in immersive media such as VR, may be an important and reusable asset when supporting maritime design processes and in maritime training and education.
... • Integrar nos mapas, conteúdos como os pontos de interesse do mundo real (Nordby et al., 2020). ...
... Tome-se, como exemplo, a importância da escolha de uma paleta cromática pode enriquecer a visualização dos conteúdos que são medidos pela RA(Frydenberg et al., 2018;Morgère, 2015). Além disso, a visualização de informações tridimensionais(Morgère, 2015;Porathe, 2006;Porathe & Ekskog, 2018) e a reprodução de sinais sonoros podem contribuir para a identificação das áreas interditais.Em síntese dos vários estudos analisados(Cojoc-wisernig, 2020;Frydenberg et al., 2018;Li & Fessenden, 2016;Nordby et al., 2020;Stähli et al., 2020), verificou-se um conjunto de aspetos para otimizar a interface de um sistema de navegação em RA: ...
A Ria de Aveiro é utilizada por inúmeros navegadores para diversas finalidades tais como o turismo, a pesca e a cultura. Navegar no mar ou num rio exige uma maior atenção e interpretação dos eventuais perigos iminentes comparativamente à condução de um carro. No caso concreto da Ria de Aveiro, a falta de boias de sinalização - responsáveis por indicar a zona navegável dos canais - e a variação da altura da maré fazem parte dos principais desafios para quem navega neste plano de água. Para além disso, as aplicações móveis atualmente disponíveis no mercado que utilizam cartografias, não mapeiam adequadamente, quer ao nível dos contornos dos canais, quer no que diz respeito à respetiva batimetria, imensas zonas da Ria de Aveiro que têm um potencial de navegação (em determinadas condições de maré) fantástico. Acresce a esta problemática o facto de que, quando são utilizados dispositivos móveis (tal como se pretende no projeto de investigação ao qual este mestrado está associado), estes geralmente não têm ecrãs com brilho e contraste suficiente para a utilização de mapas em ambientes exteriores e marítimos influenciados pela luz solar. O estudo em torno do fenómeno Situational Visual Impairments (SVI), por um lado, analisa a forma como a interação com um produto pode ser afetada pelo cenário/ambiente de uso e, por outro lado, pretende promover o design das interfaces de forma apropriada ao ambiente de utilização do website. O presente estudo, pretende colmatar as lacunas referidas através do desenvolvimento de uma solução tecnológica de apoio à navegação segura em águas interiores influenciadas pela variação das condições de maré. Neste contexto, os trajetos que os utilizadores podem fazer são, aquando da sua simulação, aumentados com uma escala cromática que identifica as respetivas condições de navegabilidade. Para além disso, deu-se outro passo a este nível, com a inclusão de limitadores e boias virtuais que permitem identificar os limites navegáveis de cada canal da ria. De forma a maximizar o acesso, em ambientes marítimos, à informação disponibilizada pelo website responsivo desenvolvido realizaram-se Guerrilla Tests, no sentido de melhorar a experiência dos navegadores da Ria de Aveiro.
Ria de Aveiro has different purposes for countless navigators, such as tourism, fishing and culture. Sailing on the sea or the river requires greater attention and interpretation due to the imminent dangers that arise when compared to driving a car. The lack of signal buoys - responsible for indicating the navigable area of the channel - and the variance of tidal height in certain places of the Ria de Aveiro are part of the main challenges for those navigating on the ria. Furthermore, the mobile applications using cartographies currently available on the market, do not map appropriately, either in terms of channel contours or concerning the extensive areas of the ria with great navigation potential (under certain tidal conditions). In addition to this problem, when visualizing outdoor maps and maritime environments in mobile devices (a requirement of the research project to which this Master’s degree is associated) without adequate brightness and contrast - influenced by sunlight –, difficulties arise and impair reading and access to essential information for navigation. The study around the phenomenon of Situational Visual Impairments (SVI), on one hand, analyses how the interaction with a product can be affected by the scenario/environment of use and, on the other hand, intends to promote the design of appropriate contents for the website's context of use. Aiming to fill these gaps, this study developed a technological solution to support safer navigation in inland waters influenced by the variation of tidal conditions. The sailable routes are augmented with a colour scale that clarifies the navigability conditions, and limiters were included to aid the identification of the navigable limits of each channel of the Ria. Moreover, to enhance the access to the application’s information in maritime environments, Guerrilla tests were carried out to improve the experience of the Ria de Aveiro’s navigators through the responsive website developed.
... • Arctic voyage optimization tools (Li, et al., 2019) (Huang, et al., 2020) • Tools for improved onboard situational awareness using augmented reality technology (Nordby, et al., 2020) • Anti-icing coatings (Choy, 2020). ...
Maritime activity in the Arctic is on the increase, driven by multiple factors including the development of Arctic natural resources, climate change, regulatory changes, improved technology and operations, national and international policies, infrastructure developments, fuel prices, and the global economy. This article aims to identify and analyze such change drivers and to discuss how these might influence the future of three specific segments of Arctic shipping: destination-Arctic shipping (operation between Arctic and non-Arctic locations), trans-Arctic shipping (operation between non-Arctic locations through Arctic waters), and Arctic cruising. The study finds that each considered segment of Arctic shipping is subject to a unique set of significant change drivers.
... Further, given the existing usability problems on ship bridges, research must lay the ground for future development to avoid the current problems related to multivendor ship bridges related to inconsistent design and a lack of user interface integration. We refer to ongoing development of a design framework for AR applications expanding the OpenBridge design system to address these issues (59). (46) Content first: A concept for industrial augmented reality maintenance applications using mobile devices Remote inspection and maintenance (47) Benefits achieved by applying augmented reality technology in the marine industry Remote inspection and maintenance (48) AR-based ship design information supporting system for pipe maintenance ...
We present a state-of-the art analysis of Augmented Reality (AR) applications for ship bridge operation. We compiled and reviewed what type of use cases were published, what type of maritime applications have been adapted to AR, how they were prototyped and evaluated and what type of technology was used. We also reviewed the user interaction mechanisms, information display and adaptation to maritime environmental conditions. Our analysis shows that although there are many examples of AR applications in ship bridges, there is still much work that needs to be done before these solutions can be suitably adapted to commercial settings. In addition, we argue there is a need to develop design requirements and regulations that can guide the safe development of AR.
