No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Approvable AI for Autonomous Ships: Challenges and Possible Solutions
Abstract
Artificial Intelligence (AI) is being promoted as an important contributor to ensuring the safety of autonomous ships. However, utilizing AI technologies to enhance safety may be problematic. For instance, AI is only capable of performing well in situations that it has been trained on, or otherwise programmed to handle. Quantifying the true performance of such technologies is, therefore, difficult. This raises the question if these technologies can be applied on larger ships that need approval and safety certification. The issue gains further complexity when introduced as an element in remote control centres. This paper presents an overview of the most relevant applications of AI for autonomous ships, as well as their limitations in the context of approval. It is found that approval processes may be eased by restricting the operational envelope of such systems, as well as leveraging recent developments in explainable and trustworthy AI. If leveraged properly, AI models can be rendered self-aware of their limitations, and applied only in low risk situations, reducing the workload of human operators. In high risk situations, e.g. high AI model uncertainty or complex navigational situations, a timely and effective handover to a human operator should take place. In this manner, AI-based systems need not be capable of handling all possible situations, but rather be capable of identifying their limitations and alerting human operators to situations that they are incapable of handling with an acceptable level of risk.
... Technological advancements in Artificial Intelligence (AI) and machine learning (ML) have played a crucial role in the development of autonomous ships [1]. As per the definition by maritime autonomous surface ships (MASSs) of the International Maritime Organization (IMO), the autonomous ships are equipped with a system with the capability to make decisions and contrive pertinent actions [2]. ...
This paper presents the comprehensive state-of-the-art on the challenges that short sea shipping currently faces across the world. The concept and its relationship with coastal shipping are introduced, followed by a review of the EU policies for short sea shipping and its practical impacts in modal split. A survey of the literature on the strong and weak points of this form of transportation is carried out, aimed at explaining the difficulties in achieving relevant modal shifts from road to sea. The experience with short sea shipping across the world is described and discussed, providing a global perspective. The paper then discusses the main challenges and opportunities in this field, namely decarbonisation, autonomous navigation, and digitalization. Conclusions are drawn on the possible impact of these game changing developments in this segment of the shipping industry.
Automatic controllers work best when the system they control can be sufficiently well modelled. This is a problem for control of autonomous ships in mixed traffic situations where the autonomous ship interacts with conventional ships, as the crew on other ships can and will exert unexpected behaviour that cannot be easily modelled. This paper analyses the problem of information acquisition, situational assessment and how to predict other ship's actions for autonomous ships that need to interact with conventional ships. We identify causes for the interaction problem and classify these into a decision making model. We also identify possible measures to overcome the problems and based on an impact analysis where technical, procedural and regulatory aspects are considered, we discuss and propose some possible ways to reduce or solve this problem. The conclusion is that the most likely and effective short-term solution is to assist the autonomous ships with human operators and the best longer-term solution may be to improve the information exchange between the ships, complemented with changes in COLREGs.
Design and verification of autonomous vessels represent a major interdisciplinary engineering challenge due to the combination of high system complexity and the interaction with dynamic, uncertain, and unstructured environments. This paper investigates the use of contract-based methods to address both design and verification challenges of control systems for autonomous vessels. The paper first presents a formal framework for specification of components and assume-guarantee contracts using the syntax of the Z3 automated theorem prover. Then, the paper proposes a methodology for contract-based verification using the formal framework. The methodology is divided into 4 steps: (1) Hazard identification between the autonomous vessel and the operative environment in order to define the top-level component and contract, (2) stepwise refinement of the top-level component into detailed sub-components and contracts, (3) definition of test setups for simulation-based testing to verify that components meet their contract, and (4) applying a recursive procedure for contracts-based system verification. The framework and methodology are demonstrated the in a case study with an autonomous passenger ferry.
Autonomous ships are expected to improve the level of safety and efficiency in future maritime navigation. Such vessels need perception for two purposes: to perform autonomous situational awareness and to monitor the integrity of the sensor system itself. In order to meet these needs, the perception system must fuse data from novel and traditional perception sensors using Artificial Intelligence (AI) techniques. This article overviews the recognized operational requirements that are imposed on regular and autonomous seafaring vessels, and then proceeds to consider suitable sensors and relevant AI techniques for an operational sensor system. The integration of four sensors families is considered: sensors for precise absolute positioning (Global Navigation Satellite System (GNSS) receivers and Inertial Measurement Unit (IMU)), visual sensors (monocular and stereo cameras), audio sensors (microphones), and sensors for remote-sensing (RADAR and LiDAR). Additionally, sources of auxiliary data, such as Automatic Identification System (AIS) and external data archives are discussed. The perception tasks are related to well-defined problems, such as situational abnormality detection, vessel classification, and localization, that are solvable using AI techniques. Machine learning methods, such as deep learning and Gaussian processes, are identified to be especially relevant for these problems. The different sensors and AI techniques are characterized keeping in view the operational requirements, and some example state-of-the-art options are compared based on accuracy, complexity, required resources, compatibility and adaptability to maritime environment, and especially towards practical realization of autonomous systems.
Automatic controllers work best when the system they control can be sufficiently well modelled. This is a problem for autonomous ships that interact with conventional ships, as the crew on other ships can and will exert unexpected behaviour that cannot be easily modelled. This paper will discuss the problem of situational assessment and prediction of other ship's actions, for autonomous ships that need to interact with conventional ships. It will provide a simple classification of root causes for the problem and will propose some possible ways to reduce or solve this problem.
Uncertainty quantification (UQ) methods play a pivotal role in reducing the impact of uncertainties during both optimization and decision making processes. They have been applied to solve a variety of real-world applications in science and engineering. Bayesian approximation and ensemble learning techniques are two of the most widely-used types of uncertainty quantification (UQ) methods. In this regard, researchers have proposed different UQ methods and examined their performance in a variety of applications such as computer vision (e.g., self-driving cars and object detection), image processing (e.g., image restoration), medical image analysis (e.g., medical image classification and segmentation), natural language processing (e.g., text classification, social media texts and recidivism risk-scoring), bioinformatics, etc. This study reviews recent advances in UQ methods used in deep learning, investigates the application of these methods in reinforcement learning, and highlights the fundamental research challenges and directions associated with the UQ field.
Current guidelines for approval of autonomous ship systems are focused on the ships' concrete operations and their geographic area. This is a natural consequence of the link between geography and the navigational complexity, but moving the ship to a new area or changing owners may require a costly re-approval. The automotive industry has introduced the Operational Design Domain (ODD) that can be used as a basis for approval. However, the ODD does not include the human control responsibilities, while most autonomous ship systems are expected to be dependent on sharing control responsibilities between humans and automation. We propose the definition of an operational envelope for autonomous ship systems that include the sharing of responsibilities between human and automation, and that is general enough to allow approval of autonomous ship systems in all geographic areas and operations that falls within the envelope. We also show how the operational envelope can be defined using a system modelling language, such as the unified modelling language (UML).
The notion of uncertainty is of major importance in machine learning and constitutes a key element of machine learning methodology. In line with the statistical tradition, uncertainty has long been perceived as almost synonymous with standard probability and probabilistic predictions. Yet, due to the steadily increasing relevance of machine learning for practical applications and related issues such as safety requirements, new problems and challenges have recently been identified by machine learning scholars, and these problems may call for new methodological developments. In particular, this includes the importance of distinguishing between (at least) two different types of uncertainty, often referred to as aleatoric and epistemic . In this paper, we provide an introduction to the topic of uncertainty in machine learning as well as an overview of attempts so far at handling uncertainty in general and formalizing this distinction in particular.
Artificial Intelligence (AI) has revolutionized today's Internet of Things (IoT) applications and services by introducing significant technological enhancements across a multitude of domains. With the deployment of the fifth generation (5G) mobile communication network, smart city visions of fast, on-demand, intelligent user-specific services are now becoming a reality. The concept of connected IoT is evolving into connected intelligent things. The advancements of both AI techniques, coupled with the sophistication of edge devices, is now leading to a new era of connected intelligence. Moving the intelligence toward end devices must account for latency demands and simplicity of selecting the type of AI technique to be used. Moreover, since most AI techniques require learning from big data sets and reasoning using a multitude of classification patterns, new simplified and collaborative solutions are now necessary more than ever. As such, the concept of introducing decentralized and distributed ‘Plug and Play’ (PnP) AI tools is now becoming more attractive given the vast numbers in edge devices, data volume and AI techniques. To this end, this article envisions a novel general AI solution that can be adapted to autonomously select the type of machine learning (ML) algorithm, the data set to be used, and provide reasoning in regards to data selection for optimal features extraction. Moreover, the solution performs the necessary training and all the necessary parameter fine-tunings to achieve the highest level of generality and simplicity for AI at the edge. We explore several aspects related to PnP-AI and its impact in the smart city ecosystem.
Dramatic success in machine learning has led to a new wave of AI applications (for example, transportation, security, medicine, finance, defense) that offer tremendous benefits but cannot explain their decisions and actions to human users. DARPA’s explainable artificial intelligence (XAI) program endeavors to create AI systems whose learned models and decisions can be understood and appropriately trusted by end users. Realizing this goal requires methods for learning more explainable models, designing effective explanation interfaces, and understanding the psychologic requirements for effective explanations. The XAI developer teams are addressing the first two challenges by creating ML techniques and developing principles, strategies, and human-computer interaction techniques for generating effective explanations. Another XAI team is addressing the third challenge by summarizing, extending, and applying psychologic theories of explanation to help the XAI evaluator define a suitable evaluation framework, which the developer teams will use to test their systems. The XAI teams completed the first of this 4-year program in May 2018. In a series of ongoing evaluations, the developer teams are assessing how well their XAM systems’ explanations improve user understanding, user trust, and user task performance.
There are two major types of uncertainty one can model. Aleatoric uncertainty captures noise inherent in the observations. On the other hand, epistemic uncertainty accounts for uncertainty in the model -- uncertainty which can be explained away given enough data. Traditionally it has been difficult to model epistemic uncertainty in computer vision, but with new Bayesian deep learning tools this is now possible. We study the benefits of modeling epistemic vs. aleatoric uncertainty in Bayesian deep learning models for vision tasks. For this we present a Bayesian deep learning framework combining input-dependent aleatoric uncertainty together with epistemic uncertainty. We study models under the framework with per-pixel semantic segmentation and depth regression tasks. Further, our explicit uncertainty formulation leads to new loss functions for these tasks, which can be interpreted as learned attenuation. This makes the loss more robust to noisy data, also giving new state-of-the-art results on segmentation and depth regression benchmarks.
Ensuring the safety of fully autonomous vehicles requires a multi-disciplinary approach across all the levels of functional hierarchy, from hardware fault tolerance, to resilient machine learning, to cooperating with humans driving conventional vehicles, to validating systems for operation in highly unstructured environments, to appropriate regulatory approaches. Significant open technical challenges include validating inductive learning in the face of novel environmental inputs and achieving the very high levels of dependability required for full-scale fleet deployment. However, the biggest challenge may be in creating an end-to-end design and deployment process that integrates the safety concerns of a myriad of technical specialties into a unified approach.
Maritime assets such as merchant and navy ships, ports, and harbors, are targets of terrorist attacks as evidenced by the
USS Cole bombing. Conventional methods of securing maritime assets to prevent attacks are manually intensive and error prone.
To address this shortcoming, we are developing a decision support system that shall alert security personnel to potential
attacks by automatically processing maritime surveillance video. An initial task that we must address is to accurately classify
maritime objects from video data, which is our focus in this paper. Object classification from video images can be problematic
due to noisy outputs from image processing. We approach this problem with a novel technique that exploits maritime domain
characteristics and formulates it as a graph of spatially related objects. We then apply a case-based collective classification algorithm on the graph to classify objects. We evaluate our approach on river traffic video data that we have processed.
We found that our approach significantly increases classification accuracy in comparison with a conventional (i.e., non-relational)
alternative.
Operators in high-risk domains such as aviation often need to make decisions under time pressure and uncertainty. One way to support them in this task is through the introduction of decision support systems (DSSs). The present study examined the effectiveness of two different DSS implementations: status and command displays. Twenty-seven pilots (9 pilots each in a baseline, status, and command group) flew 20 simulated approaches involving icing encounters. Accuracy of the decision aid (a smart icing system), familiarity with the icing condition, timing of icing onset, and autopilot usage were varied within subjects. Accurate information from either decision aid led to improved handling of the icing encounter. However, when inaccurate information was presented, performance dropped below that of the baseline condition. The cost of inaccurate information was particularly high for command displays and in the case of unfamiliar icing conditions. Our findings suggest that unless perfect reliability of a decision aid can be assumed, status displays may be preferable to command displays in high-risk domains (e.g., space flight, medicine, and process control), as the former yield more robust performance benefits and appear less vulnerable to automation biases.
The rapid development of artificial intelligence significantly promotes collision-avoidance navigation of maritime autonomous surface ships (MASS), which in turn provides prominent services in maritime environments and enlarges the opportunity for coordinated and interconnected operations. Clearly, full autonomy of the collision-avoidance navigation for the MASS in complex environments still faces huge challenges and highly requires persistent innovations. First, we survey relevant guidance of the International Maritime Organization (IMO) and industry code of each country on MASS. Then, major advances in MASS industry R&D, and collision-avoidance navigation technologies, are thoroughly overviewed, from academic to industrial sides. Moreover, compositions of collision-avoidance navigation, brain-inspired cognitive navigation, and e-navigation technologies are analyzed to clarify the mechanism and principles efficiently systematically in typical maritime environments, whereby trends in maritime collision-avoidance navigation systems are highlighted. Finally, considering a general study of existing collision avoidance and action planning technologies, it is pointed out that collision-free navigation would significantly benefit the integration of MASS autonomy in various maritime scenarios.
Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of "what to fuse", "when to fuse", and "how to fuse" remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, we first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. We then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, we provide tables that summarize topics and methods. We also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/.
As autonomous and semiautonomous systems are developed for automotive, aviation, cyber, robotics and other applications, the ability of human operators to effectively oversee and interact with them when needed poses a significant challenge. An automation conundrum exists in which as more autonomy is added to a system, and its reliability and robustness increase, the lower the situation awareness of human operators and the less likely that they will be able to take over manual control when needed. The human-autonomy systems oversight model integrates several decades of relevant autonomy research on operator situation awareness, out-of-the-loop performance problems, monitoring, and trust, which are all major challenges underlying the automation conundrum. Key design interventions for improving human performance in interacting with autonomous systems are integrated in the model, including human-automation interface features and central automation interaction paradigms comprising levels of automation, adaptive automation, and granularity of control approaches. Recommendations for the design of human-autonomy interfaces are presented and directions for future research discussed.
A quantitative and practical Bayesian framework is described for learning of mappings in feedforward networks. The framework makes possible (1) objective comparisons between solutions using alternative network architectures, (2) objective stopping rules for network pruning or growing procedures, (3) objective choice of magnitude and type of weight decay terms or additive regularizers (for penalizing large weights, etc.), (4) a measure of the effective number of well-determined parameters in a model, (5) quantified estimates of the error bars on network parameters and on network output, and (6) objective comparisons with alternative learning and interpolation models such as splines and radial basis functions. The Bayesian "evidence" automatically embodies "Occam's razor," penalizing overflexible and overcomplex models. The Bayesian approach helps detect poor underlying assumptions in learning models. For learning models well matched to a problem, a good correlation between generalization ability and the Bayesian evidence is obtained.
Ni 641: Guidelines for autonomous shipping
- B V Bv
BV, B. V. (2019). Ni 641: Guidelines for autonomous
shipping.
DNV-CG-0264 Class Guideline -Autonomous and remotely operated ships
- Dnv
DNV (2021a). DNV-CG-0264 Class Guideline -Autonomous and remotely operated ships.
DNV-RP-0510 Framework for assurance of data-driven algorithms and models
- Dnv
DNV (2021b). DNV-RP-0510 Framework for assurance of data-driven algorithms and models.
Automated machine learning: State-of-the-art and open challenges
- R Elshawi
- M Maher
- S Sakr
Elshawi, R., M. Maher, and S. Sakr (2019). Automated machine learning: State-of-the-art and open
challenges. arXiv preprint arXiv:1906.02287.
Ship collision avoidance methods: State-of-the-art
- Y Huang
- L Chen
- P Chen
- R R Negenborn
- P Van Gelder
Huang, Y., L. Chen, P. Chen, R. R. Negenborn, and
P. van Gelder (2020). Ship collision avoidance methods: State-of-the-art. Safety Science 121, 451-473.
Convention on the International Regulations for Preventing Collisions at Sea, (COLREGs) -as amended
- Imo
IMO (1972). Convention on the International Regulations for Preventing Collisions at Sea, (COLREGs) -as amended.
MSC.1/Circ.1455 Guidelines for the approval of alternatives and equivalents as provided for in various IMO instruments
- Imo
IMO (2013). MSC.1/Circ.1455 Guidelines for the approval of alternatives and equivalents as provided for
in various IMO instruments.
A benchmark for deep learning based object detection in maritime environments
- S Moosbauer
- D Konig
- J Jakel
- M Teutsch
Moosbauer, S., D. Konig, J. Jakel, and M. Teutsch
(2019). A benchmark for deep learning based object
detection in maritime environments. In Proceedings
of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition Workshops, pp. 0-0.
National Academies of Sciences, E. and Medicine
(2022). Human-AI Teaming: State-of-the-Art and
Research Needs. Washington, DC: The National
Academies Press.
Circular RSV 12-2020, Guidelines in connection with the construction or installation of automated functionality, with the intention of being able to perform uncrewed or partially uncrewed operation
- Authority Norwegian Maritime
Norwegian Maritime Authority (2020). Circular RSV
12-2020, Guidelines in connection with the construction or installation of automated functionality, with
the intention of being able to perform uncrewed or
partially uncrewed operation, In Norwegian.
On challenges in machine learning model management
- S Schelter
- F Biessmann
- T Januschowski
- D Salinas
- S Seufert
- G Szarvas
Schelter, S., F. Biessmann, T. Januschowski, D. Salinas,
S. Seufert, and G. Szarvas (2018). On challenges in
machine learning model management.
Striving for simplicity: The all convolutional net
- J T Springenberg
- A Dosovitskiy
- T Brox
- M Riedmiller
Springenberg, J. T., A. Dosovitskiy, T. Brox, and
M. Riedmiller (2015, dec). Striving for simplicity:
The all convolutional net. 3rd International Conference on Learning Representations, ICLR 2015 -Workshop Track Proceedings.
Autonomous systems: Issues for defence policymakers
- A P Williams
- P D Scharre
Williams, A. P. and P. D. Scharre (2015). Autonomous
systems: Issues for defence policymakers. NATO
Allied Command Transformation.