IMPROVED SITUATIONAL AND SPATIAL AWARENESS FOR AUTONOMOUS SYSTEMS IN HARBOUR PROTECTION BY MEANS OF VIRTUAL AND AUGMENTED REALITY

INTRODUCTION

Underwater Unmanned Vehicles (UUVs), operating in a fully autonomous way, transformed the way in which civilian and military operations are conducted. Since the 1960s, when the first Remotely Operated Vehicles (ROVs) were introduced to perform subsea operations, their potential to successfully complete missions in adverse environmental conditions has been widely recognised. However, and in spite of these benefits and the technological advances in recent years, their adoption and integration into military underwater applications has been slow and has not yet peaked^[1].

Studies published by defence organizations, such as NATO^[2] and the United States Department of Defence^[3], and scientific publications ^[1][4] reveal technical and cultural barriers, which are delaying the adoption of both remotely operated and UUVs. On the technical side, barriers in operating unmanned systems are mainly related to the adverse conditions of the underwater environment^[4]. On the cultural side, aspects such as a limited understanding of the developed systems and the logic behind the algorithms driving their behaviours are causing a lack of trust from commanders and operators^[2].

The concepts of situational and spatial awareness fall in between such technical and cultural barriers. The trust of the systems, their effectiveness, and thus the guarantees about their performance require: (i) a full understanding of the environment in which an activity is performed, i.e. spatial awareness^[5]; and (ii) the possibility of perceiving elements, comprehending their meaning, and having the ability to project their status in the near future, i.e. situational awareness^[6]. Research & Technology (R&T) is advancing to overcome the aforementioned barriers by improving human awareness and by decreasing the time needed to comprehend the available information when preparing, supervising and analysing underwater missions performed by UUVs. It is also worth bearing in mind that addressing these technical barriers could also help to address some of the cultural barriers that are hampering the adoption of underwater unmanned systems.

These premises of technical and cultural barriers have laid the foundations of this work. This work proposes innovative means of data visualization based on data fusion, artificial intelligence and augmented/ virtual reality^[7] with the objective of achieving improved situational and spatial awareness in support of underwater operations and training when adopting UUVs.

This article elaborates on the analysis of the identified gaps and the proposal of a technological solution to support on underwater operations. Moreover, it focuses on the case study represented by the operative scenario of countering UWIEDs within a harbour protection scenario. NATO recognises shortfalls in this area and the need to develop Concepts of Operation (CONOPS) or Tactics, Techniques and Procedures (TTPs) leveraging on technological innovations and the opportunities offered by the use of unmanned systems and human-machine teaming capabilities^[8][9][10] to support the tasks of detecting, classifying, identifying and neutralising UWIEDs.

The article is structured as follows: Section 2 describes the proposed solution in terms of concept of use and the technological prototype. Section 3 exemplifies the use of the proposal within the operational context of harbour protection and Counter Underwater IEDs, and describes the current state of prototype designed and developed by CMRE to support the underwater operations by means of visualization and the use of VR/AR. Finally, Section 4 draws the conclusions and proposes next steps to take the outcomes of this project forward.

DATA FUSION AND VR FOR ENHANCED SITUATIONAL AND SPATIAL AWARENESS

Acquiring precise situational awareness is paramount for the decision-making process. The amount and variety of formats of information and data presented to the decision maker can hamper this process. In fact, new warfare scenarios are characterised by an increasing volume of information and by a concurrent reduction of time available to make decisions. In this situation, among the challenges faced when trying to improve situational and spatial awareness, it is possible to enumerate challenges in the identification of what information is needed, when it is required, and how such information is shown.

Figure 1: Overview of the proposed solution.

The authors have designed and developed a prototypical solution (see Figure 1) to evaluate the technological concept of improved situational and spatial awareness in the underwater domain. The adoption of advanced visualization techniques based on virtual and augmented reality is the final outcome of a structured process that can be described as follows. First, a set of raw data inputs obtained through the persistent patrolling of the seafloor performed by the UUVs is received and transformed into point clouds or other 3D representable data. Next, the data obtained are filtered to reduce noise and fused in a common reference system. Finally, the filtered and fused data are interpolated and combined into a structured mesh before being sent to the visualization tool to reconstruct the virtual environment.

The following sections describe in depth the three building blocks of the proposed solution: sensors and data, data processing, and data visualization.

Building Block 1 – Sensors and data

Among all the sensors that are available and used in underwater operations in murky and confined water of harbour environments, in particular to detect mine-like objects or improvised explosive devices, it is possible to differentiate three broad categories: imaging sensors, field sensors, and augmenting sensors.

• Imaging sensors use optical or acoustic signals to produce images. They include side-scan sonar, synthetic aperture sonar, micro bathymetry systems, sediment penetration systems or underwater optics and LIDAR. These sensors are designed to observe the surface characteristics of the target objects.
• Field sensors include magnetic field sensors, electro-magnetic sensors and electric-field sensors. Field sensors are designed to measure variations in the constant electro-magnetic field of a given area due to the presence of a magnetic target object.
• Augmenting sensors includes chemical and internal investigation sensors. Augmenting sensors are used in UWIED detection and identification operations to overcome the limitations of visual-based or acoustic-based target detection systems by gathering auxiliary information.

Building Block 2 – Data Processing and Fusion

In the last decades, there has been a great effort in the design and development of methods for automatic target recognition in case of classical mine-like objects. Such methods usually belong to the family of data- driven methods, which are based on the prior knowledge of the characteristic of the threats extracted from historical data. Moreover, due to the different types of input data, such methods integrate image processing, pattern recognition, and data fusion methodologies^[11][12].

In case of UWIEDs, as previously mentioned, no prior information is available. Classical methods, based on the simple collection and fusion of data combined with historical archives, are not sufficient for IED detection and identification. The process requires lateral thinking and reasoning on all the characteristics of the operational theatre, leading to a context-based decision that will always be affected by uncertainty. However, in UWIED scenarios, the combination of techniques as described in the framework proposed in this paper allows the operator to visualize and manipulate all the available information. Consequently, the situational awareness is improved, and the operator is supported in making context-based decisions and thinking laterally.

Building Block 3 – Visualization and Virtual and Augmented Reality

For the development of this building block, the project team has investigated the benefits of providing more immersive environments based on VR/AR. For the sake of clarity, in this work we use the following definitions for Virtual and Augmented Reality:

• VR is defined as the computer-generated simulation of a three-dimensional image or environment that can be interacted within a seemingly real or physical way by a person using special electronic equipment, such as a helmet with a screen inside or gloves fitted with sensors^[13].
• AR is a technology that superimposes a computer-generated image on a user’s view of the real world, thus providing a composite view^[14].

In this article, we talk about VR/AR applications, which can be also viewed as Mixed Reality (MR)^[15] applications. MR is a blend of physical and virtual worlds that includes both real and computer-generated objects combining aspects of VR/AR.

A number of experiments, both in the industrial and research environments, have shown the potential of adopting VR and AR, or the mix of them, for visualizing data gathered within multi-asset or multi-sensor operations. This approach has also proven valuable to increase users’ interaction with the remote mission environment.

VR/AR is a fast-evolving technology. Today, VR/AR military research is devoted to increase the understanding of this technology and demonstrate how it can be adopted for future defence applications. Demonstrations already show that such technology is able to shape the decision-making process. The interest of the military community in VR/AR is underlined by their inclusion in programs such as the Multinational Capability Development Campaign (MCDC)^[16], an ongoing NATO initiative, or the Plan Jericho, which is devoted to implement future high tech solutions in the Royal Australian Air Force^[17].

PROTOTYPE DESIGN AND DEVELOPMENT

As mentioned in Section 1, this work proposes a VR/AR framework aiming at enhancing the maritime situational and spatial awareness. This enhancement aims at supporting an improved analysis of the entire set of data collected during underwater harbour protection operations, by fusing the collected data into 3D-representable models and representing it using virtual and augmented reality in a multi-layer environment (see Figure 2).

Figure 2: Multilayer representation of information.

The data gathered by the available sensors, together with the data produced by ad-hoc algorithms, are collected into a data structure and sent to a visualization tool that enables the virtual presence of the operator within the virtual environment. At this point, the operator is able to appreciate the 3D model of the seabed and any targets, along with all the other calculated metadata that can be used to augment the information being displayed. For instance, the operator can use historical data to perform change detection analysis and identify possible new objects lying on the seabed.

Figure 3 shows a screenshot of the current version of the DAT POW VR/AR interface. For the generation of the data and reconstruction of the underwater environment, the authors have used the data coming from the results of a simulated underwater operation where one or more autonomous vehicles explore the seabed of a port where UWIEDs have been placed and must be identified. The simulation framework used to execute this simulated operation is being developed by the authors^[18].

Figure 3: Screenshot of the VR/AR interface with the results of an operation simulated with the CMRE M&S Federation.

The VR/AR interface was tested in a workshop organised during the French Navy Northern CUTAWAY 2021 exercise. The exercise was organized by the Mine Clearance Diver Unit group of Manche (MCDU Manche) in Cherbourg (France). Figure 4 shows one of the participants testing the VR/AR interface. The feedback from the participants of the usability of the tool was positive.

Figure 4: One of the participants of the workshop testing the VR/AR interface.

CONCLUSIONS

This article summarizes the investigations and developments carried out with respect to the enhancement of situational and spatial awareness in underwater operations using Virtual and/or Augmented Reality tools. Use of these tools can improve the visualization of data and information when conducting operations in harsh and challenging environments such as the underwater case.

The prototypical solution developed addresses both the IED topic and, more relevantly, the lack of trust from the operators preventing the adoption of autonomous and unmanned systems in underwater operations. Human-friendly visualization support tools could be a solution to tackle the cultural barrier and, at the same time, could represent a future aid for operations.

In 2021, the authors had the opportunity to present the solution described here to the operational community. Participants to a workshop conducted during the Northern CUTAWAY 2021 exercise (French, Germany and USA divers and officers) tested the prototype and provided feedback. From an R&T perspective, the prototype seems to address the several operational issues related to the Underwater IED topic and seems to help to address the lack of trust of operators towards autonomous systems. The authors are continuing to work on this project by continuing to evaluate the proposed solutions with the operational community. ■

REFERENCES

1. Yvan R. Petillot, Gianluca Antonelli, Giuseppe Casalino and Fausto Ferreira, “Underwater Robots: From Remotely Operated Vehicles to Intervention-Autonomous Underwater Vehicles,” IEEE Robotics & Automation Magazine, vol. 26, no. 2, pp. 94-101, June 2019.
2. NATO Headquarters Supreme Allied Commander Transformation, Autonomous Systems – Issues for Defence Policymakers, A. P. Williams and P. D. Scharre, Eds., Norfolk, Virginia.
3. Department of Defense, Defense Science Board, The Role of Autonomy in DoD Systems, Createspace Independent Pub, 2015.
4. Geoffrey Ho, Nada Pavlovic and Robert Arrabito, “Human factors issues with operating unmanned underwater vehicles,” in Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2011.
5. Holly A. Taylor and David N. Rapp, “Where is the donut? Factors influencing spatial reference frame use,” Cognitive Processing, vol. 5, no. 3, pp. 175-188, 2004.
6. Mica. R. Endsley, “Toward a Theory of Situation Awareness in Dynamic Systems,” Human Factors: The Journal of the Human Factors and Ergonomics Society, vol. 37, no. 1, pp. 32-64, March 1995.
7. Tremori, A., Viñas, A. C., Solarna, D., Sobrino, P. C., & Godfrey, S. B. (2019, October). Virtual reality and autonomous systems to enhance underwater situational and spatial awareness. In International Conference on Modelling and Simulation for Autonomous Systems (pp. 306-316). Springer, Cham.
8. North Atlantic Treaty Organization (NATO), (NC) AXP-05 EDC V18 E – NATO Experimental Tactics and Amplifying Tactical Instructions, 2019.
9. North Atlantic Treaty Organization (NATO), (NC) AXP-05 Mine Warfare SUPP EDC V14 ENC, 2019.
10. NATO Naval Armament Group (NNAG) Underwater Warfare Capability Group (UWWCG), NATO Future Naval Mine Warfare Vision Paper (to be endorsed), 2020.
11. Thibaud Berthomier, David P. Williams, and Samantha Dugelay, “Target localization in synthetic aperture sonar imagery using convolutional neural networks”. In OCEANS 2019 MTS/IEEE SEATTLE, pp. 1-9, IEEE, 2019.
12. David Williams, “On the use of Tiny Convolutional Neural Networks for Human-Expert Level Classification Performance in Sonar Imagery,” IEEE Journal of Oceanic Engineering, Vol. 46, No. 1, pp. 236-260, January 2021.
13. Zhao, Qinping. “A survey on virtual reality.” Science in China Series F: Information Sciences 52.3 (2009): 348-400.
14. Azuma, Ronald T. “A survey of augmented reality.” Presence: teleoperators & virtual environments 6.4 (1997): 355-385.
15. Costanza, Enrico, Andreas Kunz, and Morten Fjeld. “Mixed reality: A survey.” Human machine interaction. Springer, Berlin, Heidelberg, 2009. 47-68.
16. NATO C2 Centre for Excellence, “MCDC 2017-18 – Information Age Command and Control Concepts,” 24 January 2018. [Online]. Available: https://wss.apan.org/ MCDCpub/MCDC1718/MCDC_201718_Public_Shared_ Documents/MCDC_17-18_Project_Fact_Sheets/MCDC_ 17-18_InfoAgeC2_Project.pdf. [Accessed 25 June 2018].
17. VR Scout, “The Australian Air Force Is Now Testing the Microsoft HoloLens,” 23 January 2017. [Online]. Available: https://vrscout.com/news/the-australian-air-force-is-now-testing-the-microsoft-hololens. [Accessed 25 June 2018].
18. Viñas, A. C., Mansfield, T., Sobrino, P. C., & Tremori, A. “Towards a Modelling & Simulation Capability for Training Autonomous Vehicles in Complex Maritime Operations”, NSMG 2021 Symposium, STO-MP-MSG-184, Amsterdam 2021.

 

ABOUT THE AUTHORS

Pilar Caamaño received the B.S. and M.S. degree in Computer Science from the University of A Coruña (Spain) in 2006 and a PhD. in Computer Science and Artificial Intelligence in 2011. She has more than fifteen years of experience working on research projects focused on the application of Computational Intelligence techniques and Modelling and Simulation (M&S) approaches in the maritime domain. Currently, she is an M&S scientist at the NATO STO Centre for Maritime Research and Experimentation (CMRE) working on the investigation and development of M&S methods to Concept Development and Experimen- tation, Testing, Verification and Validation of underwater autonomous systems or support to decision making.

Arnau Carrera Viñas is a scientist in the NATO STO CMRE. He received the B.S. and M.S. degree in Computer Science from the Universitat de Girona in 2012 and a PhD in 2017. His PhD is devoted to the development of artificial intelligent algorithm, specifically in learning by demonstration methods, to perform underwater intervention tasks by means of autonomous underwater systems (AUV). In 2015 he joined the modelling and simulation team at the NATO STO CMRE. His current research activities are related to the development of complex simulated environments for autonomous systems.

David Solarna received the B.Sc. degree (cum laude), the M.Sc. degree (cum laude), and the Ph.D. degree from the University of Genoa, Italy, in 2014, 2016, and 2021. In 2016, he was a Visiting Researcher with the NASA Goddard Space Flight Center, Greenbelt, MD, USA. From 2017 to 2019, he was a Research Fellow with the NATO STO CMRE,La Spezia, Italy. His area of interest covers data fusion, with focus on image registration, machine learning and pattern recognition methods applied to remote sensing and underwater sonar data, and the combined use of modelling and simulation and autonomous systems in the underwater domain.

Lucia Gazzaneo is a PhD candidate on Industrial Engineering at DIMEG, University of Calabria. She investigates the concepts of Smart Operator and knowledge management in Manufacturing Systems within the Industry 4.0 received her M.Sc. degree in paradigm. She Management Engineering (cum laude) from the University of Calabria. Since 2018, she has been carrying out research activities in Industrial and Logistics systems, including the use and development of simulations for training and decision-making at the Modelling & Simulation Centre – Laboratory of Enterprise Solutions of University of Calabria. She also collaborated with NATO STO CMRE to explore the use of M&S to support Wargaming, Concept Development, Planning, Analysis and Experimentation.

Françoise Perret is the head of the Defence Against Terrorism Programme of Work at the NATO Headquarters. Her studies are on International Public and European Laws.

Claudio Palestini is working in the Emerging Security Challenges Division in the North Atlantic Treaty Organization (NATO). He deals with capability and technological development in a number of fields, including countering drones and other asymmetric threats. Before joining NATO, he has worked in European Union institutions, dealing with the satellite navigation programme Galileo and the Single European Sky ATM Research (SESAR) deployment. He holds a PhD in telecommunication engineering from University of Bologna (Italy).

Alberto Tremori is an Electronic Engineer with a PhD in Modelling and Simulation (M&S). He has more than 24 years’ experience working in industries, academia and research on innovative projects. He is currently project leader at the NATO Science and Technology Organization Centre for Maritime Research and Experimentation (CMRE). He focuses on how blending human, technologies and data to support Decision-Making, he is also working interoperable simulation and serious gaming for concept development, testing and training.

---

Download PDF: Improved Situational and Spatial Awareness for Autonomous Systems in Harbour Protection by means of Virtual and Augmented Reality – C-IED Report SS2022

---