IDSA ISSUE BRIEFS

You are here

Home » Publications » Issue Brief

Military Applications of Virtual Reality and Beyond

Akshat Upadhyay
Lt Col Akshat Upadhyay is Research Fellow, Strategic Technologies Centre at Manohar Parrikar Institute for Defence Studies and Analyses, New Delhi. Click here for detailed profile

More from the author

  • Share
  • Tweet
  • Email
  • Whatsapp
  • Linkedin
  • Print

    Akshat Upadhyay
    September 14, 2023

    Summary

    The use of Virtual Reality (VR) for military applications has been widespread in the US and NATO countries, including for combat training and simulation, as a cultural immersion tool prior to deployment and for stress testing of soldiers. Challenges associated with the use of VR in the national security context include those relating to objective evaluation standards of training protocols, among others. In the Indian context, the use of VR can improve training standards and create a more culturally adaptable professional force.

    Introduction

    Science fiction is credited with introducing novel concepts, which were later translated into real-world applications. Helicopters, submarines, nuclear weapons, the World Wide Web (WWW), liquid-fuelled rockets and humanoid robots are some technological advancements that were initially imagined in the minds of science fiction writers. Virtual Reality (VR) is a similar concept. Apart from being inspired by science fiction writers such as John Brunner (Stand on Zanzibar1 , 1968) and Damien Broderick (The Judas Mandala2 , 1982), it borrows elements from the entertainment industry (Sensorama3 ), academia (The Sword of Damocles4 ), philosophy (phenomenology,5 presence6 ), national security (simulations), psychology (cognitive behavioural therapy or CBT7 ), and medical and information technology (IT). The Brief discusses three broad uses of VR within the broader national security context along with certain challenges.

    VR as a Concept

    VR as a concept can be thought of as an antithesis to physical reality, elements of which are accessible through the five senses. These senses, when mediated through the perceptual, vestibular, autonomic and proprioceptive nervous systems inside the human body, create a sensation known as presence or subjective reality. The interaction of humans with this reality leads to actions and reactions. VR, then, can be defined as any setup that aims to elicit human neural responses using externally and intentionally created synthetic environments, such as that obtained during interaction with the physical world.

    Once an individual is fitted with the requisite equipment, a virtual environment (VE) is created with the intention to stimulate the person’s senses by varying elements within the VE. The VE usually prioritises vision as the major element for situating the individual within the VR. The remaining sensory nodes are activated based on the objective—training, skill acquisition, experience, learning or healing. One of the most important criteria for a person inside VR is to interact with that ‘world’. Passivity or lack of interaction reduces VR to just an entertainment avenue. Jean-Paul Sartre’s philosophy of existentialism is relevant here. A person needs to act in a particular situation—nature, nurture and contextual actions have brought him/her here, and now the only thing remaining is to act. This is similar to what an individual feels inside the VR—the need to act to reinforce his/her personhood.

    VR is also connected to Pavlov’s Dog8 and Skinner’s box9 . Both are premised on isolating an individual (initial tests were attempted on dogs, pigeons and rats) inside a box. This is to prevent any external condition(s) impacting the person’s behaviour. There are movable elements which have to be acted upon. Based on choices of the individual, reinforcement feedback is provided in the form of either food, water or physical discomfort (electric shocks). Within a very constrained environment, the individual is understood to be acting autonomously and his/her actions and reactions are recorded by external observers.

    These actions can range from physical, behavioural, and cognitive to perceptual, physiological and neurological. This method is used to establish a causal link between behaviour and stimuli. These insights can be used for modulating therapy tools in order to treat psychological issues. Jaron Lanier, one of the pioneers of VR technology, has pointed to the dangers of VR, comparing it to Skinner’s box—for VR to work well, it will have to include the best sensing of human activity.10 This also includes the capability of inducing any experience as feedback, creating the vilest manifestation of a digital Skinner’s box. A number of academic studies have shown a causal link between playing violent video games and violence in the physical world.11

    Use of VR in Combat Training and Simulation

    In the military, VR has long been associated with simulators. Initially, this was used as part of a human–machine interface (HMI) for learning flying skills that required a comprehensive setup including VR software and a mockup of the aircraft’s cockpit.12 Known as ‘transfer of training’, the process refers to applying the knowledge and skills acquired during the training process to the targeted job or role.13 The aim of transfer of training is to use the modified reaction system of the body which can be later applied to similar situations in the real world. The flight simulator paradigm for training soldiers has been extended into the Navy and the Army.

    The United States Defense Advanced Research Projects Agency (DARPA) created a simulation network (SIMNET) program that ran from 1983 till 1989.14 Labelled the Tank Team Gunnery Trainer, the setup consisted of multiple networked ‘workstations’ modelled on the inside of tanks, where gunners could interact with each other and manoeuvre on a virtual battlefield in real-time. Later this setup was extended to simulating combined arms operations using mockups of armoured personnel carriers (APCs), helicopters and other platforms.

    A set of standards called the Distributed Interactive Simulation (DIS) protocols have also been developed by DARPA. DIS allows for the networking of simulators, semi-automated forces (SAF) and analysis tools for geographically disparate forces.15 One of the key advantages of the use of standard protocols is the agreement on operating in a shared reality. Disagreements may arise on how much the VR corresponds with physical reality, but since the same setup is shared by everyone on the network, the rules and constraints are similar for everyone. This means that everyone views one standardised version of reality—in essence, any major or minor deviation can be averaged away.

    In this respect, the 2003 North Atlantic Treaty Organisation (NATO) Report on VR and Military Applications is instructive.16 The report singles out man–machine interfacing as the most important aspect of VR and contends that safety and cost considerations are the reasons why VR is being used in militaries worldwide. Here another aspect of innovation may also be added since VR provides an avenue of experimenting with various permutations and combinations of doctrines and tactics with zero physical or financial cost to the individual or the exchequer.

    VR’s unique use for studying the impact of nuclear weapons on individuals and equipment, modelled and depicted as a VE, has no real-world equivalent. Scenarios similar to these are nigh impossible and morally unimaginable to simulate in a lab or on the ground. The report further calls for the integration of computer models, simulated scenarios, behavioural models of flying platforms and weapon systems, simulators and other essential warfighting systems. Weapons manufacturers use VR to find gaps in the 3D environment prior to experimental assembly, for example, as in Sikorsky's CH-53 heavy lift helicopter program simulator.17 Other military applications of VR include CT/CAT scans, tele-robotics, psycho-social interventions, and treating PTSD18 (replication of war zone sensory experience that the therapist can control).

    The most useful part of VR is immersion, defined as the presentation of an “artificial environment that replaces users’ real-world surroundings convincingly enough that they are able to suspend disbelief and fully engage with the created environment”.19 While it provides users with an advanced synthetic environment (ADS)20 , engaging their motor and cognitive skills, eliciting nervous system responses which are then measured and manipulated for the job required, the user feedback enables external observers to make amendments to the relevant aspects of VR in terms of prioritisation of reactions. For example, some VR systems specialise in understanding complex systems by magnification.21 They zoom in or slow the operational speed of mechanisms of equipment like turbine motors and gun mechanisms. Some require users to act in the environment in a manner that resembles tasks in the real-world setup, with an expectation that the skills acquired in VR can be seamlessly transferred to tasks in the physical world.

    One of the initial practical uses of VR has been in the military domain. Taking India’s example, VR is used extensively for weapon-oriented training, for example, platform simulators such as aircraft22 , surface-to-air missiles (SAMs)23 and aircraft carriers24 . It is also used for blueprint design of ships by the Directorate of Naval Design (DND)25 and for technical and small team training.26 However, these form part of the holistic training cycle and isolating the importance of VR in the context of the broader training curriculum is difficult to ascertain due to the challenges of measurement and empirical data.

    The scientific way of using VR in combat training is to first isolate the exact skills required to be acquired and/or honed, then create VR models for immersing the user and focusing on that very skill. This is comparatively easier for platforms such as aircraft, carriers and tanks but considerably challenging for the infantry or combined arms formations, where judgement calls or extensive coordination is required. For the latter, only certain aspects are used in VR such as communications, electronic warfare (EW) etc.

    Individual training has also been attempted using VR where the focus is on improving the soldier’s judgements and weapon-handling skills, rather than being instrumentation-centric. For example, a study was undertaken in Taiwan to use a game-based learning environment to improve shooting scores of children utilising a mix of a 1:1 real scale mockup of a rifle with recoil effects, connected to a simulator, infrared (IR) technology and 3D interactive VR (IVR). The results showed that the best results were obtained for users who were trained in a combat training mode (moving + static firing and hitting the target precisely) in addition to firing range mode in 3D IVR27 . Some military commentators are of the opinion that serious games when combined with realistic military training have been shown to enhance the fighting skills of US military recruits and protect them from mental stress.28 Some academics have critiqued the method of training soldiers using VR, finding only limited empirical evidence of the effectiveness of computer game-based training for soldiers.29

    Here the difference is between 2D video games and 3D interactive ones involving HMDs, the former deemed to have lesser effectiveness than the latter. In fact, as per a study conducted by the United Kingdom (UK’s) Ministry of Defence (MoD) on 39 dismounted Royal Air Force (RAF) soldiers on shoot/don’t shoot decision-making simulations, it was discerned that live fire and 3D VR posed effective decision-making dilemmas on the soldiers, effectively working to exercise decision-making processes under stress. On the other hand, 2D video presented little decision-making challenges.30

    One of the more interesting findings of the study was the concern regarding effective testing and calibration of virtual environments (VEs) before being deployed. Though fidelity of the VE with respect to the physical environment was a factor, it was adjudged through the trainee’s self-reported presence ratings.31 This translates to a subjective view of the VE, which in turn means that the VE may have to be calibrated individually to suit an individual’s subjective perceptual capabilities. Though the study further concluded that the direct benefits of increase in the sense of presence within the VR were unclear, it made positive observations regarding the use of VR to contribute towards a more realistic task behaviour and task engagement. Additionally, in VR-based scenarios, there are certain unnecessary stimulations that won’t be available in real-life in the same format. These have to be kept in mind.

    Interestingly, there is a debate about the kind of graphics required to induce a sense of presence. As discussed above, presence is a subjective perception. However, it is still not clear as to what are the critical components that go into generating presence in individuals. Two approaches are generally considered. One is data and compute heavy and which hews to high fidelity between the VE and the real world so that both are indistinguishable from one another. This is hardware-intensive and may cost significant financial resources in terms of energy, data and storage consumption. This is the approach that the US Army has gone for.

    As part of its Synthetic Training Environment Cross-Functional Team (STE-CFT) under the Army Futures Command, the US Army will deliver three major programs to its soldiers—One World Terrain (OWT), Reconfigurable Virtual Collective Trainer (RVCT) and the IVAS-Squad Immersive Virtual Trainer.32 While the scope of OWT is massive, it has already delivered close to “two million square kilometres of 3D datasets”33 to a number of countries and agencies, it is also not limited to training. OWT intends to “virtually replicate Earth ’s terrain” and “simulate the complexities of the operational environment”.34 By doing so, it not only aims to train its soldiers pre-deployment but also assist in intelligence preparation of the battlefield, feed datasets into autonomous systems, among others. The grand unifying aim is to create a common operating environment for all services as part of the Combined Joint All Domain Command and Control (CJADC2). RVCT is intended for VR training of ground soldiers with army aviation, while the IVAS project is being jointly developed with Microsoft.35

    The second approach to thinking about graphics is that of using the knowledge of the perceptual system to find out what is relevant for human representations of reality and then design VR systems for exactly those attributes, thereby saving on compute, data and sophisticated hardware. As per a study, increased fidelity of terrain representation does not increase the overall understanding of the terrain in a simulated mission planning environment.36 Called the ‘proximity compatibility principle’, it states that the actual features of the display are less important than how those display elements map to the mental model.37

    Take, for an example, an infantry battalion commander trying to understand or appreciate an attack or defence plan on a given terrain. Populating the terrain with near-real life features is considerably less important rather than displaying the comparative topological attributes such as height, slope, etc. A higher fidelity representation may add a sense of aesthetics to the VE but will not aid in formulating a military plan better. In areas such as the Line of Control (LC) or the Line of Actual Control (LAC), the ability of a battalion commander to visualise his entire attack plan on an actual terrain representation along with his subordinate commanders and important appointments by being networked in VR, ‘visiting’ the locations and taking a bird’s eye view, a ‘God’s eye view38 ’ of sorts, will be unprecedented. As per some news reports, the same is being test-bedded within certain Indian Army formations in Jammu and Kashmir (J&K)39 .

    The replicability of training using VR is another critical aspect that needs to be studied in detail especially when ‘presence’ affects the subjective training experience of each soldier individually. There is a major difference between using VR for practice (where a certain rudimentary skillset is assumed) and training (where the skill set is assumed to be non-existent). Most VR researchers working for the military are of the opinion that VR, as currently used within the military is used for the former rather than the latter. Even in the previous RAF example, the soldiers’ arms handling capabilities are assumed and only their judgmental capacity is being tested/ practised.

    It is therefore necessary to think of training using VR on a broader basis. Specific areas where VR has been extremely successful are weapons training, specialist training on systems and sub-systems of major platforms and skills acquisition in fields requiring fine motor-balancing. In the last case, VR is used to initially slowdown actions allowing individuals to gradually build up proficiency. Based on these studies, a model and sequential training paradigm is given below:

    IT skills (pre-requisite)—combat-physical skills (individual)—combat cognitive skills (individual)—combat small team skills (physical + cognitive + relational)—increase size of interaction till it assumes a state of homeostasis with respect to the appointment and nature of the duty of the individual—combined arms skills (physical + cognitive + relational + contextual)—multi-domain skills.

    VR as Cultural Immersion Tool for the Military

    VR’s importance, apart from training and therapeutic purposes, is important from a cultural point of view, especially for the Indian military which has increased its interactions with a number of friendly and partner countries, emphasising the increasing role of defence diplomacy in the overall national security structure.

    Imagine that an Indian battalion group is in the process of being deployed in a conflict-ridden zone as part of a UN peacekeeping operation (UN PKO). The unit has gone through the rigour of classes, lectures and demonstrations on the military, political, historical and cultural aspects of the conflict in question. However, the knowledge gained is second and third-hand, handed over via batches of case studies and prior experiences. Language acquisition is also passive. Since there is no interaction with any individual from the host country, most customs, traditions and even psychologies of host country are viewed from the perspective of the soldier’s own cultural mores. In certain areas, these may converge, in most they don’t.

    Therefore, a form of cultural immersion of the soldiers is required, or most personal interactions may hover at the extremes of deference or irreverence. This cultural immersion can be done now using VR and intelligent virtual agents (IVAs).40 Moreover, generative artificial intelligence (AI) when embedded within these IVAs will enable dynamic interactions within the VR. A start has already been made with some video-games.41 VR for cultural immersion has been used by the US for deployments in Iraq and Afghanistan42 for learning language, developing a sense of empathy for civilians.

    This also relates to the difficulty of conflating communication with translation and interpretation.43 Due to the difficulties of being ‘lost in translation’, there is a dire need for effective language-based skills training for soldiers who can not only understand the host country’s citizen’s dialects, but also convey their own actions and responses, in accordance with the broader goals of the UN mission and in line with their own country’s values and norms, seen as they are ambassadors for their country.

    In fact, VR systems can be used for practising with foreign delegations on joint exercises, arms negotiations and other important policy issues. The US military, due to a long-term culture of empiricism and technological solutionism, has devised multiple VR-based simulations for the same.44 The University of Southern California created Tactical Iraqi in 2004, an immersive VR whose aim was to teach Arabic language to soldiers being deployed to Iraq.45 Similarly, Virtual Environment for Operational Readiness (VECTOR) has been used to train soldiers in cultural familiarisation.46 Such scenarios are now being implemented in mission rehearsal exercises (MREs) and programs such as Full Spectrum Warrior (FSW), Sensory Environment Evaluation (SEE) Project Dark Con, Every soldier a sensor simulation (ES3).47

    While FSW focuses on threat analysis and soldier awareness in a foreign setting, the SEE project tested Dark Con as a single-user VR which created “emotionally compelling VEs”, testing the role of emotions in the future of army training. ES3 trains soldiers in collecting human intelligence (HUMINT) and conducting presence patrols48 (area domination patrols or ADPs in the case of Indian Army). All these are important systems that aim to train the soldier in conducting his/her military duty in a foreign land and at the same time, being aware of the different culture sensitivities obtained in the region.

    VR for Stress Testing of Individuals

    The third most important use of VR is for stress testing and adaptation of soldiers pre-deployment as well as dealing with issues of PTSD. VR can also be used as a stress management tool for peacetime postings in the military. It is understood that stress management training (SMT) is a holistic concept for relaxing and developing ‘cognitive coping skills’ for preventing and managing stress.49 It has two basic approaches. The first is known as stress inoculation training (SIT), which is similar to battle inoculation, simulating battlefield conditions with adequate safety precautions. SIT comprises stress tolerance through exposure to reduce the novelty of stressors, manage uncertainty and maintain high levels of performance during operational deployments. VR’s use as part of SIT is under the assumption that VR can also elicit emotions related to real-life stressors.

    The second is resilience training (RT), based on the view that it is not the event that causes the emotion but how a person appraises that event which is intertwined with the emotion. So, focus is on teaching coping skills to soldiers. The US Department of Defense (DoD) has created a program called the Comprehensive Soldier Fitness (CSF) program that treats fitness as a holistic concept which is dependent not only on the soldier but also on his/her family. The program uses “individual assessments, tailored virtual training, classroom training and embedded resilience experts”.50

    VR is also used for soldiers suffering from PTSD. One of the most influential theories used in the construction of treatment protocols for PTSD using VR is ‘emotional processing theory’. It states that pathological fear structures are activated when information represented in the structures is encountered. These fear structures are made of harmless stimuli associated with danger.51 VR exposure therapy (VRET) involves a mix of CBT with prolonged exposure (PE) where multi-sensory cues and context-relevant cues are introduced in a controlled manner in order to enable the patient to access his/her experience through effortful memory retrieval.52 VR applications like Virtual Iraq and Virtual Afghanistan have been used to treat PTSD and have been modified and improved using user feedback.53

    Challenges of Using VR

    Apart from the technical challenges of hardware and software, one of the major challenges of using VR in the military is that VR is not just the hardware-mediated world of synthetic environments. It is also how humans are setup to be connected, for instance, social media platforms and online video games can also be considered as VR. They scramble relations and connections through digital links and create alternate realities. The actions performed through these alternate realities have real world effects. The notion of self-preservation is put aside since the virtual connection dominates the senses.

    As a result, physical violence—as understood in its most visceral form—or more broadly, a militant mindset, may become only tools. Since VR constrains behaviours in the VE, such constraints may be transferred to the physical world and scenarios which may benefit from a diplomatic or empathetic approach, may not figure in the arsenal of soldiers. This has not been tested so far on soldiers due to the redundancy involved in testing for violent behaviour on individuals who are ‘trained to kill’. On a parallel level, for young adolescents, a number of studies have established a positive correlation between the time spent on playing video games and increasing violent behaviour. These have to be kept in mind.

    A second challenge relates to the evaluation of the transfer of training from the VE to the physical world. One of the main limitations of using VR for the many purposes mentioned above is its seemingly constrained use during wartime. It is difficult to isolate VR and evaluate its effectiveness to the entire operations and training processes. This also requires an empirical and technologically oriented organisational culture where measurements, records, feedback and human-oriented design form the basis of training.

    A third challenge pertains to understanding the scope of human performance to be realised within the VE and comparing it with the limitations of the VE and presence. Again, a number of subjective factors are at play here and therefore, subjectivity is introduced by the way of different individual experiences. A minimal or baseline training objective needs to be decided upon, which results in a positive feedback with respect to the training requirements, both in terms of time and financial investment.

    Conclusion

    The use of VR has been widespread in the US and NATO countries and involves a host of VR programs, either in conjunction or without the use of simulators. One of the major reasons is an organisational culture that focuses on empiricism, data generation and experimentation. A similar case can be also made for the use of VR for the Indian Armed Forces due to its many uses, known as well as unanticipated. The use of VR and similar technologies for training, medical and combat, as well as cultural and other uses, however, requires a change in orientation from a purely intuitive to a mix of intuition and empirical and technologically oriented mindset. If done in a deliberate fashion, the use of VR can improve training standards for the organisation and usher in much more culturally adaptable professionals for a force which is deemed to play a bigger role on the world stage.

    Views expressed are of the author and do not necessarily reflect the views of the Manohar Parrikar IDSA or of the Government of India.

    AttachmentSize
    Download Complete [PDF]749.74 KB
    Keywords: 
    Defence Technology
    Military Modernisation

    Top