Introduction
Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real-world are “augmented” by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory. The overlaid sensory input can be constructive (i.e. additive to the natural environment) or destructive (i.e. masking of the natural environment) and is seamlessly registered with real world spatial coordinates.
AR enhances one’s current perception of reality, whereas virtual reality immerses the user in a simulated reality. Both technologies function on a spectrum and often rely on similar scientific and engineering principles. Applications of AR technologies have included special effects in films, interactive 3D modeling and architectural planning. The applications of AR technologies are vast and are slowly spreading across almost all consumer and enterprise domains.
AR has continued to attract tremendous attention from researchers belonging to the domains of computer graphics, human-computer interaction and computer vision aiming to develop novel techniques to realize seamless AR experiences. This paper aims to provide a comprehensive review of research conducted in developing fundamental technologies and applications of augmented reality over the past few decades. The paper is organized as follows: Section 2 discusses early research on developing fundamental technologies for AR. Section 3 elaborates on the advancement of different tracking and display technologies. Section 4 touches upon key milestones in developing AR applications. Section 5 delves into emerging research topics and challenges.
Fundamental Research on AR Technologies
The main goals of the early AR research in the late 1980s and 1990s were to develop the basic principles and algorithms that enable the registration and overlaying of computer-generated 3D virtual models onto live direct or indirect real world views. One of the earliest systems developed for AR was presented by Tom Caudell and David Mizell at Boeing in 1990 known as the ‘Augmented Reality System’. It used see-through head-worn display (HWD) systems to overlay AR annotations like schematic diagrams of wire bundles onto Boeing engineer’s live view of aircraft cabling. It demonstrated the potential of AR for hands-free interactive help for complex physical tasks.
In the 1990s, significant research was conducted by researchers like Ronald Azuma, Steve Feiner and colleagues at Columbia University, Hirokazu Kato and Mark Billinghurst at HIT Lab New Zealand and others to develop the basic principles, algorithms and prototypes for various fundamental AR technologies. Key areas of research included realistic alignment of virtual information with the user’s view of the real world, natural interaction modalities, 3D tracking techniques and wearable see-through display technologies.
Important advancements made in these early days include the concepts of optical and video see-through AR displays. Optical see-through (OST) displays reflect computer graphics into the user’s eyes using semi-transparent mirrors or beam splitters thus allowing the user to see the real environment along with the overlaid graphics directly. Video see-through (VST) displays use video cameras to capture live images and overlay graphics on them which are displayed instead of directly viewing the real environment.
Significant progress was also made in developing effective user tracking techniques which estimate the user’s location and orientation in real-time through inertial, magnetic or optical methods. They are crucial for correctly registering the virtual information onto the live view. Early techniques experimented with systems like inertial and magnetic trackers, optical tracking using markers or natural features for real-time tracking of the user’s head position.
Advancement of Tracking and Display Technologies
Over the past two decades, tracking technologies for AR have grown tremendously sophisticated. Modern inside-out tracking systems leverage advanced Machine Learning algorithms to allow highly accurate 6DoF tracking without any external infrastructure. Popular techniques used augmented reality applications now include visual-inertial odometry (VIO) and simultaneous localization and mapping (SLAM). VIO leverages data from onboard cameras and inertial measurement units to estimate headset motion. SLAM builds and maintains an actively updated model of the surrounding environment for robust tracking.
Display technologies for AR have also advanced greatly. While early AR systems used bulky HWDs, current technologies feature compact wearable designs. Optical see-through displays have transitioned from simple video mixing approach to advanced waveguide-based designs. Companies like Magic Leap, Microsoft HoloLens and others have launched innovative mixed reality smartglasses using waveguides.
Advancements in display resolution, Field-of-View (FOV), brightness and form factors have improved the perception quality and immersiveness of AR experiences. Displays with resolutions over 4K and FOVs exceeding 40 degrees now exist. Companies are exploring projections, micro-OLED and holographic technologies that offer higher resolutions, lower power consumption and wider FOVs. Lightfield and holographic displays show promise for future AR systems with true lightfield rendering. Overall, mature tracking and display technologies now enable robust standalone and untethered AR experiences.
Key Application Areas and Milestones
With the advancement in core AR technologies, there has been immense progress in developing applications across domains. Major application areas can be broadly categorized as below:
Industrial training and maintenance: Early adoption areas where AR helps provide interactive procedures, manuals and simulations for training workers and remote experts. Popular applications include automation, manufacturing, aviation, etc.
Gaming: Prominent examples include Pokémon GO’s widespread success. AR games offer novel engagement through blending of digital content with the physical world.
Education and collaboration: AR-based teaching simulations are widely used. Remote AR tools are enhancing collaboration in design, engineering, construction, etc.
Retail and E-commerce: AR technologies are transforming shopping by allowing consumers to visualize, interact with and customize products digitally.
Healthcare: Applications involve medical imaging, surgical planning and simulations, remote assistance, treatment and post-operation care through AR visualizations.
Some of the key technological and commercial milestones achieved in developing mature AR applications include Google Glass (2012), Microsoft HoloLens (2015), DAQRI Smart Helmet (2016), Magic Leap One (2018) and Meta Quest Pro (2022). While earlier devices focused on education, healthcare and enterprise use cases, recent products are bridging the gap between consumer and commercial applications of AR.
Emerging Research Topics and Challenges
Major areas of ongoing AR research include overlay accuracy and registration fidelity, real-time 3D reconstruction, photorealistic rendering, novel interaction paradigms, scalable shared experiences, privacy issues, standardization and cross-reality platforms.
Challenges such as limited computing power are being addressed through accelerated graphics on specialized processors, foveated rendering and edge/cloud computing. Researchers are also exploring machine learning techniques involving deep neural networks to efficiently handle tasks like occlusion handling, scene understanding, predictive rendering and object recognition.
Standardization efforts through groups like WebXR and cross-platform frameworks like Unity, Unreal Engine and Apple ARKit/Google ARCore are helping develop applications portable across different devices. Long-term challenges remain around developing lightweight, see-through displays with adequate resolution and field-of-view while being aesthetically desirable as standard glasses or contact lenses.
Overall, continued advancements in fundamental technologies, applications and experiences are expected to drive rapid adoption of AR across both enterprise and consumer spaces over the coming decades. AR is projected to have over a trillion dollar impact on the global economy by 2040 according to various forecasts. This research aimed to provide an overview of the rich evolution of AR technologies and applications over the past 30+ years through a chronological discussion of academic research and tech milestones. Much progress still remains but the future of AR indeed looks very promising.
Andrew Stroud