Project

Augmented Reality with X-Ray Vision

Copyright

Signal Kinetics

Signal Kinetics 

X-AR is an augmented reality (AR) system that gives humans "X-Ray Vision"

 X-AR is a new AR headset that enables users to see things that are hidden. This new headset helps users find items that are inside closed boxes, under piles, or behind occlusions.  The headset uses holograms to guide users toward desired items and verify when they have picked them up. 

How does it work?

X-AR uses wireless signals and computer vision to enable users to perceive things that are invisible to the human eye (i.e., to deliver non-line-of-sight perception). It combines new antenna designs, wireless signal processing algorithms, and AI-based fusion of different sensors.

This design introduces three main innovations:

1)  AR-conformal wide-band antenna that tightly matches the shape of the AR headset visor and provides the headset with Radio Frequency (RF) sensing capabilities. The antenna is flexible, lightweight, and fits on existing headsets without obstructing any of their cameras or the user's field of view.

Copyright

Jimmy Day

2) AR-Based SAR localization algorithm that fuses RF sensing from the antenna and visual sensing from the headset's cameras to localize RF-tagged objects, even when they  are hidden from the view, as the user naturally walks in the environment.

Copyright

Signal Kinetics


3) RF-Visual Verification Primitive that fuses RF and vision to deliver actionable tasks to end users such as picking verification.

Copyright

Signal kinetics

How well does X-AR work?

X-AR is very accurate:

  • It can locate hidden items with an accuracy of less than a foot (9.8 centimeters).
  • It achieves over 95% accuracy in picking verification.

These results show that X-AR is successful in extending AR systems to non-line-of-sight perception, with important implications to warehousing, retail, e-commerce fulfillment, and manufacturing applications.

To learn more about X-AR, check out our paper, which was published at the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2023):