Augmented reality visualization system

• US 10,114,127 B2
• Filed: 05/11/2016
• Issued: 10/30/2018
• Est. Priority Date: 05/11/2015
• Status: Active Grant

• Current Assignee: the united states of america as represented by the secretary of the navy
• Original Assignee: the united states of america as represented by the secretary of the navy
• Inventors: Brown, Christopher A, Juhl, Matthew
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US15/152,533
• Publication Number:
  US 20160379414A1
• Time in Patent Office: 902 Days
• Field of Search: 345633
• CPC Class Codes: F41G 3/02 using an independent line of sight
• CPC Class Codes: F41G 3/06 with rangefinder rangefinders per se G01C
• CPC Class Codes: F41G 3/142 based on observation of a first shoot; using a simulated shoot training F41G3/26
• CPC Class Codes: F41G 3/165 using a TV-monitor
• CPC Class Codes: G01S 19/47 the supplementary measurement being an inertial measurement, e.g. tightly coupled inertial
• CPC Class Codes: G01S 19/53 Determining attitude
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G02B 27/0189 Sight systems
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

An augmented reality (AR) system comprising a head mounted display (HMD) configured to display one or more AR visualizations within an operator'"'"'s field of view (FOV), a control system including a processor and a storage system configured to store machine readable instructions, sensors configured to determine at least location and/or orientation of said sensors including a head mounted and device mounted sensor, and a communication system configured to communicate data between elements of the AR system. The software including various subroutines or machine readable instructions including an orientation/location instructions for determining orientation and/or position of the sensors, a visualizations generation instructions section configured to generate a visualization showing an aim point of a device coupled to said device mounted sensor, a path of travel of a projectile launched from said device, or an impact point of said projectile. Embodiments can include one or more photogrammetry processing sections.

Claims

1. An augmented reality (AR) system comprising:

a head mounted display (HMD) comprising a visual display system and a head mounting structure adapted to fix the HMD with respect to a human operator'"'"'s field of view (FOV), and a camera system oriented towards said FOV configured to capture a plurality of video images of entities, elements or objects within the FOV;

a plurality of sensors comprising a head mounted position sensor and a first position sensor receiver positioned on a first device, said plurality of sensors comprising one or more inertial navigation units (INU) or an inertial measuring unit (IMU), said INU comprising a plurality of accelerometers, said IMU comprising an INU and a global position system (GPS) configured to determine a location of the GPS system, wherein said INU and IMU are configured to determine an orientation of at least a first device and each said sensor is coupled with an output of said location as an orientation data;

a laser rangefinder mounted to said first device used to measure distance between projectile launcher and impact point, or moving object and intended flight path;

a control system comprising a storage medium comprising a plurality of machine readable instructions operable to control elements of said AR system, a processor adapted to execute said plurality of machine readable instructions, said control system is configured to control said AR system, said sensors, and said HMD;

a communication system configured to couple said HMD, said sensors, and said control system, said communication system is further configured to transmit electronic data between said HMD and a remote source using a radio frequency system or a laser data transfer system;

wherein said plurality of machine readable instructions comprises;

a first plurality of instructions configured to operate laser range-finder and ballistic trajectory calculation system as well as an impact point determination system based on at least a first, second, and third data associated with at least one said first device, wherein said first data comprises an aim point of said first device, said second data comprises a computed impact point of a projectile fired from said first device, said third data comprises a path of travel of said projectile from a point of origin from said first device to said impact point, wherein said first, second and third data each comprise one or more geo-reference system position coordinates;

a second plurality of instructions configured to control said processor to operate said plurality of sensors to generate data used by said control system to determine three dimensional orientations of each of said sensors based determinations of location of the devices relative to at least the head mounted position system (HMPS) and said device mounted position sensor;

a third plurality of instructions configured to operate a video processing system to determine and create a wire frame model data of objects within the FOV based on the plurality of video images captured by the camera system, using a photogrammetry processing system

a fourth plurality of instructions configured to control said processor to operate said HMD to generate and display on said HMD a plurality of AR visualizations comprising a first visualization within said FOV of said operator comprising at least a first, second, third, and fourth visualization element, wherein said first and second visualization elements are generated based on a first, second, and third plurality of instructions outputs, said first visualization element comprises a visual designator showing said aim point on a surface defined by said wire frame model data, said impact point on said wire frame model data, or said path of travel with respect to at least a portion of said wire frame model data, said second visualization element comprises a visual damage overlay displaying a predicted blast radius associated with a projectile fired from first device, said third visualization element comprises a visual overlay highlighting or designating of a plurality of entities, elements or objects, said fourth visualization element comprises a plurality of metadata associated with said plurality of entities, elements or objects highlighted or designated by said third plurality element; and

a fifth plurality of instructions configured to provide a graphical user interface (GUI) control interfaces enables said operator to select and control one or more of said plurality of visualization elements, wherein a cursor control software module generates and displays a cursor on said HMD, wherein said operator is able to manipulate said cursor to interact with said plurality of visualization elements.

2. The AR system of claim 1, wherein said point of origin is a muzzle of a projectile launching system.

3. The AR system of claim 1, wherein the AR system further comprises of an augmented reality glove, wherein said AR glove is a wearable device worn on an operator'"'"'s hand comprising of a plurality of sensors configured to generate a plurality of cursor control instructions based on a plurality of specific motions of said AR glove, wherein said AR glove is coupled with said communication system to transmit said plurality of cursor control instructions from the AR glove to said cursor control software module which generates and displays the movement of said cursor correlating to said plurality of specific motions of said AR glove.

4. The AR system of claim 1, wherein said laser rangefinder and ballistic trajectory calculation system adjust the aim point, impact point and path of travel based on a plurality of weather data obtained from a local source or a remote source using said communication system.

5. The AR system of claim 1, wherein said communication system configured to couple elements of the AR system transmit signals wirelessly or wired.

6. The AR system of claim 1, wherein said fourth visualization elements are generated and displayed on the HMD include detailed descriptions, accuracy of object recognition, or any other pertinent information associated with identified entities, elements or objects whenever said identified entities, elements or objects are within the FOV of the HMD.

7. The AR system of claim 1, wherein said fifth plurality of instructions enables the operator to select a single entity, element, or object from a plurality of entities, elements or objects within the FOV of the HMD to distinguish said entity, element, or object using said third visualization element, and display pertinent information associated with said single entity, element, or object using said fourth visualization element, wherein a single entity comprises a particular individual within a crowd.

8. The AR system of claim 1, wherein said third visualization element further comprises a shape recognition software system that enables distinguishing, identifying, and tracking a plurality of entities, elements, or objects.

9. The AR system of claim 1, wherein said first, second, third and fourth visualization elements are either stored on said storage medium within said control system or remotely transmitted from a remote source to the HMD using the communication system.

10. A process for operating an augmented reality control system comprising:

operating a laser rangefinder and ballistic trajectory calculation system comprising computing ballistic trajectory and impact point of a projectile or moving object based on a first, second, third, and fourth ballistic solution data, wherein said first ballistic solution data comprises a laser range finder output, said second ballistic solution data comprises a weather data associated with an environment a projectile launcher or moving object is operating within, said third ballistic solution data comprises a weapon angle data obtained from a position sensor mounted on the projectile launcher or moving object, said fourth ballistic solution data comprises a ballistic or flight performance characteristics of said projectiles or moving objects;

operating a position tracker system comprising of measuring a relative position between a projectile launcher'"'"'s launch axis or a moving object'"'"'s flight orientation with respect to a three axis frame of reference and a HMD camera'"'"'s or a set of HMD camera'"'"'s field of view (FOV) orientation with respect to the three axis frame of reference based on a first and second plurality of outputs respectively from a plurality of position tracking nodes comprising a first and second position sensor respectively mounted on said HMD and said projectile launcher or moving object;

executing a photogrammetry process comprising using the HMD camera or set of HMD cameras mounted on the HMD to capture a plurality of images to create a wire frame model data structure to compute offset measurements of a plurality of objects within said plurality of images by using geometric triangulation computations to determine distances between said plurality of objects and calculating range by determining angle offset between said plurality of objects from at least two said plurality of images, wherein the range and distance data are saved into a wire frame model object entities data structure;

generating visual overlays for display over a HMD FOV comprising receiving inputs from said laser rangefinder and ballistic trajectory calculation process, said position tracker process, and said photogrammetry processes and computing parallax for a line of sight within said HMD FOV versus the point of impact, calculating the impact within the wire frame model data structure, generating shaped overlays including occlusions based on the wire frame model data structure, and outputting generated said video overlays to said HMD that are positioned and aligned with said HMD FOV; and

projecting the video overlays on said HMD.

11. A method as in claim 10, wherein said first and second position sensors each comprise a three axis inertial navigation unit (INU).

12. A method as in claim 10, wherein said operating the laser rangefinder further comprises lasing a target with said laser rangefinder and calculating a range based on travel time of a laser range finder output.

13. A method as in claim 10, wherein said position tracker system further comprises interrogating said plurality of position tracking nodes, reading signal strength for said plurality of position tracking nodes, calculating a relative node position by triangulation based on differences of signal strength between plurality of position tracking nodes.

14. A method as in claim 10, wherein a damage overlay is generated based on the shaped overlay, wherein the shaped overlay is formed based on the predicted blast radius of an ammunition associated with said fourth ballistic solution data, wherein said fourth ballistic solution data further comprises of a plurality of ammunition data.