Techniques for accurate pose estimation

• US 10,304,199 B2
• Filed: 09/14/2017
• Issued: 05/28/2019
• Est. Priority Date: 08/22/2014
• Status: Active Grant

• Current Assignee: Applied Research Associates Incorporated
• Original Assignee: Applied Research Associates Incorporated
• Inventors: Menozzi, Alberico
• Primary Examiner(s): Salvucci, Matthew
• Application Number:
  US15/704,934
• Publication Number:
  US 20180018781A1
• Time in Patent Office: 621 Days
• CPC Class Codes: G01C 21/1654 with electromagnetic compass
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0181 Adaptation to the pilot/driver
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

The described technology regards an augmented reality system and method for estimating a position of a location of interest relative to the position and orientation of a display, including a retroactive update to a previously rendered estimate of the position of the orientation of the display stored in a rewind buffer. Systems of the described technology include including a plurality of sensors, a processing module or other computation means, and a database. Methods of the described technology use data from the sensor package useful to accurately render graphical user interface information on a display.

Claims

1. A method of enhancing the accuracy of pose estimation of a display, comprising the steps of:

receiving position/orientation update data for a first time epoch;

storing the position/orientation data for the first time epoch in a rewind buffer;

updating, by a processor and based on the position/orientation update data, an Extended Kalman Filter-based estimation of a position and orientation of the display;

predicting, by the processor and based on the updated Extended Kalman Filter-based estimation of the position and orientation of the display, a future position and a future orientation for the display for the first time epoch using the Extended Kalman Filter;

after the expiration of the first time epoch, receiving absolute position/orientation data for a second time epoch, the second time epoch preceding the first time epoch;

retrieving, from the rewind buffer, the position/orientation update data for the first time epoch;

reupdating, by the processor and based on the absolute position/orientation data and the position/orientation update data, the Extended Kalman Filter-based estimation of the position and orientation of the display; and

repredicting, by the processor and based on the absolute position/orientation data, the position/orientation update data, and the updated Extended Kalman Filter-based estimation of the position and orientation of the display, the future position and the future orientation for the display for the first time epoch using the Extended Kalman Filter.

2. The method of claim 1, further comprising the steps of:

receiving, from a camera, video data for the first time epoch;

displaying, on the display, the video data for the first time epoch;

receiving geodetic coordinates of a point of interest for the first time epoch;

overlaying, on the video data on the display, indicia of a location of the point of interest based on the geodetic coordinates and the future position and the future orientation for the display for the first epoch.

3. The method of claim 2, wherein the point of interest corresponds to a obect in motion.

4. The method of claim 1, wherein the absolute position/orientation data is received from an absolute position/orientation sensor selected from the set consisting of a Global Positioning System receiver and a three-axis magnetometer.

5. The method of claim 4, wherein the position/orientation update data is received from a relative position/orientation sensor selected from the set consisting of a three-axis accelerometer, a three-axis angular rate gyro, and a camera.

6. The method of claim 5, wherein the display, the absolute position/orientation sensor, and the relative position/orientation sensor are integrated into a head-mounted display.

7. The method of claim 5, wherein the display, the absolute position/orientation sensor, and the relative position/orientation sensor are integrated into a smartphone.

8. A system for augmented reality with enhanced pose estimation comprising:

a relative position/orientation sensor;

an absolute position/orientation sensor;

a display;

a geospatial data store;

a processor; and

one or more non-transitory computer-readable media storing computer-executable instructions which, when executed by the processor perform a method of estimating a pose of the display, the method comprising the steps of;

receiving, from the relative position/orientation sensor, position/orientation update data for a first time epoch;

storing the position/orientation update data for the first time epoch in a rewind buffer;

updating, by the processor and based on the position/orientation update data, a state-based estimation of a position and orientation of the display;

predicting, by the processor and based on the state-based estimation of the position and orientation of the display, a future position and a future orientation for the display for the first time epoch;

receiving, from the geospatial data store, geodetic coordinates of a point of interest for the first time epoch;

displaying, on the display, indicia of a location of the point of interest based on the geodetic coordinates and the future position and the future orientation for the display for the first time epoch;

receiving, from the absolute position/orientation sensor, absolute position/orientation data for a second time epoch, the second time epoch preceding the first time epoch;

retrieving, from the rewind buffer, the position/orientation update data for the first time epoch;

reupdating, by the processor and based on the absolute position/orientation data and the position/orientation update data, the state-based estimation of the position and orientation of the display; and

repredicting, by the processor and based on the absolute position/orientation data, the position/orientation update data and the state-based estimation of the position and orientation of the display, the future position and the future orientation for the display for the first time epoch.

9. The system of claim 8, further comprising a camera,

wherein the method further comprises the steps of:

receiving, from the camera video data; and

displaying, on the display, the video data; and

wherein the indicia of the location of the point of interest is overlaid on the point of interest in the displayed video data.

10. The system of claim 8, wherein the system is integrated into a head-mounted display.

11. The system of claim 10, wherein the indicia of the location of the point of interest is displayed as a geo-registered icon on a situational awareness ring in the view of a wearer of the head-mounted display.

12. The system of claim 8, wherein the system is integrated into a smartphone.

13. The system of claim 8, wherein the absolute position/orientation sensor is selected from the set consisting of a Global Positioning System receiver and a three-axis magnetometer.

14. The system of claim 8, wherein the relative position/orientation sensor is selected from the set consisting of a three-axis accelerometer, a three-axis angular rate gyro, and a camera.

15. One or more non-transitory computer-readable media storing computer-executable instructions which, when executed by a processor, perform a method of estimating the pose of a display, the method comprising the steps of:

receiving, from a relative position/orientation sensor, position/orientation update data for a first time epoch;

storing the position/orientation data for the first time epoch in a rewind buffer;

updating, by a processor and based on the position/orientation update data, a state-based estimation of a position and orientation of the display;

predicting, by the processor and based on the state-based estimation of the position and orientation of the display, a future position and a future orientation for the display for the first time epoch;

receiving geodetic coordinates of a point of interest for the first time epoch;

displaying, on the display, indicia of a location of the point of interest based on the geodetic coordinates and the future position and the future orientation for the display for the first time epoch;

after the expiration of the first time epoch, receiving, from an absolute position/orientation sensor, absolute position/orientation data for a second time epoch, the second time epoch preceding the first time epoch;

retrieving, from the rewind buffer, the position/orientation update data for the first time epoch;

reupdating, by the processor and based on the absolute position/orientation data and the position/orientation update data, the state-based estimation of the position and orientation of the display; and

repredicting, by the processor and based on the absolute position/orientation data, the position/orientation update data and the state-based estimation of the position and orientation of the display, the future position and the future orientation for the display for the first time epoch.

16. The media of claim 15, wherein the absolute position/orientation sensor is selected from the set consisting of a Global Positioning System receiver and a three-axis magnetometer.

17. The media of claim 16, wherein the relative position/orientation sensor is selected from the set consisting of a three-axis accelerometer, a three-axis angular rate gyro, and a camera.

18. The media of claim 17, wherein the display, the absolute position/orientation sensor, the relative position/orientation sensor, the processor, and the media are integrated into a head-mounted display.

19. The media of claim 15, wherein the state-based estimation of the position and the orientation is an Extended Kalman Filter-based estimation of the position and the orientation.

20. The media of claim 15, wherein the geodetic coordinates of the point of interest are received via a networked radio.