Techniques for accurate pose estimation

• US 10,304,200 B2
• Filed: 09/14/2017
• Issued: 05/28/2019
• Est. Priority Date: 08/22/2014
• Status: Active Grant

• Current Assignee: Applied Research Associates Incorporated
• Original Assignee: Applied Research Associates Incorporated
• Inventors: Menozzi, Alberico
• Primary Examiner(s): Salvucci, Matthew
• Application Number:
  US15/704,954
• Publication Number:
  US 20180018782A1
• Time in Patent Office: 621 Days
• Field of Search: None
• CPC Class Codes: G01C 21/1654 with electromagnetic compass
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0181 Adaptation to the pilot/driver
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

The described technology regards an augmented reality system and method for estimating a position of a location of interest relative to the position and orientation of a display, using a forward buffer to store current and predicted position estimates calculated by the methods of the present invention. Systems of the described technology include including a plurality of sensors, a processing module or other computation means, and a database. Methods of the described technology use data from the sensor package useful to accurately render graphical user interface information on a display.

Claims

1. A method of enhancing the accuracy of pose estimation of a display, comprising the steps of:

receiving, from a plurality of sensors, position/orientation data for a given time epoch;

updating, by a processor and based on the position/orientation data received from the sensors, an Extended Kalman Filter-based estimation of a position and orientation of the display;

predicting, by the processor and based on the Extended Kalman Filter-based estimation of the position and orientation of the display, a plurality of future positions and future orientations for the display for a corresponding plurality of time epochs, each time epoch of the plurality of time epochs being after the given time epoch;

storing the plurality of future positions and future orientations for the display into a forward buffer for estimating position and orientation of the display at future times;

determining an update time for the display based at least in part on the current time and a previously determined update latency for the display;

interpolating an update position and an update orientation for the display at the update time for the display based at least in part on the forward buffer;

receiving geodetic coordinates of a point of interest at the update time for the display;

displaying, on the display, indicia of a location of the point of interest based on the geodetic coordinates and the interpolated update position and update orientation for the display at the update time for the display.

2. The method of claim 1, wherein the plurality of sensors include at least one absolute sensor and at least one relative sensor.

3. The method of claim 2, wherein the absolute sensor is selected from the set consisting of a Global Positioning System receiver and a three-axis magnetometer.

4. The method of claim 2, wherein the relative sensor is selected from the set consisting of a three-axis accelerometer, a three-axis angular rate gyro, and a camera.

5. The method of claim 1, further comprising the steps of:

receiving, from a camera, video data for the given time epoch; and

displaying, on the display and at the update time, the video data for the given time epoch,

wherein the indicia of a location of the point of interest is overlain on the point of interest on the video data.

6. The method of claim 1, wherein the point of interest corresponds to a object in motion.

7. The method of claim 1, wherein the display and sensors are integrated into a smartphone.

8. A system for augmented reality with enhanced pose estimation comprising:

a plurality of position/orientation sensors;

a display;

a geospatial data store;

a processor; and

one or more non-transitory computer-readable media storing computer-executable instructions which, when executed by the processor perform a method of estimating a pose of the display, the method comprising the steps of;

receiving, from the plurality of position/orientation sensors, position/orientation data for a given time epoch;

updating, by the processor and based on the position/orientation data received from the position/orientation sensors, an state-based estimation of a position and orientation of the display;

predicting, by the processor and based on the state-based estimation of the position and orientation of the display, a plurality of estimated future positions and estimated future orientations for the display for a corresponding plurality of time epochs, each time epoch of the plurality of time epochs being after the given time epoch;

storing the plurality of estimated future positions and estimated future orientations for the display into a forward buffer for estimating position and orientation of the display at future times;

determining an update time for the display based at least in part on the current time and a previously determined update latency for the display;

interpolating an update position and an update orientation for the display at the update time for the display based at least in part on the forward buffer;

receiving, from the geospatial data store, geodetic coordinates of a point of interest;

displaying, on the display, indicia of a location of the point of interest based on the geodetic coordinates and the interpolated update position and update orientation for the display at the update time for the display.

9. The system of claim 8, further comprising a camera, and wherein the method further comprises the steps of:

receiving, from the camera, video data for the given time epoch; and

displaying, on the display and at the update time, the video data for the given time epoch,

wherein the indicia of a location of the point of interest is overlain on the point of interest on the video data.

10. The system of claim 8, wherein the indicia of the location of the point of interest is displayed as a geo-registered icon on a situational awareness ring in the display.

11. The system of claim 8, wherein the plurality of position/orientation sensors, the display, the processor, and the media are incorporated into a head-mounted display.

12. The system of claim 8, wherein the plurality of position/orientation sensors includes at least one absolute position/orientation sensor and at least one relative position/orientation sensor.

13. The system of claim 8, wherein the absolute position/orientation sensor is selected from the set consisting of a Global Positioning System receiver and a three-axis magnetometer.

14. The system of claim 8, wherein relative position/orientation sensor is selected from the set consisting of a three-axis accelerometer and a three-axis angular rate gyro.

15. One or more non-transitory computer-readable media storing computer-executable instructions which, when executed by a processor, perform a method of estimating the pose of a display, the method comprising the steps of:

receiving, from a plurality of sensors, position/orientation data for a given time epoch;

updating, by a processor and based on the position/orientation data received from the sensors, a state-based estimation of a position and orientation of the display;

predicting, by the processor and based on the state-based estimation of the position and orientation of the display, a plurality of future positions and future orientations for the display for a corresponding plurality of time epochs, each time epoch of the plurality of time epochs being after the given time epoch;

storing the plurality of future positions and future orientations for the display into a forward buffer for estimating position and orientation of the display at future times;

determining an update time for the display based at least in part on the current time and a previously determined update latency for the display;

interpolating an update position and an update orientation for the display at the update time for the display based at least in part on the forward buffer;

receiving geodetic coordinates of a point of interest at the update time for the display;

displaying, on the display, indicia of a location of the point of interest based on the geodetic coordinates and the interpolated update position and update orientation for the display at the update time for the display.

16. The media of claim 15, wherein the method further comprises the steps of:

receiving, from a camera, video data for the given time epoch; and

displaying, on the display and at the update time, the video data for the given time epoch,

wherein the indicia of a location of the point of interest is overlain on the point of interest on the video data.

17. The media of claim 15, wherein the state-based estimation of the position and orientation of the display is an Extended Kalman Filter-based estimation of the position and orientation of the display.

18. The media of claim 15, wherein the plurality of sensors include at least one absolute sensor and at least one relative sensor.

19. The media of claim 18, wherein the absolute sensor is selected from the set consisting of a Global Positioning System receiver and a three-axis magnetometer.

20. The media of claim 18, wherein the relative sensor is selected from the set consisting of a three-axis accelerometer, a three-axis angular rate gyro, and a camera.