Techniques for accurate pose estimation

• US 10,424,071 B2
• Filed: 09/14/2017
• Issued: 09/24/2019
• Est. Priority Date: 08/22/2014
• Status: Active Grant

• Current Assignee: Applied Research Associates Incorporated
• Original Assignee: Applied Research Associates Incorporated
• Inventors: Menozzi, Alberico
• Primary Examiner(s): Salvucci, Matthew
• Application Number:
  US15/704,986
• Publication Number:
  US 20180018783A1
• Time in Patent Office: 740 Days
• Field of Search: None
• CPC Class Codes: G01C 21/1654 with electromagnetic compass
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0181 Adaptation to the pilot/driver
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

The described technology regards an augmented reality system and method for estimating a position of a location of interest relative to the position and orientation of a display based upon a retroactive adjustment of a previously rendered position and orientation of the display, by means of an adjust-update-predict (AUP) cycle, and calculating the location of interest relative to the position and orientation of the display. Systems of the described technology include including a plurality of sensors, a processing module or other computation means, and a database. Methods of the described technology use data from the sensor package useful to accurately render graphical user interface information on a display.

Claims

1. A method of enhancing the accuracy of pose estimation of a display, comprising the steps of:

receiving, from a camera, video data;

receiving geodetic coordinates of a point of interest;

receiving, from a plurality of sensors, position/orientation data;

retroactively adjusting, by a processor and based on the position/orientation data received from the sensors, a historical position prediction and a historical orientation prediction in an Extended Kalman Filter-based estimation of a position and orientation of the display,

wherein the historical position prediction and the historical orientation prediction were previously used to overlay indicia of a historical location of the point of interest on historical video data on the display;

updating, by the processor and based on the position/orientation data received from the sensors and the adjusted historical position prediction and adjusted historical orientation prediction for the display, the Extended Kalman Filter-based estimation of the position and orientation of the display;

predicting, by the processor and based on the updated Extended Kalman Filter-based estimation of the position and orientation of the display, a future position and a future orientation for the display using the Extended Kalman Filter;

displaying, on the display, the video data;

overlaying, on the video data on the display, indicia of a location of the point of interest based on the geodetic coordinates and the predicted future position and the predicted future orientation for the display.

2. The method of claim 1, wherein the plurality of sensors includes a three-axis accelerometer and a three-axis gyroscope.

3. The method of claim 1, wherein the plurality of sensors includes the camera.

4. The method of claim 1, wherein the point of interest corresponds to a object in motion.

5. The method of claim 1, wherein the position/orientation data consists of orientation data.

6. The method of claim 1, wherein the display and the plurality of sensors are integrated into a head-mounted display.

7. The method of claim 1, wherein the display and the plurality of sensors are integrated into a smartphone.

8. A head-mounted display with enhanced pose estimation, comprising:

a plurality of sensors;

a see-through display;

a processor; and

one or more non-transitory computer-readable media storing computer-executable instructions which, when executed by the processor perform a method of estimating a pose of the display, the method comprising the steps of;

receiving geodetic coordinates of a point of interest

receiving, from the plurality of sensors, position/orientation data;

retroactively adjusting, based on the position/orientation data received from the sensors, a historical position prediction and a historical orientation prediction in an Extended Kalman Filter-based estimation of a position and orientation of the display,

wherein the historical position prediction and the historical orientation prediction were previously used to display indicia of a historical location of the point of interest on the display;

updating, based on the position/orientation data received from the sensors and the adjusted historical position prediction and historical orientation prediction for the display, the Extended Kalman Filter-based estimation of the position and orientation of the display;

predicting, based on the updated Extended Kalman Filter-based estimation of the position and orientation of the display, a future position and a future orientation for the display;

displaying, on the see-through display, indicia of a location of the point of interest based on the geodetic coordinates and the predicted future position and the predicted future orientation for the display.

9. The head-mounted display of claim 8, wherein the plurality of sensors includes a three-axis accelerometer and a three-axis gyroscope.

10. The head-mounted display of claim 9, further comprising a wireless receiver and wherein the location of the point of interest is received via the wireless receiver.

11. The head-mounted display of claim 9, wherein the location of the point of interest is received from a list of points of interest stored on the computer-readable medium.

12. The head-mounted display of claim 9, wherein the indicia of the location of the point of interest is overlaid on the point of interest in the view of a wearer of the head-mounted display.

13. The head-mounted display of claim 9, wherein the indicia of the location of the point of interest is displayed as a geo-registered icon on a situational awareness ring in the view of a wearer of the head-mounted display.

14. The head-mounted display of claim 8, wherein the plurality of sensors includes a camera.

15. One or more non-transitory computer-readable media storing computer-executable instructions which, when executed by a processor, perform a method of estimating the pose of a display, the method comprising the steps of:

receiving geodetic coordinates of a point of interest;

receiving, from a plurality of sensors, position/orientation data;

retroactively adjusting, by the processor and based on the position/orientation data received from the sensors, a historical position prediction and a historical orientation prediction for the display in a state-based estimation of a position and orientation of the display;

wherein the historical position prediction and the historical orientation prediction were previously used to display indicia of a historical location of the point of interest on the display;

updating, by the processor and based on the position/orientation data received from the sensors and the updated historical position prediction and historical orientation prediction of the display, the state-based estimation of the position and orientation of the display;

predicting, by the processor and based on the updated state-based estimation of the position and orientation of the display, a future position and a future orientation for the display using the state-based estimation of the position and orientation of the display;

displaying, on the display, indicia of a location of the point of interest based on the geodetic coordinates and the predicted future position and the predicted future orientation for the display.

16. The non-transitory media of claim 15, wherein the plurality of sensors includes a three-axis accelerometer and a three-axis rate gyroscope.

17. The non-transitory media of claim 16, wherein the state-based estimation of the position and orientation of the display is an Extended Kalman Filter-based estimation of the position and orientation of the display.

18. The non-transitory media of claim 15, wherein the display is a video display, wherein the plurality of sensors includes a camera, and wherein the method further comprises the steps of:

receiving, from the camera, video data; and

displaying, on the display, the video data.

19. The non-transitory media of claim 15, wherein the indicia of the location of the point of interest is overlaid as a geo-registered icon on the point of interest on the display.

20. The non-transitory media of claim 15, wherein the indicia of the location of the point of interest is displayed on a situational awareness ring on the display.