Techniques for Accurate Pose Estimation

• US 20180018783A1
• Filed: 09/14/2017
• Published: 01/18/2018
• Est. Priority Date: 08/22/2014
• Status: Active Grant

• Current Assignee: Applied Research Associates Incorporated
• Original Assignee: Applied Research Associates Incorporated
• Inventors: Menozzi, Alberico
• Granted Patent: US 10,424,071 B2
• Time in Patent Office: Days
• CPC Class Codes: G01C 21/1654 with electromagnetic compass
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0181 Adaptation to the pilot/driver
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

The described technology regards an augmented reality system and method for estimating a position of a location of interest relative to the position and orientation of a display based upon a retroactive adjustment of a previously rendered position and orientation of the display, by means of an adjust-update-predict (AUP) cycle, and calculating the location of interest relative to the position and orientation of the display. Systems of the described technology include including a plurality of sensors, a processing module or other computation means, and a database. Methods of the described technology use data from the sensor package useful to accurately render graphical user interface information on a display.

Claims

1. A pose estimation system for use with a display, the system comprising:

a. a plurality of sensors,

b. a database comprising geodetic coordinates of a location of interest, and

c. a processing module communicatively coupled with the database and the sensors, the processing module receiving over time data from the sensors comprising delayed data measurements, and data representing the spatial position of the display relative to the sensors, and applies an Extended Kalman Filter (EKF) to the received data to estimate the position and orientation of the display, and based upon a retroactive adjustment of a previously rendered position and orientation of the display, estimates the position and orientation of the display using the delayed data measurements and predicts future positions of the display by means of an adjust-update-predict (AUP) cycle, and calculates the location of interest relative to the display, wherein a state estimate is reprocessed based on a delayed measurement within a regular processing interval and wherein past data is reprocessed within a single EKF epoch.

2. The pose estimation system of claim 1, wherein the plurality of sensors comprise a camera, a 3-axis accelerometer, a 3-axis angular rate gyro, a 3-axis magnetometer, a barometric pressure sensor, and a GPS receiver.

3. The pose estimation system of claim 1, wherein the database further comprises digital terrain and elevation data, and the processing module ingests said data to aid in the estimation of altitude of the display.

4. The pose estimation system of claim 1, further comprising the display, and wherein graphical user interface information comprising a graphical icon representing the location of interest, is rendered on the display at the position estimated by the processing module.

5. The pose estimation system of claim 4, further comprising a radio configured to send and receive geo-spatial data relating to objects in the environment, wherein the processing module receives, stores and uses the geo-spatial data to render graphical user interface information on the display comprising a graphical icon representing the objects in the environment.

6. The pose estimation system of claim 1, wherein the processing module implements a baseline GPS/INS to estimate the position of the location of interest, and adjusts the estimated values over time with absolute orientation information.

7. The pose estimation system of claim 1, wherein the position of the location of interest is the vector r_ds^d, calculated using the formula:

r_ds^d=(C_n^eC_bⁿC_d^b)^T[r_es^e−

(r_ep^e+C_n^eC_bⁿr_pd^b)]

wherein r_pd^b is the position of the origin of the display coordinate system d with respect to the point of origin of the body coordinate system p, expressed in body system coordinates b;

r_es^e is the position vector of the location of interest s in the environment with respect to the origin of the earth centered, earth fixed coordinate system e, expressed in e coordinates, derived from the geodetic coordinates of the location of interest in the database; and

r_ep^e is the position of location p with respect to the origin of e, expressed in e coordinates, wherein C_n^e is an orientation of a North-East-Down (n) coordinate system with respect to e, wherein C_bⁿ is an orientation of b with respect to the North-East-Down (n) coordinate system, and wherein C_d^b is the orientation of the display relative to the body system coordinates b.

8. The pose estimation system of claim 1, wherein the plurality of sensors comprises a 3-axis magnetometer, and wherein the processing module compares the norm of data from the 3-axis magnetometer to the Earth'"'"'s field strength to detect the presence or absence of magnetic disturbances and, upon detection of the presence of magnetic disturbances, rejects data from the magnetometer corresponding to the times of detected magnetic disturbances.

9. A method for providing an estimate of position for a location of interest relative to a display, the method comprising:

a. receiving sensor data from a plurality of sensors comprising delayed data measurements and data representing the spatial position and orientation of the display,

b. applying an Extended Kalman Filter (EKF) to the received data and geodetic coordinates of a location of interest, to estimate the position and orientation of the display, and calculating the location of interest relative to the display based upon a retroactive adjustment of a previously rendered position estimate of the display, using the delayed data measurements, a recalculation of the estimated position, and a prediction of future positions by means of an adjust-update-predict (AUP) cycle, wherein a state estimate is reprocessed based on a delayed measurement within a regular processing interval and wherein past data is reprocessed within a single EKF epoch.

10. The method of claim 9, wherein the plurality of sensors comprise a camera, a 3-axis accelerometer, a 3-axis angular rate gyro, a 3-axis magnetometer, a barometric pressure sensor, and a GPS receiver.

11. The method of claim 9, rendering a graphical icon on the display at a position representing the location of interest relative to the display.

12. The method of claim 9, further comprising sending and receiving geo-spatial data relating to objects in the environment, and calculating a position vector of an object in the environment using the estimated position and orientation of the display and the geo-spatial data relating to the object.

13. The method of claim 9, further comprising implementing a baseline GPS/INS to estimate the position of the location of interest, and adjusting the estimated values over time with absolute orientation information.

14. The method of claim 9, wherein the position of the location of interest is the vector r_ds^d, calculated using the formula:

r_ds^d=(C_n^eC_bⁿC_d^b)^T[r_es^e−

(r_ep^e+C_n^eC_bⁿr_pd^b)]

wherein r_pd^b is the position of the origin of the display coordinate system d with respect to the point of origin of the body coordinate system p, expressed in body system coordinates b;

r_es^e is the position vector of the location of interest s in the environment with respect to the origin of the earth centered, earth fixed coordinate system e, expressed in e coordinates, derived from the geodetic coordinates of the location of interest in a database; and

r_ep^e is the position of location p with respect to the origin of e, expressed in e coordinates, wherein C_n^e is an orientation of a North-East-Down (n) coordinate system with respect to e, wherein C_bⁿ is an orientation of b with respect to the North-East-Down (n) coordinate system, and wherein C_d^b is the orientation of the display relative to the body system coordinates b.

15. The method of claim 9, wherein the plurality of sensors comprises a 3-axis magnetometer, and wherein the processing module compares the norm of data from the 3-axis magnetometer to the Earth'"'"'s field strength to detect the presence or absence of magnetic disturbances and, upon detection of the presence of magnetic disturbances, rejects data from the magnetometer corresponding to the times of detected magnetic disturbances.

16. A pose estimation system for use with a display, the system comprising

a. means for receiving sensor data from a plurality of sensors comprising delayed data measurements and data representing the spatial position of the display,

b. means for applying an Extended Kalman Filter (EKF) to the received data and geodetic coordinates of a location of interest, to estimate the position and orientation of the display based upon a retroactive adjustment of a previously rendered position estimate using the delayed data measurements, and recalculating the estimated position and prediction of future positions, by means of an adjust-update-predict (AUP) cycle, and calculating the position of the location of interest relative to the display, and

c. means for providing a current state estimate through interpolation of state vectors in a forward buffer comprising a plurality of predicted state estimates that are a plurality of time steps ahead.

17. The pose estimation system of claim 16, wherein the plurality of sensors comprise a camera, a 3-axis accelerometer, a 3-axis angular rate gyro, a 3-axis magnetometer, a barometric pressure sensor, and a GPS receiver.

18. The pose estimation system of claim 16, further comprising a means for rendering a graphical icon on the display, at the position of the location of interest relative to the display.

19. The pose estimation system of claim 16, wherein the position of the location of interest is the vector r_ds^d, calculated using the formula:

r_ds^d=(C_n^eC_bⁿC_d^b)^T[r_es^e−

(r_ep^e+C_n^eC_bⁿr_pd^b)]

wherein r_pd^b is the position of the origin of the display coordinate system d with respect to the point of origin of the body coordinate system p, expressed in body system coordinates b;

r_es^e is the position vector of the location of interest s in the environment with respect to the origin of the earth centered, earth fixed coordinate system e, expressed in e coordinates, derived from the geodetic coordinates of the location of interest in a database; and

r_ep^e is the position of location p with respect to the origin of e, expressed in e coordinates, wherein C_n^e is an orientation of a North-East-Down (n) coordinate system with respect to e, wherein C_bⁿ is an orientation of b with respect to the North-East-Down (n) coordinate system, and wherein C_d^b is the orientation of the display relative to the body system coordinates b.

20. The pose estimation system of claim 16, wherein the plurality of sensors comprises a 3-axis magnetometer, and wherein the processing module compares the norm of data from the 3-axis magnetometer to the Earth'"'"'s field strength to detect the presence or absence of magnetic disturbances and, upon detection of the presence of magnetic disturbances, rejects data from the magnetometer corresponding to the times of detected magnetic disturbances.