Pose error estimation and localization using static features

• US 10,282,861 B2
• Filed: 12/12/2016
• Issued: 05/07/2019
• Est. Priority Date: 12/12/2016
• Status: Active Grant

• Current Assignee: HERE Global B.V. (HERE Holding Corporation)
• Original Assignee: HERE Global B.V. (HERE Holding Corporation)
• Inventors: Kwant, Richard, Mittal, Anish
• Primary Examiner(s): Gilliard, Delomia L
• Application Number:
  US15/375,513
• Publication Number:
  US 20180165831A1
• Time in Patent Office: 876 Days
• Field of Search: None
• CPC Class Codes: G01C 21/165 combined with non-inertial navigation instruments
• CPC Class Codes: G01C 21/28 with correlation of data from several navigational instruments
• CPC Class Codes: G01S 19/40 Correcting position, velocity or attitude
• CPC Class Codes: G01S 19/47 the supplementary measurement being an inertial measurement, e.g. tightly coupled inertial
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

The position and/or pose of a vehicle is determined in real time. An observed position and an observed pose of a vehicle are determined. A reference image is generated based on the observed position and the observed pose. The reference image comprises one or more reference static features. A captured image and the reference image are implicitly compared. Based on a result of the comparison, a correction to the observed position, the observed pose, or both is determined.

Claims

1. A method comprising:

determining an observed position and an observed pose of a vehicle;

generating a reference image based on the observed position and observed pose, wherein (a) the reference image comprises one or more reference static features and (b) the reference image is generated from the perspective of the observed position and the observed pose;

implicitly comparing the reference image to a captured image using a pose error network to determine a misalignment between at least one feature of the reference image and a corresponding feature of the captured image, wherein the pose error network comprises a trained neural network; and

based on the determined misalignment, determining, by the pose error network, a correction to the observed position, the observed pose, or both.

2. The method according to claim 1, wherein the reference image comprises a binary map that is a two dimensional projection of a three dimensional feature map at the observed position and observed pose.

3. The method according to claim 1, wherein the observed position and observed pose are determined based on a position and pose determined by a global positioning system (GPS), a position and pose determined by an inertial measurement unit (IMU), or a combination thereof.

4. The method according to claim 1, wherein the reference image is generated based on reference static feature information that comprises ground truth information for one or more reference static features.

5. The method according to claim 1, wherein generating the reference image comprises:

selecting one or more reference static features from a plurality of reference static features stored in association with a map tile of a digital map based on the observed position, observed pose, or both;

accessing reference static feature information corresponding to the selected reference static features from the map tile;

generating a three dimensional feature map based on the accessed reference static feature information; and

generating the reference image, wherein the reference image is a two dimensional projection of the three dimensional feature map based on the observed position, observed pose, or both.

6. The method according to claim 1, wherein the captured image comprises multi-channel color information.

7. The method according to claim 1, further comprising:

identifying one or more static features within the captured image;

identifying at least one of the one or more static features within at least one second captured image;

generating three dimensional information corresponding to the at least one static feature identified in both the captured image and the second captured image based at least in part on the captured image and the second captured image; and

generating a three dimensional image based at least in part on the three dimensional information, and

determining, by the pose error network, a correction to the observed position, the observed pose, or both based on implicitly comparing the generated three dimensional image and the reference image.

8. An apparatus comprising at least one processor, at least one memory storing computer program code, and a pose error network, the at least one memory and the computer program code configured to, with the processor, cause the apparatus to at least:

determine an observed position and an observed pose of a vehicle;

generate a reference image based on the observed position and observed pose, wherein (a) the reference image comprises one or more reference static features and (b) the reference image is generated from the perspective of the observed position and the observed pose;

implicitly compare the reference image to a captured image using the pose error network to determine a misalignment between at least one feature of the reference image and a corresponding feature of the captured image, wherein the pose error network comprises a trained neural network; and

based on the determined misalignment, determine, by the pose error network, a correction to the observed position, the observed pose, or both.

9. The apparatus according to claim 8, wherein the reference image comprises a binary map that is a two dimensional projection of a three dimensional feature map at the observed position and observed pose.

10. The apparatus according to claim 8, wherein the observed position and observed pose are determined based on a position and pose determined by a global positioning system (GPS), a position and pose determined by an inertial measurement unit (IMU), or a combination thereof.

11. The apparatus according to claim 8, wherein the reference image is generated based on reference static feature information that comprises ground truth information for one or more reference static features.

12. The apparatus according to claim 8, wherein to generate the reference image the at least one memory and the computer program code are configured to, with the processor, cause the apparatus to at least:

select one or more reference static features from a plurality of reference static features stored in association with a map tile of a digital map based on the observed position, observed pose, or both;

access reference static feature information corresponding to the selected reference static features from the map tile; and

generate a three dimensional feature map based on the accessed reference static feature information; and

generate the reference image, wherein the reference image is a two dimensional projection of the three dimensional feature map based on the observed position, observed pose, or both.

13. The apparatus according to claim 8, wherein the captured image comprises multi-channel color information.

14. The apparatus according to claim 9, wherein the at least one memory and the computer program code are further configured to, with the processor, cause the apparatus to at least:

identify one or more static features within the captured image;

identify at least one of the one or more static features within at least one second captured image;

generate three dimensional information corresponding to the at least one static feature identified in both the captured image and the second captured image based at least in part on the captured image and the second captured image; and

generate a three dimensional image based at least in part on the three dimensional information; and

determine, by the pose error network, a correction to the observed position, the observed pose, or both based on implicitly comparing the generated three dimensional image and the reference image.

15. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions configured to:

determine an observed position and an observed pose of a vehicle;

generate a reference image based on the observed position and observed pose, wherein (a) the reference image comprises one or more reference static features and (b) the reference image is generated from the perspective of the observed position and the observed pose;

implicitly compare the reference image to a captured image using a pose error network to determine a misalignment between at least one feature of the reference image and a corresponding feature of the captured image, wherein the pose error network comprises a trained neural network; and

based on the determined misalignment, determine, by the pose error network, a correction to the observed position, the observed pose, or both.

16. The computer program product according to claim 15, wherein the reference image comprises a binary map that is a two dimensional projection of a three dimensional feature map at the observed position and observed pose.

17. The computer program product according to claim 15, wherein the observed position and observed pose are determined based on a position and pose determined by a global positioning system (GPS), a position and pose determined by an inertial measurement unit (IMU), or a combination thereof.

18. The computer program product according to claim 15, wherein the reference image is generated based on reference static feature information that comprises ground truth information for one or more reference static features.

19. The computer program product according to claim 15, wherein the computer-executable program code instructions comprise program code instructions configured to generate the reference image by:

selecting one or more reference static features from a plurality of reference static features stored in association with a map tile of a digital map based on the observed position, observed pose, or both;

accessing reference static feature information corresponding to the selected reference static features from the map tile; and

generating a three dimensional feature map based on the accessed reference static feature information; and

generating the reference image, wherein the reference image is a two dimensional projection of the three dimensional feature map based on the observed position, observed pose, or both.

20. The computer program product according to claim 15, wherein the computer-executable program code instructions further comprise program code instructions configured to:

identify one or more static features within the captured image;

identify at least one of the one or more static features within at least one second captured image;

generate three dimensional information corresponding to the at least one static feature identified in both the captured image and the second captured image based at least in part on the captured image and the second captured image;

generate a three dimensional image based at least in part on the three dimensional information; and

determine, by the pose error network, a correction to the observed position, the observed pose, or both based on implicitly comparing the generated three dimensional image and the reference image.