Detection of invariant features for localization

• US 9,940,729 B1
• Filed: 11/18/2016
• Issued: 04/10/2018
• Est. Priority Date: 11/18/2016
• Status: Active Grant

• Current Assignee: HERE Global B.V. (Nokia Corporation)
• Original Assignee: HERE Global B.V. (Nokia Corporation)
• Inventors: Kwant, Richard, Mittal, Anish, Viswanathan, Anirudh
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US15/355,727
• Time in Patent Office: 508 Days
• CPC Class Codes: G01C 21/28 with correlation of data from several navigational instruments
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

A first image and a second image are provided to a trained neural network. The first image comprises one or more static features and the second image comprises at least one of the one or more static features. A static feature is identified in both the first and second images by a branch of the trained neural network. A three dimensional image comprising the identified static feature is generated and three dimensional geometric information/data related to the static feature is extracted and stored in association with a tile of a digital map. A set of training images may be used to train the trained neural network comprises training image subsets comprising two or more images that substantially overlap that were (a) captured at different times; (b) captured under different (i) weather conditions, (ii) lighting conditions, or (iii) weather and lighting conditions; or both a and b.

Claims

1. A method comprising:

providing to a trained neural network a first image, wherein the first image (a) is captured by an image capture device at a first location and at a first pose, (b) is associated with geophysical coordinates corresponding to the first location, and (c) comprises one or more static features;

providing to the trained neural network a second image, wherein the second image (a) is captured by an image capture device at a second location and at a second pose, (b) is associated with geophysical coordinates corresponding to the second location, and (c) comprises at least one of the one or more static features;

identifying the at least one of the one or more static features in both the first and second images with a branch of the trained neural network;

generating a three dimensional image comprising the at least one identified static feature and based on the first and second images;

extracting a three dimensional geometry of the at least one identified static feature;

encoding the extracted three dimensional geometry as an array;

based on the first location and second location, determining a static feature location for the at least one identified static feature; and

storing the array in association with a map tile, wherein the map tile is selected based on the static feature location.

2. A method according to claim 1, wherein a set of training images used to train the trained neural network comprises training image subsets comprising two or more images that substantially overlap that were captured at different times.

3. A method according to claim 1, wherein a training set of images used to train the trained neural network comprises training image subsets comprising two or more images that substantially overlap that were captured under different (a) weather conditions, (b) lighting conditions, or (c) weather and lighting conditions.

4. A method according to claim 1, wherein the three dimensional image is a binary feature map.

5. A method according to claim 1, wherein the array comprises shape information for the corresponding static feature.

6. A method according to claim 1, wherein at least one training image of a set of training images used to train the trained network comprises at least one pixel that is masked, the at least one masked pixel corresponding to a non-static feature in the training image.

7. A method according to claim 1, wherein (a) the first location differs from the second location, (b) the first pose differs from the second pose, or (c) both the first location differs from the second location and the first pose differs from the second pose.

8. A method according to claim 1, wherein the static feature of the one or more static features is a generic static feature.

9. A method according to claim 1, wherein static feature information is stored in association with the map tile, the static feature information comprising the array and the static feature location.

10. A method according to claim 9, wherein static feature information is stored in association with a road segment of the map tile.

11. A method according to claim 9, wherein (a) a static feature orientation is determined from at least one of the first location, first pose, second location, or second pose and (b) the static feature information further comprises the static feature orientation.

12. An apparatus comprising at least one processor and at least one memory storing computer program code, the at least one memory and the computer program code configured to, with the processor, cause the apparatus to at least:

provide to a trained neural network a first image, wherein the first image (a) is captured by an image capture device at a first location and at a first pose, (b) is associated with geophysical coordinates corresponding to the first location, and (c) comprises one or more static features;

provide to the trained neural network a second image, wherein the second image (a) is captured by an image capture device at a second location and at a second pose, (b) is associated with geophysical coordinates corresponding to the second location, and (c) comprises at least one of the one or more static features;

identify the at least one of the one or more static features in both the first and second images with a branch of the trained neural network;

generate a three dimensional image comprising the at least one identified static feature and based on the first and second images;

extract a three dimensional geometry of the at least one identified static feature;

encode the extracted three dimensional geometry as an array;

based on the first location and second location, determine a static feature location for the at least one identified static feature; and

store the array in association with a map tile, wherein the map tile is selected based on the static feature location.

13. An apparatus according to claim 12, wherein a set of training images used to train the trained neural network comprises training image subsets comprising two or more images that substantially overlap that were captured at different times.

14. An apparatus according to claim 12, wherein a training set of images used to train the trained neural network comprises training image subsets comprising two or more images that substantially overlap that were captured under different (a) weather conditions, (b) lighting conditions, or (c) weather and lighting conditions.

15. An apparatus according to claim 12, wherein at least one training image of a set of training images used to train the trained network comprises at least one pixel that is masked, the at least one masked pixel corresponding to a non-static feature in the training image.

16. An apparatus according to claim 12, wherein (a) the first location differs from the second location, (b) the first pose differs from the second pose, or (c) both the first location differs from the second location and the first pose differs from the second pose.

17. An apparatus according to claim 12, wherein static feature information is stored in association with the map tile, the static feature information comprising the array and the static feature location.

18. An apparatus according to claim 17, wherein static feature information is stored in association with a road segment of the map tile.

19. An apparatus according to claim 17, wherein (a) a static feature orientation is determined from at least one of the first location, first pose, second location, or second pose and (b) the static feature information further comprises the static feature orientation.

20. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions configured to:

provide to a trained neural network a first image, wherein the first image (a) is captured by an image capture device at a first location and at a first pose, (b) is associated with geophysical coordinates corresponding to the first location, and (c) comprises one or more static features;

provide to the trained neural network a second image, wherein the second image (a) is captured by an image capture device at a second location and at a second pose, (b) is associated with geophysical coordinates corresponding to the second location, and (c) comprises at least one of the one or more static features;

identify the at least one of the one or more static features in both the first and second images with a branch of the trained neural network;

generate a three dimensional image comprising the at least one identified static feature and based on the first and second images;

extract a three dimensional geometry of the at least one identified static feature;

encode the extracted three dimensional geometry as an array;

based on the first location and second location, determine a static feature location for the at least one identified static feature; and

store the array in association with a map tile, wherein the map tile is selected based on the static feature location.