Method and apparatus for localization based on images and map data

• US 11,176,719 B2
• Filed: 04/01/2019
• Issued: 11/16/2021
• Est. Priority Date: 10/24/2018
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Chang, Hyun Sung, Son, Minjung, Sagong, Donghoon, Lee, Wonhee, Jung, Kyungboo
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Vu, Khoa
• Application Number:
  US16/371,305
• Publication Number:
  US 20200134896A1
• Time in Patent Office: 960 Days
• CPC Class Codes: G01C 21/3638 including 3D objects and buildings three dimensional [3D] modelling, e.g. data description of 3D objects G06T17/00; geographic models G06T17/05
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

Localization apparatuses and methods are disclosed where a localization apparatus extracts a feature of an object from an input image, generates an image in which the object is projected with respect to localization information of a device based on map data, and evaluates the localization information based on feature values corresponding to vertices included in a projection image.

Claims

1. A localization method, comprising:

generating a first image of an object from an input image;

generating a second image to project the object with respect to a plurality of candidate localization information of a device, based on map data comprising a position of the object;

pooling, from the first image, feature values corresponding to vertices in the second image;

determining scores of the plurality of candidate localization information based on the pooled feature values; and

determining a candidate localization information corresponding to a highest score to be localization information of the device

wherein the determining of the scores comprises;

determining rotational degree of freedom (DOF)s based on long-distance vertices affected below a first threshold by translational DOFs, from among vertices included in the second image;

after the determination of the rotational DOFs, determining a left and right translational DOF based on vanishing point-oriented short-distance vertices affected below a second threshold by a forward and backward translational DOF, from among short-distance vertices excluding the long-distance vertices from the second image; and

after the determination of the left and right translational DOF, determining the forward and backward translational DOF based on non-vanishing point-oriented short-distance vertices excluding the vanishing point-oriented short-distance vertices from the short-distance vertices.

2. The localization method of claim 1, wherein the generating of the first image comprises generating feature maps corresponding to a plurality of features.

3. The localization method of claim 1, wherein the generating of the second image comprises:

extracting a region corresponding to a field of view in the candidate localization information from the map data; and

projecting vertices included in the region into a projection point corresponding to the candidate localization information.

4. The localization method of claim 1, wherein the pooling comprises:

selecting pixels in the first image based on coordinates of the vertices; and

obtaining feature values of the selected pixels.

5. The localization method of claim 1, wherein the determining of the scores comprise determining a sum of the pooled feature values.

6. The localization method of claim 5, wherein the determining of the sum comprises, determining a weighted sum of the feature values based on weights determined for the features, in response to the first image comprising feature maps corresponding to features.

7. The localization method of claim 1, wherein the determining of the localization information of the device comprises:

segmenting the second image into regions; and

sequentially determining a plurality of degree of freedom (DOF) values included in the candidate localization information using scores calculated in the regions.

8. The localization method of claim 7, wherein the plurality of DOF values comprises:

three translational DOF values; and

three rotational DOF values.

9. The localization method of claim 7, wherein the segmenting comprises:

segmenting the second image into a long-distance region and a short-distance region based on a first criterion associated with a distance; and

segmenting the short-distance region into a vanishing point-oriented short-distance region and a non-vanishing point-oriented short-distance region based on a second criterion associated with a vanishing point.

10. The localization method of claim 9, wherein the sequentially determining comprises:

determining rotational DOFs based on the long-distance region;

determining a left and right translational DOF based on the vanishing point-oriented short-distance region; and

determining a forward and backward translational DOF based on the non-vanishing point-oriented short-distance region.

11. The localization method of claim 1, wherein the determining of the localization information of the device comprises:

determining a direction to improve the score based on a distribution of the pooled feature values; and

correcting the candidate localization information based on the direction.

12. The localization method of claim 11, wherein the first image comprises a probability distribution indicating a degree of closeness to the object,

wherein the determining of the direction comprises determining the direction based on the probability distribution.

13. The localization method of claim 11, wherein the determining of the localization information of the device comprises:

generating a corrected second image in which the object is projected with respect to the corrected candidate localization information; and

determining a corrected score of the corrected candidate localization information by pooling, from the first image, feature values corresponding to vertices in the corrected second image,

wherein the determining of the direction, the correcting of the candidate localization information, the generating of the corrected second image, and the calculating of the corrected score are iteratively performed until the corrected score satisfies a condition.

14. The localization method of claim 1, further comprising:

determining a virtual object on the map data to provide an augmented reality (AR) service; and

displaying the virtual object based on the determined localization information.

15. The localization method of claim 14, wherein the input image comprises a driving image of a vehicle, and

the virtual object indicates driving route information.

16. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the localization method of claim 1.

17. The localization method of claim 1, wherein the first image comprises a probability distribution indicating a degree of closeness to the object;

and

the determining of the localization information comprises determining a direction to improve the score based on a distribution of the pooled feature values, the information related to a direction towards the object being included in the first image, and correcting the candidate localization information based on the direction.

18. A localization method, comprising:

generating a first image of an object from an input image;

generating a second image to project the object with respect to candidate localization information of a device, based on map data comprising a position of the object;

segmenting the second image into a long-distance region and a short-distance region based on a first threshold associated with a distance;

segmenting the short-distance region into a vanishing point-oriented short-distance region and a non-vanishing point-oriented short-distance region based on a second criterion associated with a vanishing point; and

determining degree of freedom (DOF) values included in the candidate localization information through matching between the first image and the regions,

wherein the determining of the DOF comprises;

determining rotational DOFs based on long-distance vertices affected below a first threshold by translational DOFs, from among vertices included in the second image,

after the determination of the rotational DOFs, determining a left and right translational DOF based on vanishing point-oriented short-distance vertices affected below a second threshold by a forward and backward translational DOF, from among short-distance vertices excluding the long-distance vertices from the second image, and

after the determination of the left and right translational DOF, determining the forward and backward translational DOF based on non-vanishing point-oriented short-distance vertices excluding the vanishing point-oriented short-distance vertices from the short-distance vertices.

19. The localization method of claim 18, wherein the determining comprises determining the DOF values included in the candidate localization information by sequentially using scores calculated through the matching in the regions.

20. The localization method of claim 18, wherein the determining comprises:

calculating, while changing DOF values determined for the regions, scores corresponding to the changed DOF values by pooling, from the first image, feature values corresponding to vertices in the regions; and

selecting a DOF value corresponding to a highest score.

21. The localization method of claim 18, wherein the plurality of DOF values comprises:

three translational DOF values; and

three rotational DOF values.

22. The localization method of claim 18, further comprising:

determining a virtual object on the map data to provide an augmented reality (AR) service; and

displaying the virtual object based on the determined DOF values.

23. The localization method of claim 22, wherein the input image comprises a driving image of a vehicle, and

the virtual object indicates driving route information.

24. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the localization method of claim 18.

25. A localization apparatus, comprising:

a processor configured to;

generate a first image of an object from an input image,

generate a second image to project the object with respect to a plurality of candidate localization information of a device, based on map data comprising a position of the object,

pool, from the first image, feature values corresponding to vertices in the second image,

determine scores of the plurality of candidate localization information based on the pooled feature values; and

determine a candidate localization information corresponding to a highest score to be localization information of the device

wherein the determining of the scores comprises;

determining rotational degree of freedom (DOF)s based on long-distance vertices affected below a first threshold by translational DOFs, from among vertices included in the second image;

after the determination of the rotational DOFs, determining a left and right translational DOF based on vanishing point-oriented short-distance vertices affected below a second threshold by a forward and backward translational DOF, from among short-distance vertices excluding the long-distance vertices from the second image; and

after the determination of the left and right translational DOF, determining the forward and backward translational DOF based on non-vanishing point-oriented short-distance vertices excluding the vanishing point-oriented short-distance vertices from the short-distance vertices.

26. A localization apparatus, comprising:

a processor configured to;

generate a first image of an object from an input image,

generate a second image to project the object with respect to candidate localization information of a device, based on map data comprising a position of the object,

segment the second image into a long-distance region and a short-distance region based on a first threshold associated with a distance,

segment the short-distance region into a vanishing point-oriented short-distance region and a non-vanishing point-oriented short-distance region based on a second criterion associated with a vanishing point; and

determine degree of freedom (DOF) values included in the candidate localization information through matching between the first image and the regions,

wherein the processor is further configured to;

determine rotational degree of freedom (DOF) values based on the long-distance region,

after the determination of the rotational DOF, determine a left and right translational DOF based on the vanishing point-oriented short-distance region, and

after the determination of the left and right translational DOF, determine a forward and backward translational DOF based on the non-vanishing point-oriented short-distance region.

27. A localization apparatus comprising:

a sensor disposed on a device, and being configured to sense one or more of an image and candidate localization information of the device;

a processor configured to

generate a first image of an object from the image,

generate a second image to project the object with respect to the candidate localization information, based on map data comprising a position of the object,

segment the second image into a long-distance region and a short-distance region based on a distance,

segment the short-distance region into a vanishing point-oriented short-distance region and a non-vanishing point-oriented short-distance region based on a vanishing point,

determine a score of the candidate localization information based on pooling, from the first image, feature values corresponding to vertices in the second image,

determine localization information of the device based on the score, and

a head-up display (HUD) configured to visualize a virtual object on the map data based on the determined localization information,

wherein the processor to determine is configured to;

determine rotational degree of freedom (DOF) values based on the long-distance region;

after the determination of the rotational DOF, determine a left and right translational DOF based on the vanishing point-oriented short-distance region; and

after the determination of the left and right translational DOF, determine a forward and backward translational DOF based on the non-vanishing point-oriented short-distance region.

28. The localization apparatus of claim 27, wherein the processor is further configured to generate, using a neural network, the first image comprising feature maps corresponding to a plurality of features.

29. The localization apparatus of claim 27, wherein the second image comprise a projection of two-dimensional (2D) vertices corresponding to the object.

30. The localization apparatus of claim 27, further comprising a memory configured to store the map data, the image, the first image, the second image, the score, and instructions that, when executed, configures the processor to determine any one or any combination of the determined localization information and the virtual object.