Managing feature data for environment mapping on an electronic device

• US 9,940,542 B2
• Filed: 08/11/2015
• Issued: 04/10/2018
• Est. Priority Date: 08/11/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Nerurkar, Esha, Hesch, Joel, Lynen, Simon
• Primary Examiner(s): WERNER, BRIAN P
• Application Number:
  US14/823,283
• Publication Number:
  US 20170046594A1
• Time in Patent Office: 973 Days
• Field of Search: None
• CPC Class Codes: G05D 1/0268 using internal positioning means
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/40 Extraction of image or video features

Abstract

An electronic device reduces localization data based on feature characteristics identified from the data. Based on the feature characteristics, a quality value can be assigned to each identified feature, indicating the likelihood that the data associated with the feature will be useful in mapping a local environment of the electronic device. The localization data is reduced by removing data associated with features have a low quality value, and the reduced localization data is used to map the local environment of the device by locating features identified from the reduced localization data in a frame of reference for the electronic device.

Claims

1. A method comprising:

generating, at an electronic device, localization data for the electronic device based on imagery captured at the electronic device and based on received non-visual pose information obtained from non-visual sensors;

identifying a plurality of features of an environment of the electronic device based on the captured imagery, at least one image of the captured imagery comprising multiple features of the plurality of features, each feature of the plurality of features having a quality value corresponding to the feature, and wherein the multiple features of the at least one image have different quality values;

identifying a quality value threshold (QVT) for the localization data, wherein the QVT is a value derived from one of an average of quality values, a median of quality values, and a value based on a number of quality values that exceed the QVT;

reducing the localization data based on characteristics of the plurality of features and the corresponding quality value by removing data associated with features in the plurality of features having the quality value below the QVT; and

mapping the environment of the electronic device based on the reduced localization data.

2. The method of claim 1, wherein the characteristics of the plurality of features comprise at least one of a number of observations of each of the plurality of features in different images of the captured imagery and a consistency of position of each of the plurality of features in different images of the captured imagery.

3. The method of claim 1, wherein the quality value comprises a likelihood that the localization data associated with the feature will be useful in the mapping.

4. The method of claim 2, wherein the characteristics of the plurality of features further comprise one of a distance from the electronic device of each of the plurality of features in different images of the captured imagery and a variability of the plurality of features in different images of the captured imagery.

5. The method of claim 3, wherein the quality value is generated based on one or more of a number of observations of a feature across images, a consistency of the feature position in different images, and a distance of the feature from the electronic device when one or more of the associated images is captured.

6. The method of claim 1, wherein reducing the localization data further comprises removing the non-visual pose information from the data.

7. The method of claim 1, wherein reducing the localization data further comprises geometrically compressing the localization data by replacing a plurality of feature descriptors representing geometric points with a feature descriptor representing a geometric shape corresponding to the geometric points.

8. The method of claim 1, wherein reducing the localization data further comprises:

identifying keyframes of the localization data; and

removing data that does not correspond to a keyframe of the localization data.

9. A method, comprising,

generating quality values for a plurality of features identified from imagery, at least one image of the imagery comprising multiple features of the plurality of features, and wherein the multiple features of the at least one image have different quality values;

reducing, at an electronic device, localization data of the electronic device based on the generated quality values, wherein the quality values are based on one or more of a consistency of position of a feature across images, a number of observations of a feature across images, and a distance of a feature from the electronic device upon capture of associated images; and

mapping a local environment of the electronic device based on the reduced localization data.

10. The method of claim 9, wherein reducing the localization data further comprises:

removing non-visual sensor data from the localization data; and

performing geometric compression by replacing a plurality of feature descriptors representing geometric points with a feature descriptor representing a geometric shape corresponding to the geometric points.

11. The method of claim 9, wherein reducing the localization data further comprises:

identifying a quality value for each feature based on characteristics of features in the localization data, the characteristics comprising one or more of a dynamic score for a feature, a variability of location of a feature in different images, and a number of observations of a feature in different images.

12. The method of claim 9, wherein the mapping comprises:

matching a feature from the reduced localization data to a feature descriptor of a database of feature descriptors; and

mapping the local environment based on matching the feature.

13. An electronic device comprising:

an imaging sensor to capture imagery;

a plurality of non-visual sensors;

a processor comprising;

a feature identification module configured to;

generate, at an electronic device, localization data for the electronic device based on based on imagery captured at the electronic device and based on received non-visual pose information obtained from non-visual sensors; and

identify characteristics of a plurality of features of an environment of the electronic device based on the captured imagery, at least one image of the captured imagery comprising multiple features of the plurality of features, and wherein the multiple features of the at least one image have different quality values;

a data reducer configured to identify a quality value threshold (QVT) and reduce the localization data based on removing features of the plurality of features with quality values lower that the QVT, wherein the QVT is a value derived from one of an average of the quality values, a median of the quality values, and a value based on a number of quality values that exceed the QVT; and

an environment mapper configured to generate a map of the environment of the electronic device based on the reduced localization data.

14. The electronic device of claim 13, wherein the characteristics of the plurality of features comprise a number of observations of each of the plurality of features in different images of the captured imagery.

15. The electronic device of claim 13, wherein the characteristics of the plurality of features comprise a consistency of position of each of the plurality of features in different images of the captured imagery.

16. The electronic device of claim 13, wherein the characteristics of the plurality of features comprise a distance from the electronic device of each of the plurality of features in different images of the captured imagery.

17. The electronic device of claim 13, wherein the characteristics of the plurality of features comprise a variability of the plurality of features in different images of the captured imagery.

18. The electronic device of claim 13, wherein the data reducer is further configured to reduce the localization data by removing the non-visual pose information from the localization data and performing a geometric compression by replacing a plurality of feature descriptors representing geometric points with a feature descriptor representing a geometric shape corresponding to the geometric points.

19. The electronic device of claim 13, wherein the processor reduces the localization data further by geometrically compressing the localization data.

20. The electronic device of claim 13, wherein the processor reduces the localization data by:

identifying keyframes of the localization data; and

removing data that does not correspond to a keyframe of the localization data.