Map summarization and localization

• US 10,339,708 B2
• Filed: 07/25/2017
• Issued: 07/02/2019
• Est. Priority Date: 11/01/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lynen, Simon, Zeisl, Bernhard
• Primary Examiner(s): Mushambo, Martin
• Application Number:
  US15/659,358
• Publication Number:
  US 20180122136A1
• Time in Patent Office: 707 Days
• Field of Search: None
• CPC Class Codes: G01C 17/28 Electromagnetic compasses with north seeking magnetic elements and having electric pick-offs G01C17/26
• CPC Class Codes: G01C 17/38 Testing, calibrating, or compensating of compasses
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 1/325 Power saving in peripheral device
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 40/164 : using holistic features

Abstract

An electronic device generates a summary map of a scene based on data representative of objects having a high utility for identifying the scene when estimating a current pose of the electronic device and localizes the estimated current pose with respect to the summary map. The electronic device identifies scenes based on groups of objects appearing together in consistent configurations over time, and identifies utility weights for objects appearing in scenes, wherein the utility weights indicate a predicted likelihood that the corresponding object will be persistently identifiable by the electronic device in the environment over time and are based at least in part on verification by one or more mobile devices. The electronic device generates a summary map of each scene based on data representative of objects having utility weights above a threshold.

Claims

1. A method, comprising:

generating, at an electronic device, a first set of data representative of one or more objects in an environment of the electronic device, wherein the first set of data is based on images captured from one or more visual sensors and non-visual data from one or more non-visual sensors;

identifying, based on the first set of data, a first set of one or more groups of objects, wherein each group comprises objects appearing together in a configuration in the environment

identifying a first scene comprising the first set of one or more groups of objects based on a consistency with which the configuration appears over time or over a plurality of motion tracking sessions being above a first threshold;

identifying a utility weight of each of the one or more objects for identifying the first scene, the utility weight indicating a predicted likelihood that the corresponding object will be persistently identifiable by the electronic device in the first scene over time and based at least in part on verification by one or more mobile devices;

generating a three-dimensional representation of the first scene based a subset of the first set of data representative of a subset of the one or more objects in the first scene, wherein the subset of the one or more objects comprises objects having utility weights above a threshold, wherein the subset of the first set of data is limited to a threshold amount of data; and

localizing, at the electronic device, an estimated current pose of the electronic device based on the three-dimensional representation of the first scene.

2. The method of claim 1, further comprising updating the utility weight of a first object included in the subset based on generating, at the electronic device, additional data representative of the first object.

3. The method of claim 1, further comprising:

receiving a second set of data representative of one or more objects in the first scene from a set of one or more mobile devices;

identifying a utility weight for identifying the first scene of each of the one or more objects represented by the second set of data; and

updating the three-dimensional representation of the first scene, wherein updating comprises adding to the three-dimensional representation data representative of a subset of the one or more objects represented by the second set of data, wherein the subset comprises objects having utility weights above the threshold.

4. The method of claim 3, further comprising updating the utility weight of a first object included in the subset based on receiving, at the electronic device, additional data representative of the first object.

5. The method of claim 1, wherein the utility weight is based on a persistence of the object over a number of visits to the first scene.

6. The method of claim 1, wherein the utility weight is based on how recently data representative of the object was generated.

7. The method of claim 1, wherein the utility weight is based on a consistency of the appearance of the object over time.

8. A method, comprising:

generating, at an electronic device, first data representative of one or more objects in an environment of the electronic device, wherein the first data is based on images captured from one or more visual sensors and non-visual data from one or more non-visual sensors;

identifying a first set of one or more groups of objects wherein each group comprises objects appearing together in a configuration in the environment;

identifying a first scene comprising the first set of one or more groups of objects based on a consistency with which the configuration appears over time or over a plurality of motion tracking sessions being above a first threshold, the consistency based at least in part on verification by one or more mobile devices; and

generating a reference map of the first scene based on a subset of the first data, the subset representative of the first set of one or more groups of objects, wherein the subset of the first data is limited to a threshold amount of data.

9. The method of claim 8, further comprising:

localizing, at the electronic device, an estimated current pose of the electronic device based on the reference map of the first scene.

10. The method of claim 8, further comprising:

generating, at an electronic device, second data representative of one or more objects in an environment of the electronic device, wherein the second data is based on images captured from one or more visual sensors and non-visual data from one or more non-visual sensors;

comparing the one or more objects represented by the second data to the objects comprising the one or more groups of objects comprising the first scene; and

in response to identifying that objects represented by the second data generated at the electronic device have a correlation to the objects comprising the one or more groups of objects in the first scene that is below a second threshold, identifying a second set of one or more groups of objects represented by the second data wherein each group comprises objects appearing in a configuration;

identifying a second scene comprising the second set of one or more groups of objects based on the consistency with which the configuration appears being above the first threshold;

generating a reference map of the second scene based on the second data; and

localizing, at the electronic device, an estimated current pose of the electronic device based on the reference map of the second scene.

11. The method of claim 8, further comprising:

identifying, at the electronic device, a utility weight of each of the one or more objects for identifying a scene, the utility weight indicating a predicted likelihood that the corresponding object will be persistently identifiable by the electronic device in the environment over time and based at least in part on verification by one or more mobile devices; and

wherein generating the reference map comprises basing the reference map on a subset of the first data representative a subset of the one or more objects in the environment, wherein the subset of the one or more objects comprises objects having utility weights above a third threshold.

12. The method of claim 11, further comprising:

receiving third data representative of one or more objects in the environment of the electronic device from a set of one or more mobile devices;

comparing the one or more objects represented by the third data to the objects comprising the one or more groups of objects comprising the first scene;

identifying a utility weight of each of the one or more objects represented by the third data, the utility weight indicating a predicted likelihood that the corresponding object will be persistently identifiable by the electronic device in the environment over time and based at least in part on verification by one or more mobile devices; and

updating the reference map of the first scene, wherein updating comprises adding to the reference map data representative of a subset of the one or more objects represented by the third data, wherein the subset comprises objects having utility weights above the threshold.

13. The method of claim 12, wherein the utility weight is based on a persistence of the object over a number of visits to the environment.

14. The method of claim 12, wherein the utility weight is based on how recently data representative of the object was generated.

15. The method of claim 12, wherein the utility weight is based on a consistency of the appearance of the object over time.

16. A non-transitory computer-readable storage medium embodying a set of executable instructions, the set of executable instructions to manipulate at least one processor to:

generate a first set of data representative of one or more objects in an environment of an electronic device, wherein the first set of data is based on images captured from one or more visual sensors and non-visual data from one or more non-visual sensors;

identify a first set of one or more groups of objects wherein each group comprises objects appearing together in a configuration in the environment;

identify a first scene comprising the first set of one or more groups of objects based on the consistency with which the configuration appears over time or over a plurality of motion tracking sessions being above a first threshold;

identify a utility weight for each of the one or more objects comprising the first scene, wherein the utility weight indicates a predicted likelihood that the corresponding object will be persistently identifiable by the electronic device in the environment over time and is based at least in part on verification by one or more mobile devices;

generate a reference map of the first scene based on a subset of the first set of data representative of a subset of the one or more objects comprising the first scene, wherein the subset of the one or more objects comprises objects having utility weights above a threshold, and wherein the subset of the first set of data is limited to a threshold amount of data; and

localize an estimated current pose of the electronic device based on the reference map of the first scene.

17. The non-transitory computer-readable storage medium of claim 16, wherein the utility weight for each of the one or more objects is based on a comparison of data representative of one or more objects in the environment of the electronic device received from a set of one or more mobile devices to the subset of the first set of data upon which the reference map is based.