Crowd-sourced creation and updating of area description file for mobile device localization

• US 9,811,734 B2
• Filed: 05/11/2015
• Issued: 11/07/2017
• Est. Priority Date: 05/11/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Williams, Brian Patrick, Hickman, Ryan Michael, Tu, Laurent, Nerurkar, Esha, Lynen, Simon
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US14/708,877
• Publication Number:
  US 20160335497A1
• Time in Patent Office: 911 Days
• Field of Search: None
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A computing system includes a network interface, a first datastore, a second datastore, and a merge module. The merge module is to receive a set of one or more area description files from a set of one or more first mobile devices. Each area description file represents a point cloud of spatial features detected by a corresponding first mobile device at an area. The computing system further includes a localization module and a query module. The localization generation module is to generate a localization area description file for the area from the set of one or more area description files and to store the localization area description file in the second datastore. The localization area description file represents a point cloud of spatial features for the area. The query module is to provide the localization area description file to a second mobile device via the network interface.

Claims

1. A method comprising:

receiving, at a computing system, a set of one or more area description files from a set of one or more first mobile devices, each area description file representing a point cloud of spatial features detected by a corresponding first mobile device at an area;

generating, at the computing system, a localization area description file for the area from the set of one or more area description files, the localization area description file representing a point cloud of spatial features for the area; and

providing the localization area description file from the computing system to a second mobile device, wherein;

the set of one or more area description files comprises a plurality of area description files; and

generating the localization area description file comprises;

identifying the plurality of area description files as being associated with the area;

deduplicating the point clouds represented by the plurality of area description files;

determining a relative alignment of the point clouds represented by the plurality of area description files; and

incorporating the point clouds in the localization area description file.

2. The method of claim 1, further comprising:

performing a localization process at the second mobile device using the localization area description file.

5. The method of claim 1, wherein generating the localization area description file further comprises:

eliminating spatial features from the point clouds based on at least one stability criterion.

6. The method of claim 1, wherein generating the localization area description file further comprises:

identifying a plurality of sub-areas based on the point clouds; and

generating a local area description file for each sub-area from the spatial features of the point clouds that are located in the sub-area.

7. The method of claim 1, further comprising:

receiving, at the computing system, feedback data from the second mobile device, the feedback data identifying whether one or more of the spatial features represented by the localization area description file was detected by the second mobile device while in the area;

updating, at the computing system, the localization area description file based on the feedback data to generate an updated localization area description file; and

providing the updated localization area description file from the computing system to a third mobile device.

3. A method comprising:

receiving, at a computing system, a set of one or more area description files from a set of one or more first mobile devices, each area description file representing a point cloud of spatial features detected by a corresponding first mobile device at an area;

generating, at the computing system, a localization area description file for the area from the set of one or more area description files, the localization area description file representing a point cloud of spatial feature for the area; and

providing the localization area description file from the computing system to a second mobile device,

wherein each area description file of the set of one or more area description files comprises;

spatial feature data specifying, for each spatial feature of the point cloud represented by the area description file, a multidimensional coordinate of the spatial feature based on a corresponding coordinate frame;

statistical data representing, for each spatial feature, a visual appearance of the spatial feature independent of angle of view or lighting conditions; and

sensor data captured from one or more non-image sensors concurrent with detection of the spatial features of the point cloud.

4. The method of claim 3, wherein generating the localization area description file comprises:

incorporating at least a subset of the spatial feature data and the statistical data of each area description file of the set in the localization area description file; and

excluding the sensor data of each area description file in the set from the localization area description file.

8. A method comprising;

receiving, at a computing system, a set of one or more area description files from a set of one or more first mobile devices, each area description file representing a point cloud of spatial feature detected by a corresponding first mobile device at an area;

generating, at the computing system, a localization area description file for the area from the set of one or more description files, the localization area description file representing a point cloud of spatial feature for the area;

providing the localization area description file from the computing system to a second mobile device;

receiving, at the computing system, feedback data from the second mobile device, the feedback data identifying whether one or more of the spatial feature represented by the localization area description file was detected by the second mobile device while in the area;

updating, at the computing system, the localization area description file based on the feedback data to generate an update localization area description file; and

providing the update localization area description file from the computing system to a third mobile device,

wherein updating the localization area description file comprises;

maintaining a score for each spatial feature represented by the local area description file; and

for a spatial feature represented by the localization area description file;

increasing a score of the spatial feature in response to the feedback data indicating that the spatial feature was detected by the second mobile device;

decreasing the score of the spatial feature in response to the feedback data indicating that the spatial feature was not detected by the second mobile device; and

selectively removing the spatial feature from the localization area description file based on a comparison of the score to at least one threshold.

9. A method comprising;

receiving, at a computing system, a set of one or more area description files from a set of one or more first mobile devices, each area description file representing a point cloud of spatial feature detected by a corresponding first mobile device at an area;

generating, at the computing system, a localization area description file for the area from the set of one or more area description files, the localization area description file representing a point cloud of spatial feature for the area;

providing the localization area description file from the computing system to a second mobile device;

receiving, at the computing system, feedback data from the second mobile device, the feedback data identifying whether one or more of the spatial feature represented by the localization area description file was detected by the second mobile device while in the area;

updating, at the computing system, the localization area description file based on the feedback data to generate an update localization area description file; and

providing the update localization area description file from the computing system to a third mobile device, wherein;

the feedback data identifies a spatial feature detected by the second mobile device and which was not represented by the localization area description file; and

wherein updating the localization area description file comprises;

increasing a score associated with the spatial feature detected by the second mobile device; and

selectively modifying the localization area description file to include the spatial feature detected by the second mobile device responsive to a comparison of the score and at least one threshold.

10. A computing system comprising;

a network interface to couple to a plurality of mobile devices;

a first datastore;

a second datastore;

a merge module couple to the network interface and the datastore, the merge module to receive a set of one or more area description files from a set of one or more first mobile devices, each area description file representing a point cloud of spatial feature detected by a corresponding first mobile device at an area;

a localization generation module coupled to the first datastore and the second datastore, the localization generation module to generate a localization area description file for the area from the set of one or more area description files and to store the localization area description file in the second datastore, the localization area description file representing a point cloud of spatial features for the area; and

a query module coupled to the second datastore and the network interface, the query module to provide the localization area description file to at least one second mobile device via the network interface, wherein;

each area description file of the set comprises;

spatial feature data specifying, for each spatial feature of the point cloud represented by the area description file, a multidimensional coordinate of the spatial feature based on a corresponding coordinate frame; and

statistical data representing, for each spatial feature, a visual appearance of the spatial feature independent of angle of view or lighting conditions; and

sensor data captured from one or more non-image sensors concurrent with detection of the spatial features of the point cloud; and

the localization area description file comprises;

at least a subset of the spatial feature data of each area description file of the set; and

at least a subset of the statistical data of each area description file of the set.

11. The computing system of claim 10, wherein:

the set of one or more area description files comprises a plurality of area description files;

the query module is to identify the plurality of area description files as being associated with the area; and

the merge module is to generate the localization area description file by;

deduplicating the point clouds represented by the plurality of area description files;

determining a relative alignment of the point clouds; and

incorporating the point clouds in the localization area description file.

12. The computing system of claim 11, wherein:

the merge module further is to generate the localization area description file by eliminating spatial features from the point clouds based on at least one stability criterion.

13. The computing system of claim 11, wherein:

the merge module further is to generate the localization area description file further by;

identifying a plurality of sub-areas based on the point clouds; and

generating a local area description file for each sub-area from the spatial features of the point clouds that are located in the sub-area.

14. The computing system of claim 10, further comprising:

a spatial feature filter module coupled to the first datastore and the network interface, the spatial feature filter module to receive feedback data from the second mobile device, the feedback data identifying whether one or more of the spatial features represented by the localization area description file was detected by the second mobile device using the localization area description file, and to update the localization area description file based on the feedback data to generate an updated localization area description file; and

wherein the query module is to provide the updated localization area description file to at least one third mobile device via the network interface.

15. The computing system of claim 14, wherein:

the spatial feature filter module is to update the localization area description file by;

maintaining a score for each spatial feature represented by the local area description file; and

for a spatial feature represented by the localization area description file;

increasing a score of the spatial feature in response to the feedback data indicating that the spatial feature was detected by the second mobile device;

decreasing the score of the spatial feature in response to the feedback data indicating that the spatial feature was not detected by the second mobile device; and

selectively removing the spatial feature from the localization area description file based on a comparison of the score to at least one threshold.

16. The computing system of claim 15, wherein:

the feedback data identifies a spatial feature detected by the second mobile device and which was not represented by the localization area description file; and

the spatial feature filter module is to update the localization area description file by;

increasing a score associated with the spatial feature detected by the second mobile device; and

selectively modifying the localization area description file to include the spatial feature detected by the second mobile device responsive to a comparison of the score and at least one threshold.

17. A mobile device comprising:

a network interface to couple to a remote computing system;

a plurality of imaging sensors to capture imagery at an area in which the mobile device is located;

a spatial feature detection module to detect a first set of spatial features in the captured imagery; and

a localization module to obtain a localization area description file for the area from the remote computing system via the network interface, the localization area description file representing a second set of spatial features for the area, and to localize the mobile device to a coordinate frame associated with the area based on a comparison of the first set of spatial features to the second set of spatial features, wherein the localization area description file comprises statistical data representing, for each of the second set of spatial feature, a visual appearance of the spatial feature independent of angle of view or lighting conditions.

18. The mobile device of claim 17, wherein:

the localization module further is to provide feedback data to the remote computing system via the network interface, the feedback data identifying at least one of;

whether a spatial feature of the second set was detected in the first set; and

whether a spatial feature in the first set was not detected in the second set.

19. A mobile device comprising:

a network interface to couple to a remote computing system;

a plurality of imaging sensors to capture imagery at an area at which the mobile device is located;

a spatial feature detection module coupled to the plurality of imaging sensors, the spatial feature detection module to detect a set of spatial features in the captured imagery;

a set of one or more one non-image sensors to capture sensor data concurrent with the capture of the imagery; and

an assembly module coupled to the network interface, the spatial feature detection module, and the set of one or more non-image sensors, the assembly module to provide an area description file to the remote computing system via the network interface, the area description file representing a point cloud of a selected subset of the set of spatial features and including the sensor data, wherein the area description file comprises;

spatial feature data specifying, for each spatial feature of the point cloud represented by the area description file, a multidimensional coordinate of the spatial feature based on a corresponding coordinate frame;

statistical data representing, for each spatial feature, a visual appearance of the spatial feature independent of angle of view or lighting conditions; and

sensor data captured from one or more non-image sensors.

20. The mobile device of claim 19, wherein:

the assembly module further is to identify an area at which the mobile device is located and to request a localization area description file for the identified area from the remote computing system; and

the assembly module is to provide the area description file responsive to an indication from the remote computing system that the requested localization area description file is unavailable for the identified area.