Image registration with device data

• US 10,282,856 B2
• Filed: 09/25/2017
• Issued: 05/07/2019
• Est. Priority Date: 01/03/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Klingner, Bryan, Kohler, Damon
• Primary Examiner(s): Chang, Jon
• Application Number:
  US15/714,353
• Publication Number:
  US 20180012371A1
• Time in Patent Office: 589 Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

Systems and methods for image registration using data collected by an electronic device, such as a mobile device, capable of simultaneous localization and mapping are provided. An electronic device, such as a mobile device, can be can be configured to collect data using a variety of sensors as the device is carried or transported through a space. The collected data can be processed and analyzed to generate a three-dimensional representation of the space and objects in the space in near real time as the device is carried through the space. The data can be used for a variety of purposes, including registering imagery for localization and image processing.

Claims

1. A computer-implemented method, comprising:

obtaining, by a mobile device, location data indicative of a location of the mobile device;

obtaining, by the mobile device, depth data acquired by the mobile device, the depth data indicative of the location of one or more surfaces proximate the mobile device;

generating, by the mobile device in near real time while obtaining the location data and the depth data, a three-dimensional map of a space through which the mobile device is moving, the three-dimensional map of the space being defined relative to a first coordinate system associated with the mobile device;

obtaining, by the mobile device, an image from an image database, the image being defined relative to a second coordinate system that differs from the first coordinate system; and

registering, by the mobile device, the image obtained from the image database with one or more images captured by the mobile device, wherein registering the image obtained from the image database with the one or more images captured by the mobile device comprises aligning the first coordinate system associated with the mobile device and the second coordinate system associated with the image obtained from the image database.

2. The method of claim 1, wherein the method comprises identifying, by the mobile device, one or more candidate images from the image database based on the location data, the image that is registered with the one or more images captured by the mobile device being one of the plurality of candidate images.

3. The method of claim 1, wherein the location data comprises a sparse point cloud and the depth data is generated based at least in part on the sparse point cloud.

4. The method of claim 1, wherein the depth data is acquired by the mobile device using one or more depth sensors included in the mobile device.

5. The method of claim 1, wherein the image obtained from the image database is registered with the one or more images captured by the mobile device based at least in part by matching the depth data acquired by the mobile device with depth data associated with the image obtained from the image database.

6. The method of claim 1, further comprising obtaining, by the mobile device, localization information for the mobile device based at least in part on location information associated with the image obtained from the image database.

7. The method of claim 6, wherein obtaining, by the mobile device, localization information for the mobile device comprises refining, by the mobile device, the location data indicative of the location of the mobile device based at least in part on the location information associated with the image obtained from the image database.

8. The method of claim 6, wherein obtaining, by the mobile device, localization information for the mobile device comprises associating, by the mobile device, the location data indicative of the location of the mobile device with the second coordinate system corresponding to the image obtained from the image database.

9. The method of claim 1, wherein the method comprises generating, by the mobile device, an image composition based at least in part on the image obtained from the image database and the one or more images captured by the mobile device.

10. The method of claim 9, wherein generating, by the mobile device, the image composition comprises removing, by the mobile device, at least one object depicted in the image obtained from the image database or the one or more images captured by the mobile device.

11. The method of claim 9, wherein generating, by the mobile device, the image composition comprises inserting, by the mobile device, a portion of the image obtained from the image database into at least one of the one or more images captured by the mobile device.

12. The method of claim 11, wherein the portion of the image obtained from the image database comprises a three-dimensional model.

13. A mobile device, comprising:

one or more processors; and

one or more memory devices, the one or more memory devices storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising;

obtaining location data indicative of a location of the mobile device, wherein the location data comprises a sparse point cloud of data points acquired by the mobile device, wherein the sparse point cloud of data points comprises a plurality of data points representative of points on surfaces proximate to the mobile device;

generating, in near real time while obtaining the location data, a three-dimensional map of a space through which the mobile device is moving, the three-dimensional map of the space being defined relative to a first coordinate system associated with the mobile device;

obtaining a first image captured by the mobile device the first image defined relative to the first coordinate system; and

obtaining a second image from an image database, the second image being defined relative to a second coordinate system that differs from the first coordinate system; and

registering the first image captured by the mobile device with the second image based at least in part the sparse point cloud of data points acquired by the mobile device, wherein registering the first image with the second image comprises aligning the first coordinate system associated with the mobile device and the second coordinate system associated with the second image obtained from the image database.

14. The computing system of claim 13, wherein the sparse point cloud of data points is generated via structure from motion techniques applied to multiple images captured by the mobile device.

15. The computing system of claim 13, wherein the operations further comprise obtaining localization information for the mobile device based at least in part on data associated with the second image.

16. The computing system of claim 13, wherein the operations further comprise inserting a portion of the second image into the first image.

17. The computing system of claim 16, wherein the portion of the second image comprises a three-dimensional model.

18. One or more tangible, non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors of a mobile device cause the one or more processors of the mobile device to perform operations, the operations comprising:

obtaining data acquired by the mobile device, the data comprising location data indicative of a location of the mobile device, the data further comprising depth data acquired by the mobile device, the depth data indicative of the location of one or more surfaces proximate the mobile device;

generating, in near real time while obtaining the location data and the depth data, a three-dimensional map of a space through which the mobile device is moving, the three-dimensional map of the space being defined relative to a first coordinate system associated with the mobile device;

accessing data associated with a geographic information system, the data associated with a geographic information system associated with a second coordinate system that is different than the first coordinate system associated with the mobile device;

registering the data acquired by the mobile device with the data associated with the geographic information system based at least in part on matching images or depth data captured by the mobile device with one or more images or depth data associated with the geographic information system, wherein registering the data acquired by the mobile device with the data associated with the geographic information system comprises aligning the first coordinate system and the second coordinate system.