3D model updates using crowdsourced video

• US 9,721,347 B2
• Filed: 10/27/2015
• Issued: 08/01/2017
• Est. Priority Date: 10/16/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lookingbill, Andrew, Seitz, Steven Maxwell
• Primary Examiner(s): AHMED, SAMIR ANWAR
• Application Number:
  US14/924,041
• Publication Number:
  US 20160180540A1
• Time in Patent Office: 644 Days
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

An exemplary method includes prompting a user to capture video data at a location. The location is associated with navigation directions for the user. Information representing visual orientation and positioning information associated with the captured video data is received by one or more computing devices, and a stored data model representing a 3D geometry depicting objects associated with the location is accessed. Between corresponding images from the captured video data and projections of the 3D geometry, one or more candidate change regions are detected. Each candidate change region indicates an area of visual difference between the captured video data and projections. When it is detected that a count of the one or more candidate change regions is below a threshold, the stored model data is updated with at least part of the captured video data based on the visual orientation and positioning information associated with the captured video data.

Claims

1. A computer-implemented method, comprising:

obtaining, using one or more computing devices, a plurality of images depicting a location, the plurality of images associated with positioning data representing a visual orientation and position of each image with respect to the location;

identifying, by the one or more computing devices, data representing a 3D geometry of objects associated with the location;

comparing, by the one or more computing devices, visual features of objects depicted in one or more corresponding areas between images from the plurality of images and image projections of the 3D geometry; and

updating, by the one or more computing devices, the data representing the 3D geometry based at least in part on a result of the comparison between the visual features of the objects depicted in the one or more corresponding areas in the images from the plurality of images when the result indicates a change.

2. The method of claim 1, wherein further comprising receiving an indication that a user arrived at the location.

3. The method of claim 1, wherein identifying the data representing the 3D geometry further comprises determining whether positioning data associated with the plurality of images overlaps with positioning data for the 3D geometry.

4. The method of claim 1, further comprising determining a visual correlation between one or more objects depicted in the images from the plurality of images and one or more objects depicted in the image projections.

5. The method of claim 1, wherein the comparing further comprises detecting visual differences between features of objects depicted in a given area of a particular image and features of objects depicted in a corresponding area of the image projections.

6. The method of claim 1, further comprising determining an alignment of an integrated image from the plurality of images based on a corresponding image projection from the 3D geometry.

7. The method of claim 6, further comprising adjusting the alignment of the integrated images in the 3D geometry using the positioning data representing the visual orientation information.

8. A non-transitory computer readable medium, storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:

obtaining, using one or more computing devices, a plurality of images depicting a location, the plurality of images associated with positioning data representing a visual orientation and position of each image with respect to the location;

identifying, by the one or more computing devices, data representing a 3D geometry of objects associated with the location;

comparing, by the one or more computing devices, visual features of objects depicted in one or more corresponding areas between images from the plurality of images and image projections of the 3D geometry; and

updating, by the one or more computing devices, the data representing the 3D geometry based at least in part on a result of the comparison between the visual features of the objects depicted in the one or more corresponding areas in the images from the plurality of images when the result indicates a change.

9. The computer readable medium of claim 8, wherein the method further comprises receiving an indication that a user arrived at the location.

10. The computer readable medium of claim 8, wherein identifying the data representing the 3D geometry further comprises determining whether positioning data associated with the plurality of images overlaps with positioning data for the 3D geometry.

11. The computer readable medium of claim 8, wherein the comparing further comprises detecting visual differences between features of objects depicted in a given area of a particular image from the plurality of images and features objects depicted in a corresponding area of the image projections.

12. The computer readable medium of claim 11, further comprising determining an alignment of an integrated image from the plurality of images based on a corresponding image projection from the 3D geometry.

13. The computer readable medium of claim 12, further comprising adjusting the alignment of the integrated image in the 3D geometry using the positioning data representing the visual orientation associated the corresponding image projection from the 3D geometry.

14. A system, comprising:

a memory; and

one or more processors coupled to the memory, the one or more processors being configured to;

obtain a plurality of images depicting a location, the plurality of images being associated with positioning data representing a visual orientation and position of each image with respect to the location;

identify data representing a 3D geometry of objects associated with the location;

compare visual features of objects depicted in one or more corresponding areas between images from the plurality of images and image projections of the 3D geometry; and

update the data representing the 3D geometry based at least in part on a result of the comparison between the visual features of the objects depicted in the one or more corresponding areas in the images from the plurality of images when the result indicates a change.

15. The system of claim 14, wherein to obtain the plurality of images, the one or more processors are further configured to receive an indication that a user arrived at the location.

16. The system of claim 14, wherein to identify the data representing the 3D geometry, the one or more processors are further configured to determine whether positioning data associated with the plurality of images overlaps with positioning data for the 3D geometry.

17. The system of claim 14, wherein the one or more processors are further configured to determine a visual correlation between one or more objects depicted in the images from the plurality of images and one or more objects depicted in the image projections.

18. The system of claim 14, wherein to compare visual features of objects, the one or more processors are further configured to detect visual differences between features of objects depicted in a given area of a particular image from the plurality of images and features of objects depicted in a corresponding area of the image projections.

19. The system of claim 14, wherein the one or more processors are further configured to determine an alignment of an integrated image from the plurality of images based on a corresponding image projection from the 3D geometry.

20. The system of claim 19, wherein the one or more processors are further configured to adjust the alignment of the integrated images in the 3D geometry using the positioning data representing the visual orientation associated with the plurality of images.