3D model updates using crowdsourced video

• US 9,224,206 B1
• Filed: 07/24/2014
• Issued: 12/29/2015
• Est. Priority Date: 10/16/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lookingbill, Andrew, Seitz, Steven Maxwell
• Primary Examiner(s): Le, Brian
• Application Number:
  US14/339,966
• Time in Patent Office: 523 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

An exemplary method includes prompting a user to capture video data at a location. The location is associated with navigation directions for the user. Information representing visual orientation and positioning information associated with the captured video data is received by one or more computing devices, and a stored data model representing a 3D geometry depicting objects associated with the location is accessed. Between corresponding images from the captured video data and projections of the 3D geometry, one or more candidate change regions are detected. Each candidate change region indicates an area of visual difference between the captured video data and projections. When it is detected that a count of the one or more candidate change regions is below a threshold, the stored model data is updated with at least part of the captured video data based on the visual orientation and positioning information associated with the captured video data.

Claims

1. A computer-implemented method, comprising:

obtaining, using one or more computing devices, video data comprised of a plurality of images depicting a location, the video data including positioning data representing a visual orientation and position of each image with respect to the location;

identifying, by the one or more computing devices, data representing a 3D geometry of objects associated with the location;

comparing, by the one or more computing devices, visual features of objects depicted in one or more corresponding areas between images from the video data and image projections of the 3D geometry; and

updating, by the one or more computing devices, the data representing the 3D geometry based at least in part on the comparison by integrating images from the video data into the 3D geometry.

2. The method of claim 1, wherein obtaining the video data further comprises receiving an indication that a user arrived at the location.

3. The method of claim 1, wherein identifying the data representing the 3D geometry further comprises determining whether positioning data associated with the video data overlaps with positioning data for the 3D geometry.

4. The method of claim 1, further comprising determining a visual correlation between one or more objects depicted in images from the video data and one or more objects depicted in the image projections.

5. The method of claim 1, wherein the comparing further comprises detecting visual differences between features of objects depicted in a given area of a particular image from the video data and features objects depicted in a corresponding area of the image projections.

6. The method of claim 1, further comprising determining an alignment of an integrated image from the video data based on a corresponding image projection from the 3D geometry.

7. The method of claim 6, further comprising adjusting the alignment of the integrated images in the 3D geometry using the visual orientation information associated with the video data.

8. A non-transitory computer readable medium, storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:

obtaining, using one or more computing devices, video data comprised of a plurality of images depicting a location, the video data including positioning data representing a visual orientation and position of each image with respect to the location;

identifying, by the one or more computing devices, data representing a 3D geometry of objects associated with the location;

comparing, by the one or more computing devices, visual features of objects depicted in one or more corresponding areas between images from the video data and image projections of the 3D geometry; and

updating, by the one or more computing devices, the data representing the 3D geometry based at least in part on the comparison by integrating images from the video data into the 3D geometry.

9. The computer readable medium of claim 8, wherein obtaining the video data further comprises receiving an indication that a user arrived at the location.

10. The computer readable medium 8, wherein identifying the data representing the 3D geometry further comprises determining whether positioning data associated with the video data overlaps with positioning data for the 3D geometry.

11. The computer readable medium 8, wherein the comparing further comprises detecting visual differences between features of objects depicted in a given area of a particular image from the video data and features objects depicted in a corresponding area of the image projections.

12. The computer readable medium 11, further comprising determining an alignment of an integrated image from the video data based on a corresponding image projection from the 3D geometry.

13. The computer readable medium 12, further comprising adjusting the alignment of the integrated images in the 3D geometry using the using the visual orientation information associated the corresponding image projection from the 3D geometry.

14. A system, comprising:

a memory; and

one or more processors coupled to the memory, the one or more processors being configured to;

obtain video data comprised of a plurality of images depicting a location, the video data including positioning data representing a visual orientation and position of each image with respect to the location;

identify data representing a 3D geometry of objects associated with the location;

compare visual features of objects depicted in one or more corresponding areas between images from the video data and image projections of the 3D geometry; and

update the data representing the 3D geometry based at least in part on the comparison by integrating images from the video data into the 3D geometry.

15. The system of claim 14, wherein to obtain the video data, the one or more processors are further configured to receive an indication that a user arrived at the location.

16. The system of claim 14, wherein to identify the data representing the 3D geometry, the one or more processors are further configured to determine whether positioning data associated with the video data overlaps with positioning data for the 3D geometry.

17. The system of claim 14, wherein the one or more processors are further configured to determine a visual correlation between one or more objects depicted in images from the video data and one or more objects depicted in the image projections.

18. The system of claim 14, wherein to compare visual features of objects, the one or more processors are further configured to detect visual differences between features of objects depicted in a given area of a particular image from the video data and features objects depicted in a corresponding area of the image projections.

19. The system of claim 14, wherein the one or more processors are further configured to determine an alignment of an integrated image from the video data based on a corresponding image projection from the 3D geometry.

20. The system of claim 19, wherein the one or more processors are further configured to adjust the alignment of the integrated images in the 3D geometry using the visual orientation information associated with the video data.