Systems and methods for assisting a user in capturing images for three-dimensional reconstruction

• US 9,648,297 B1
• Filed: 12/28/2012
• Issued: 05/09/2017
• Est. Priority Date: 12/28/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ettinger, Scott M., Ogale, Abhijit S.
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): Baptiste, Jerry Jean
• Application Number:
  US13/730,402
• Time in Patent Office: 1,593 Days
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597

Abstract

Systems and methods for assisting a user in capturing images for three-dimensional (3D) reconstruction are provided. A system includes an input module configured to receive current video image data from a device. The system also includes an analysis module configured to determine a 3D reconstruction indicator based on the current video image data and a previous image (or images) of a target captured by the device. The analysis module is also configured to determine whether the 3D reconstruction indicator is sufficient to allow a 3D reconstruction of the target based on the current video image data and the previous image. The system also includes a guidance module configured to assist the user in capturing a current image of the target from the current video image data based on the determination of whether the 3D reconstruction indicator is sufficient.

Claims

1. A system for assisting a user in capturing images for generation of a three-dimensional (3D) model of a target, the system comprising:

one or more processors; and

memory storing instructions executable by the processor;

wherein the instructions comprise;

receiving current video image data from a device, the current video image data being associated with a current pose of the device;

identifying 3D feature points of the current video image data and of at least one previous image, to generate at least a partial intermediate 3D motion model based on the 3D feature points, to identify two-dimensional (2D) feature points of the current video image data and of the at least one previous image, and to generate at least a partial 2D motion model based on the 2D feature points, the at least one previous image being associated with a previous pose of the device;

determining a 3D information ratio based on a number of the 3D feature points and a number of the 2D feature points;

assisting the user with capturing an additional image of the target based on whether the 3D information ratio is greater than or equal to a threshold; and

generating a 3D model of the target based on video image data from the device.

2. The system of claim 1, wherein the instructions further comprise receiving at least one of (i) a current position of the device from a global positioning sensor and (ii) a current orientation of the device from an inertial sensor.

3. The system of claim 2, wherein the instructions further comprise comparing the current position of the device to a previous position of the device, and comparing the current orientation of the device to a previous orientation of the device, wherein assisting the user with capturing an additional image of the target is further based on the comparisons.

4. The system of claim 1, wherein the instructions further comprise determining the amount of overlap between the current video image data and the at least one previous image based on at least one of (i) an amount of translation of the current video image data relative to the at least one previous image and (ii) an amount of rotation of the current video image data relative to the at least one previous image, and wherein assisting the user with capturing an additional image of the target is further based on the amount of overlap.

5. The system of claim 1, wherein the instructions further comprise determining whether an amount of motion of the device is greater than or equal to a minimum motion threshold, and wherein assisting the user with capturing an additional image of the target is further based on whether the amount of motion of the device is greater than or equal to the minimum motion threshold.

6. The system of claim 1, wherein the instructions further comprise determining an amount of overlap between the current video image data and the at least one previous image, comparing the amount of overlap to a maximum overlap threshold, and wherein assisting the user with capturing an additional image of the target is further based on determining whether the amount of overlap is less than or equal to the maximum overlap threshold.

7. The system of claim 1, wherein the instructions further comprise identifying one or more multiview tracks of the current video image data, each of the one or more multiview tracks comprising an image feature visible in the current video image data and at least two previous images of the target, and wherein assisting the user with capturing an additional image of the target is further based on the number of the one or more multiview tracks.

8. The system of claim 7, wherein the instructions further comprise comparing the number of the one or more multiview tracks to a multiview track threshold, and wherein assisting the user with capturing an additional image of the target is further based on comprises determining whether the number of the one or more multiview tracks is greater than or equal to the multiview track threshold.

9. The system of claim 1, wherein the instructions further comprise maintaining a graph of the at least one previous pose.

10. The system of claim 9, wherein two of the at least one previous poses are connected to one another in the graph if the previous image of one of the two previous poses overlaps with the previous image of the other of the two previous poses.

11. The system of claim 9, wherein the instructions further comprise adding the current pose to the graph, the current pose being connected to the at least one previous pose in the graph.

12. The system of claim 1, wherein assisting the user with capturing an additional image further comprises determining the pose of the device for capturing the additional image based on the at least one previous image and the at least one previous pose.

13. The system of claim 1, wherein assisting the user with capturing an additional image further comprises displaying to the user an indication of where to capture the current image, the indication of where to capture the current image comprising at least one of a map, a grid, a direction, a route, and an arrow.

14. A method for assisting a user in capturing images for generation of a three-dimensional (3D) model of a target, the method comprising:

receiving, with one or more processors, current video image data from a device, the current video image data being associated with a current pose of the device;

identifying, with one or more processors, 3D feature points of the current video image data and of at least one previous image, to generate at least a partial intermediate 3D motion model based on the 3D feature points, to identify two-dimensional (2D) feature points of the current video image data and of the at least one previous image, and to generate at least a partial 2D motion model based on the 2D feature points, the at least one previous image being associated with a previous pose of the device;

determining, with one or more processors, a 3D information ratio based on a number of the 3D feature points and a number of the 2D feature points;

assisting, with one or more processors, the user with capturing an additional image of the target based on whether the 3D information ratio is greater than or equal to a threshold; and

generating, with one or more processors, a 3D model of the target based on video image data from the device.

15. The method of claim 14, further comprising determining an amount of overlap between the current video image data and the at least one previous image of the target, and wherein assisting the user with capturing an additional image of the target is further based on a comparison of the amount of overlap with an overlap threshold.

16. A machine-readable non-transitory medium encoded with executable instructions for assisting a user in capturing images for generation of a three-dimensional (3D) model, the instructions comprising code for:

receiving current video image data from a device, the current video image data being associated with a current pose of the device;

identifying 3D feature points of the current video image data and of at least one previous image, to generate at least a partial intermediate 3D motion model based on the 3D feature points, to identify two-dimensional (2D) feature points of the current video image data and of the at least one previous image, and to generate at least a partial 2D motion model based on the 2D feature points, the at least one previous image being associated with a previous pose of the device;

determining a 3D information ratio based on a number of the 3D feature points and a number of the 2D feature points;

assisting the user with capturing an additional image of the target based on whether the 3D information ratio is greater than or equal to a threshold; and

generating a 3D model of the target based on video image data from the device.

17. The machine-readable medium of claim 16, wherein assisting the user comprises displaying to the user an indication of when and/or where to capture the current image.