OBJECT RECONSTRUCTION FROM DENSE LIGHT FIELDS VIA DEPTH FROM GRADIENTS

• US 20180139436A1
• Filed: 10/31/2017
• Published: 05/17/2018
• Est. Priority Date: 11/11/2016
• Status: Active Grant

• Current Assignee: ETH Zurich, Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: ETH Zurich, Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Ycer, Kaan, Kim, Changil, Sorkine-Hornung, Alexander, Sorkine-Hornung, Olga
• Granted Patent: US 10,887,581 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/564 from contours
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

The present disclosure relates to techniques for reconstructing an object in three dimensions that is captured in a set of two-dimensional images. The object is reconstructed in three dimensions by computing depth values for edges of the object in the set of two-dimensional images. The set of two-dimensional images may be samples of a light field surrounding the object. The depth values may be computed by exploiting local gradient information in the set of two-dimensional images. After computing the depth values for the edges, depth values between the edges may be determined by identifying types of the edges (e.g., a texture edge, a silhouette edge, or other type of edge). Then, the depth values from the set of two-dimensional images may be aggregated in a three-dimensional space using a voting scheme, allowing the reconstruction of the object in three dimensions.

Claims

1. A method for object reconstruction, the method comprising:

receiving a light field represented by a plurality of images of an object, wherein the plurality of images include a first image and a second image, and wherein the second image is at a viewpoint different than the first image;

identifying a first point of the object in the first image;

identifying a second line in the second image based on the first point in the first image;

identifying one or more second points along the second line;

identifying one or more first points along a first line intersecting the first point, wherein each point of the one or more first points is projected from the one or more second points;

determining that a second point of the one or more second points corresponds to the first point based on a color of each of the one or more first points and a color of each of the one or more second points;

computing a depth for the first point based on the first point and the second point, wherein the depth is computed in response to determining that the second point corresponds to the first point;

generating a depth map for the first image using the depth of the first point; and

generating a mesh for the object based on the depth map.

2. The method of claim 1, wherein the second line has a direction of an epipolar line for the first image.

3. The method of claim 1, wherein the second point is an estimate of where the first point is in the second image.

4. The method of claim 1, wherein a color gradient is computed using colors of the one or more first points and the one or more second points, and wherein the color gradient is used to compute the depth.

5. The method of claim 1, wherein the first point is a color edge of the object, and wherein the second point corresponds to the first point such that the second point is the color edge of the object.

6. The method of claim 1, wherein the light field is unstructured.

7. The method of claim 1, wherein a viewpoint of the first image is similar to a viewpoint of the second image.

8. A non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors, the plurality of instructions when executed by the one or more processors cause the one or more processors to:

receive a light field represented by a plurality of images of an object, wherein the plurality of images include a first image and a second image, and wherein the second image is at a viewpoint different than the first image;

identify a first point of the object in the first image;

identify a second line in the second image based on the first point in the first image;

identify one or more second points along the second line;

identify one or more first points along a first line intersecting the first point, wherein each point of the one or more first points is projected from the one or more second points;

determine that a second point of the one or more second points corresponds to the first point based on a color of each of the one or more first points and a color of each of the one or more second points;

compute a depth for the first point based on the first point and the second point, wherein the depth is computed in response to determining that the second point corresponds to the first point;

generate a depth map for the first image using the depth of the first point; and

generate a mesh for the object based on the depth map.

9. The non-transitory computer-readable storage medium of claim 8, wherein the second line has a direction of an epipolar line for the first image.

10. The non-transitory computer-readable storage medium of claim 8, wherein the second point is an estimate of where the first point is in the second image.

11. The non-transitory computer-readable storage medium of claim 8, wherein a color gradient is computed using colors of the one or more first points and the one or more second points, and wherein the color gradient is used to compute the depth.

12. The non-transitory computer-readable storage medium of claim 8, wherein the first point is a color edge of the object, and wherein the second point corresponds to the first point such that the second point is the color edge of the object.

13. The non-transitory computer-readable storage medium of claim 8, wherein the light field is unstructured.

14. The non-transitory computer-readable storage medium of claim 8, wherein a viewpoint of the first image is similar to a viewpoint of the second image.

15. A system comprising:

one or more processors; and

a non-transitory computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to;

receive a light field represented by a plurality of images of an object, wherein the plurality of images include a first image and a second image, and wherein the second image is at a viewpoint different than the first image;

identify a first point of the object in the first image;

identify a second line in the second image based on the first point in the first image;

identify one or more second points along the second line;

identify one or more first points along a first line intersecting the first point, wherein each point of the one or more first points is projected from the one or more second points;

determine that a second point of the one or more second points corresponds to the first point based on a color of each of the one or more first points and a color of each of the one or more second points;

compute a depth for the first point based on the first point and the second point, wherein the depth is computed in response to determining that the second point corresponds to the first point;

generate a depth map for the first image using the depth of the first point; and

generate a mesh for the object based on the depth map.

16. The system of claim 15, wherein the second line has a direction of an epipolar line for the first image.

17. The system of claim 15, wherein the second point is an estimate of where the first point is in the second image.

18. The system of claim 15, wherein a color gradient is computed using colors of the one or more first points and the one or more second points, and wherein the color gradient is used to compute the depth.

19. The system of claim 15, wherein the first point is a color edge of the object, and wherein the second point corresponds to the first point such that the second point is the color edge of the object.

20. The system of claim 15, wherein the light field is unstructured.