Multi-view back-projection to a light-field

• US 10,354,399 B2
• Filed: 05/25/2017
• Issued: 07/16/2019
• Est. Priority Date: 05/25/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Wang, Zejing, Nonn, Thomas
• Primary Examiner(s): Aghevli, Reza
• Application Number:
  US15/605,037
• Publication Number:
  US 20180342075A1
• Time in Patent Office: 782 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/218 using spatial multiplexing
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules

Abstract

Dense light-field data can be generated from image data that does not include light-field data, or from image data that includes sparse light-field data. In at least one embodiment, the source light-field data may include one or more sub-aperture images that may be used to reconstruct the light-field in denser form. In other embodiments, the source data can take other forms. Examples include data derived from or ancillary to a set of sub-aperture images, synthetic data, or captured image data that does not include full light-field data. Interpolation, back-projection, and/or other techniques are used in connection with source sub-aperture images or their equivalents, to generate dense light-field data.

Claims

1. A method for generating a reconstructed light-field, the method comprising:

from a storage device, retrieving a first image depicting a light-field from a first viewpoint;

in a processor, obtaining a first value of interest pertinent to the first image; and

in the processor, using the first value of interest to generate the reconstructed light-field by extrapolating corresponding values, corresponding to the first value of interest, for a first plurality of other images;

wherein;

the first plurality of other images comprises at least a second image depicting the light-field from a second viewpoint, and a third image depicting the light-field from a third viewpoint; and

using the first value of interest to extrapolate the corresponding values comprises;

based on relative positions of the first viewpoint and the second viewpoint, extrapolating a second value of interest pertinent to the second image; and

based on relative positions of the first viewpoint and the third viewpoint, extrapolating a third value of interest pertinent to the third image,

wherein;

the second viewpoint is displaced from the first viewpoint along a second direction; and

the third viewpoint is displaced form the first viewpoint along a third direction opposite to the second direction,

wherein;

using the first value of interest to generate the reconstructed light-field further comprises extrapolating corresponding values, corresponding to the first value of interest, for a second plurality of other images;

the second plurality of other images comprises at least a fourth image depicting the light-field from a fourth viewpoint, and a fifth image depicting the light-field from a fifth viewpoint;

the fourth viewpoint is displaced from the first viewpoint along a fourth direction perpendicular to the second direction and the third direction;

the fifth viewpoint is displaced from the first viewpoint along a fifth direction opposite to the fourth direction; and

using the first value of interest to extrapolate the corresponding values comprises;

based on relative positions of the first viewpoint and the fourth viewpoint, extrapolating a fourth value of interest pertinent to the fourth image; and

based on relative positions of the first viewpoint and the fifth viewpoint, extrapolating a fifth value of interest pertinent to the fifth image.

2. The method of claim 1, wherein the first value of interest comprises a depth of at least one object depicted in the first image.

3. The method of claim 2, wherein:

extrapolating the corresponding values, for the first plurality of other images comprises;

back-projecting a second depth map for the second image; and

back-projecting a third depth map for the third image; and

the method further comprises using weighted averages of at least the second depth map and the third depth map to generate a reconstructed depth-field for objects within the reconstructed light-field.

4. The method of claim 1, wherein the first value of interest comprises an image value of at least one portion of the first image.

5. The method of claim 4, wherein extrapolating the corresponding values, for the first plurality of other images comprises:

back-projecting a second light-field for the second image;

back-projecting a third light-field for the third image; and

using weighted averages of at least the second light-field and the third light-field to generate the reconstructed light-field.

6. The method of claim 4, wherein extrapolating the corresponding values, for the first plurality of other images comprises:

generating a reconstructed depth-field for objects within the reconstructed light-field; and

using the reconstructed depth-field to back-project image values for the reconstructed light-field.

7. The method of claim 1, further comprising, after generation of the reconstructed light-field, using the reconstructed light-field to project a new image from a new viewpoint.

8. The method of claim 1, wherein:

the first image comprises a first sub-aperture image projected from light-field data captured with a light-field camera;

the second image comprises a second sub-aperture image projected from the light-field data; and

the third image comprises a third sub-aperture image projected from the light-field data.

9. The method of claim 1, wherein:

the first image comprises first image data derived from a first sub-aperture image projected from light-field data captured with a light-field camera;

the second image comprises second image data derived from a second sub-aperture image projected from the light-field data; and

the third image comprises third image data derived from a third sub-aperture image projected from the light-field data.

10. The method of claim 1, wherein each of the first image, the second image, and the third image consists exclusively of data generated without involving capture of image data by a camera.

11. The method of claim 1, wherein each of the first image, the second image, and the third image comprises a conventional image captured with a camera that is not a light-field camera.

12. A non-transitory computer-readable medium for generating a reconstructed light-field, the non-transitory computer-readable medium comprising instructions stored thereon, that when executed by a processor, perform the steps of:

causing a storage device to retrieve a first image depicting a light-field from a first viewpoint;

obtaining a first value of interest pertinent to the first image; and

using the first value of interest to generate the reconstructed light-field by extrapolating corresponding values, corresponding to the first value of interest, for a first plurality of other images;

wherein;

the first plurality of other images comprises at least a second image depicting the light-field from a second viewpoint, and a third image depicting the light-field from a third viewpoint; and

using the first value of interest to extrapolate the corresponding values comprises;

based on relative positions of the first viewpoint and the second viewpoint, extrapolating a second value of interest pertinent to the second image; and

based on relative positions of the first viewpoint and the third viewpoint, extrapolating a third value of interest pertinent to the third image,

wherein;

the second viewpoint is displaced from the first viewpoint along a second direction;

the third viewpoint is displaced form the first viewpoint along a third direction opposite to the second direction;

using the first value of interest to generate the reconstructed light-field further comprises extrapolating corresponding values, corresponding to the first value of interest, for a second plurality of other images;

the second plurality of other images comprises at least a fourth image depicting the light-field from a fourth viewpoint, and a fifth image depicting the light-field from a fifth viewpoint;

the fourth viewpoint is displaced from the first viewpoint along a fourth direction perpendicular to the second direction and the third direction;

the fifth viewpoint is displaced from the first viewpoint along a fifth direction opposite to the fourth direction; and

using the first value of interest to extrapolate the corresponding values comprises;

based on relative positions of the first viewpoint and the fourth viewpoint, extrapolating a fourth value of interest pertinent to the fourth image; and

based on relative positions of the first viewpoint and the fifth viewpoint, extrapolating a fifth value of interest pertinent to the fifth image.

13. The non-transitory computer-readable medium of claim 12, wherein:

the first value of interest comprises a depth of at least one object depicted in the first image;

extrapolating the corresponding values, for the first plurality of other images comprises;

back-projecting a second depth map for the second image; and

back-projecting a third depth map for the third image; and

the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, use weighted averages of at least the second depth map and the third depth map to generate a reconstructed depth-field for objects within the reconstructed light-field.

14. The non-transitory computer-readable medium of claim 12, wherein:

the first value of interest comprises an image value of at least one portion of the first image; and

extrapolating the corresponding values, for the first plurality of other images comprises;

back-projecting a second light-field for the second image;

back-projecting a third light-field for the third image; and

using weighted averages of at least the second light-field and the third light-field to generate the reconstructed light-field.

15. The non-transitory computer-readable medium of claim 12, wherein:

the first value of interest comprises an image value of at least one portion of the first image; and

extrapolating the corresponding values, for the first plurality of other images comprises;

generating a reconstructed depth-field for objects within the reconstructed light-field; and

using the reconstructed depth-field to back-project image values for the reconstructed light-field.

16. The non-transitory computer-readable medium of claim 12, further comprising instructions stored thereon, that when executed by a processor, after generation of the reconstructed light-field, use the reconstructed light-field to project a new image from a new viewpoint.

17. The non-transitory computer-readable medium of claim 12, wherein:

the first image comprises a first sub-aperture image projected from light-field data captured with a light-field camera;

the second image comprises a second sub-aperture image projected from the light-field data; and

the third image comprises a third sub-aperture image projected from the light-field data.

18. The non-transitory computer-readable medium of claim 12, wherein:

the first image comprises first image data derived from a first sub-aperture image projected from light-field data captured with a light-field camera;

the second image comprises second image data derived from a second sub-aperture image projected from the light-field data; and

the third image comprises third image data derived from a third sub-aperture image projected from the light-field data.

19. The non-transitory computer-readable medium of claim 12, wherein each of the first image, the second image, and the third image consists exclusively of data generated without involving capture of image data by a camera.

20. The non-transitory computer-readable medium of claim 12, wherein each of the first image, the second image, and the third image comprises a conventional image captured with a camera that is not a light-field camera.

21. A system for generating a reconstructed light-field, the system comprising:

a storage device configured to retrieve a first image depicting a light-field from a first viewpoint; and

a processor, communicatively coupled to the storage device, configured to;

obtain a first value of interest pertinent to the first image; and

use the first value of interest to generate the reconstructed light-field by extrapolating corresponding values, corresponding to the first value of interest, for a first plurality of other images;

wherein;

the first plurality of other images comprises at least a second image depicting the light-field from a second viewpoint, and a third image depicting the light-field from a third viewpoint; and

the processor is further configured to use the first value of interest to extrapolate the corresponding values by;

based on relative positions of the first viewpoint and the second viewpoint, extrapolating a second value of interest pertinent to the second image; and

based on relative positions of the first viewpoint and the third viewpoint, extrapolating a third value of interest pertinent to the third image,

wherein;

the second viewpoint is displaced from the first viewpoint along a second direction,

the third viewpoint is displaced form the first viewpoint along a third direction opposite to the second direction;

using the first value of interest to generate the reconstructed light-field further comprises extrapolating corresponding values, corresponding to the first value of interest, for a second plurality of other images;

the second plurality of other images comprises at least a fourth image depicting the light-field from a fourth viewpoint, and a fifth image depicting the light-field from a fifth viewpoint;

the fourth viewpoint is displaced from the first viewpoint along a fourth direction perpendicular to the second direction and the third direction;

the fifth viewpoint is displaced from the first viewpoint along a fifth direction opposite to the fourth direction; and

using the first value of interest to extrapolate the corresponding values comprises;

based on relative positions of the first viewpoint and the fourth viewpoint, extrapolating a fourth value of interest pertinent to the fourth image; and

based on relative positions of the first viewpoint and the fifth viewpoint, extrapolating a fifth value of interest pertinent to the fifth image.

22. The system of claim 21, wherein:

the first value of interest comprises a depth of at least one object depicted in the first image;

extrapolating the corresponding values, for the first plurality of other images comprises;

back-projecting a second depth map for the second image; and

back-projecting a third depth map for the third image; and

the processor is further configured to use weighted averages of at least the second depth map and the third depth map to generate a reconstructed depth-field for objects within the reconstructed light-field.

23. The system of claim 21, wherein:

the first value of interest comprises an image value of at least one portion of the first image; and

extrapolating the corresponding values, for the first plurality of other images comprises;

back-projecting a second light-field for the second image;

back-projecting a third light-field for the third image; and

using weighted averages of at least the second light-field and the third light-field to generate the reconstructed light-field.

24. The system of claim 21, wherein:

the first value of interest comprises an image value of at least one portion of the first image; and

extrapolating the corresponding values, for the first plurality of other images comprises;

generating a reconstructed depth-field for objects within the reconstructed light-field; and

using the reconstructed depth-field to back-project image values for the reconstructed light-field.

25. The system of claim 21, wherein the processor is further configured to, after generation of the reconstructed light-field, use the reconstructed light-field to project a new image from a new viewpoint.

26. The system of claim 21, wherein:

the first image comprises a first sub-aperture image projected from light-field data captured with a light-field camera;

the second image comprises a second sub-aperture image projected from the light-field data; and

the third image comprises a third sub-aperture image projected from the light-field data.

27. The system of claim 21, wherein:

the first image comprises first image data derived from a first sub-aperture image projected from light-field data captured with a light-field camera;

the second image comprises second image data derived from a second sub-aperture image projected from the light-field data; and

the third image comprises third image data derived from a third sub-aperture image projected from the light-field data.

28. The system of claim 21, wherein each of the first image, the second image, and the third image consists exclusively of data generated without involving capture of image data by a camera.

29. The system of claim 21, wherein each of the first image, the second image, and the third image comprises a conventional image captured with a camera that is not a light-field camera.