IMAGE CAPTURE FOR VIRTUAL REALITY DISPLAYS

• US 20170237971A1
• Filed: 04/28/2017
• Published: 08/17/2017
• Est. Priority Date: 04/15/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Pitts, Colvin
• Granted Patent: US 10,412,373 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/388 Volumetric displays, i.e. systems where the image is built up from picture elements distributed through a volume
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules

Abstract

A light-field camera system such as a tiled camera array may be used to capture a light-field of an environment. The tiled camera array may be a tiered camera array with a first plurality of cameras and a second plurality of cameras that are arranged more densely, but have lower resolution, than those of the first plurality of cameras. The first plurality of cameras may be interspersed among the second plurality of cameras. The first and second pluralities may cooperate to capture the light-field. According to one method, a subview may be captured by each camera of the first and second pluralities. Estimated world properties of the environment may be computed for each subview. A confidence map may be generated to indicate a level of confidence in the estimated world properties for each subview. The confidence maps and subviews may be used to generate a virtual view of the environment.

Claims

1. A tiled camera array comprising:

a framework comprising an imaging surface;

a first plurality of cameras arranged, with a first density, in a first tiled array on the imaging surface, wherein each camera of the first plurality of cameras comprises a first resolution; and

a second plurality of cameras arranged, with a second density, in a second tiled array on the imaging surface, wherein each camera of the second plurality of cameras comprises a second resolution;

wherein;

the first tiled array is interspersed among the second tiled array;

the first resolution is greater than the second resolution;

the second density is greater than the first density; and

the first plurality of cameras and the second plurality of cameras cooperate with each other to capture a light-field volume within an environment.

2. The tiled camera array of claim 1, further comprising a processor configured to use the light-field volume to generate a virtual view depicting the environment from a virtual viewpoint.

3. The tiled camera array of claim 1, wherein the first tiled array comprises a first hexagonal lattice.

4. The tiled camera array of claim 3, wherein the second tiled array comprises a second hexagonal lattice that is denser than the first hexagonal lattice, the second tiled array defining a first plurality of spaced-apart voids that accommodate the first hexagonal lattice.

5. The tiled camera array of claim 4, further comprising a third plurality of cameras arranged, with a third density, in a third tiled array on the imaging surface, wherein each camera of the third plurality of cameras comprises a third resolution;

wherein;

the first and second tiled arrays are interspersed among the third tiled array;

the second resolution is greater than the third resolution;

the third density is greater than the second density; and

the third plurality of cameras is configured to cooperate with the first plurality of cameras and the second plurality of cameras to capture the light-field volume.

6. The tiled camera array of claim 5, wherein the third tiled array comprises a third hexagonal lattice that is denser than the second hexagonal lattice, the third tiled array defining a second plurality of spaced-apart voids that accommodate the second hexagonal lattice.

7. The tiled camera array of claim 1, wherein the imaging surface comprises a hexagonal shape.

8. A method for generating a virtual view of an environment, the method comprising:

with a light-field camera system, capturing video data depicting the environment, the video data comprising a plurality of subviews that provide a plurality of views of the environment;

computing estimated world properties of the environment;

for each of the subviews, generating a confidence map indicative of a level of confidence in the estimated world properties of each of a plurality of regions within the subview; and

using the subviews and the confidence maps to generate the virtual view.

9. The method of claim 8, wherein computing the estimated world properties comprises ascertaining one or more shapes of one or more objects appearing in the environment.

10. The method of claim 9, wherein computing the estimated world properties further comprises, for each of the subviews, generating a depth map indicative of distance between one or more objects appearing in the subview, and the one of a plurality of cameras of the light-field cameras system, to which the subview pertains.

11. The method of claim 8, wherein computing the estimated world properties comprises ascertaining a reflectance of at least part of one or more objects appearing in the environment.

12. The method of claim 8, wherein computing the estimated world properties comprises ascertaining a specularity of at least part of one or more objects appearing in the environment.

13. The method of claim 8, wherein computing the estimated world properties comprises ascertaining at least one of a translucency of at least part of one or more objects appearing in the environment.

14. The method of claim 8, further comprising, for each of the subviews, prior to generating the confidence map, calculating a confidence level for each of the plurality of regions of the subview.

15. The method of claim 14, wherein calculating the confidence level for each of the plurality of regions comprises, for each region, comparing a first subset of the video data corresponding to the region with one or more other subsets of the video data from other subviews that depict a portion of the environment similar to that depicted in the region.

16. The method of claim 8, wherein:

the light-field camera system comprises a tiled camera array comprising a plurality of cameras; and

capturing the video data comprises capturing one of the subviews with each of the cameras.

17. The method of claim 16, wherein:

generating the confidence map for each of the plurality of subviews comprises;

generating a primary confidence map for a primary subview of the plurality of subviews; and

generating a secondary confidence map for a secondary subview of the plurality of subviews; and

using the confidence maps and the subviews to generate the virtual view comprises;

selecting, for inclusion in the virtual view, one or more regions of the primary subview having high confidence in the primary confidence map; and

selecting, for inclusion in the virtual view, one or more regions of the secondary subview having higher confidence in the secondary confidence map than corresponding regions of the primary confidence map.

18. The method of claim 17, wherein:

the plurality of cameras comprises;

a first plurality of cameras, each of which has a first resolution; and

a second plurality of cameras, each of which has a second resolution lower than the first resolution; and

capturing the video data comprises;

capturing the primary subview with one of the first plurality of cameras; and

capturing the secondary subview with one of the second plurality of cameras.

19. The method of claim 18, wherein:

the plurality of cameras further comprises a third plurality of cameras, each of which has a third resolution lower than the second resolution;

capturing the video data comprises capturing a tertiary subview with one of the third plurality of cameras;

generating the confidence map for each of the plurality of subviews further comprises generating a tertiary confidence map for the tertiary subview; and

using the confidence maps and the subviews to generate the virtual view further comprises selecting, for inclusion in the virtual view, one or more regions of the tertiary subview having higher confidence in the tertiary confidence map than corresponding regions of the secondary confidence map.

20. A non-transitory computer-readable medium for generating a virtual view of an environment, the non-transitory computer-readable medium comprising instructions stored thereon, that when executed by a processor, perform the steps of:

receiving video data depicting the environment after capture of the video data by a light-field camera system, the video data comprising a plurality of subviews that provide a plurality of views of the environment;

computing estimated world properties of the environment;

for each of the subviews, generating a confidence map indicative of a level of confidence in the estimated world properties of each of a plurality of regions within the subview; and

using the subviews and the confidence maps to generate the virtual view.

21. The non-transitory computer-readable medium of claim 20, wherein computing the estimated world properties comprises ascertaining one or more shapes of one or more objects appearing in the environment.

22. The non-transitory computer-readable medium of claim 20, wherein computing the estimated world properties further comprises, for each of the subviews, generating a depth map indicative of distance between one or more objects appearing in the subview, and the one of a plurality of cameras of the light-field camera system, to which the subview pertains.

23. The non-transitory computer-readable medium of claim 20, wherein computing the estimated world properties comprises at least one of:

ascertaining a reflectance of at least part of one or more objects appearing in the environment;

ascertaining a specularity of at least part of one or more objects appearing in the environment; and

ascertaining at least one of a translucency of at least part of one or more objects appearing in the environment.

24. The non-transitory computer-readable medium of claim 20, further comprising instructions stored thereon, that when executed by a processor, for each of the subviews, prior to generating the confidence map, calculate a confidence level for each of the plurality of regions of the subview by, for each region, comparing a first subset of the video data corresponding to the region with one or more other subsets of the video data from other subviews that depict a portion of the environment similar to that depicted in the region.

25. The non-transitory computer-readable medium of claim 20, wherein:

the light-field camera system comprises a tiled camera array comprising a plurality of cameras; and

the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, cause the light-field camera system to capture the video data by capturing one of the subviews with each of the cameras.

26. The non-transitory computer-readable medium of claim 25, wherein:

generating the confidence map for each of the plurality of subviews comprises;

generating a primary confidence map for a primary subview of the plurality of subviews; and

generating a secondary confidence map for a secondary subview of the plurality of subviews; and

using the confidence maps and the subviews to generate the virtual view comprises;

selecting, for inclusion in the virtual view, one or more regions of the primary subview having high confidence in the primary confidence map; and

selecting, for inclusion in the virtual view, one or more regions of the secondary subview having higher confidence in the secondary confidence map than corresponding regions of the primary confidence map.

27. The non-transitory computer-readable medium of claim 26, wherein:

the plurality of cameras comprises;

a first plurality of cameras, each of which has a first resolution; and

a second plurality of cameras, each of which has a second resolution lower than the first resolution; and

capturing the video data comprises;

capturing the primary subview with one of the first plurality of cameras; and

capturing the secondary subview with one of the second plurality of cameras.

28. The non-transitory computer-readable medium of claim 27, wherein:

the plurality of cameras further comprises a third plurality of cameras, each of which has a third resolution lower than the second resolution;

capturing the video data comprises capturing a tertiary subview with one of the third plurality of cameras;

generating the confidence map for each of the plurality of subviews further comprises generating a tertiary confidence map for the tertiary subview; and

using the confidence maps and the subviews to generate the virtual view further comprises selecting, for inclusion in the virtual view, one or more regions of the tertiary subview having higher confidence in the tertiary confidence map than corresponding regions of the secondary confidence map.

29. A system for generating a virtual view of an environment, the system comprising:

a light-field camera system configured to capture video data depicting the environment, the video data comprising a plurality of subviews that provide a plurality of views of the environment; and

a processor configured to;

compute estimated world properties of the environment;

for each of the subviews, generate a confidence map indicative of a level of confidence in the estimated world properties of each of a plurality of regions within the subview; and

use the subviews and the confidence maps to generate the virtual view.

30. The system of claim 29, wherein the processor is further configured to compute the estimated world properties by ascertaining one or more shapes of one or more objects appearing in the environment.

31. The system of claim 30, wherein the processor is further configured to compute the estimated world properties by, for each of the subviews, generating a depth map indicative of distance between one or more objects appearing in the subview, and the one of a plurality of cameras of the light-field camera system, to which the subview pertains.

32. The system of claim 29, wherein the processor is further configured to compute the estimated world properties by performing at least one of:

ascertaining a reflectance of at least part of one or more objects appearing in the environment;

ascertaining a specularity of at least part of one or more objects appearing in the environment; and

ascertaining at least one of a translucency of at least part of one or more objects appearing in the environment.

33. The system of claim 29, wherein the processor is further configured, for each of the subviews, prior to generating the confidence map, to calculate a confidence level for each of the plurality of regions of the subview by, for each region, comparing a first subset of the video data corresponding to the region with one or more other subsets of the video data from other subviews that depict a portion of the environment similar to that depicted in the region.

34. The system of claim 29, wherein:

the light-field camera system comprises a tiled camera array comprising a plurality of cameras; and

the light-field camera system is configured to capture the video data by capturing one of the subviews with each of the cameras.

35. The system of claim 34, wherein:

the processor is further configured to generate the confidence map for each of the plurality of subviews by;

generating a primary confidence map for a primary subview of the plurality of subviews; and

generating a secondary confidence map for a secondary subview of the plurality of subviews; and

the processor is further configured to use the confidence maps and the subviews to generate the virtual view by;

selecting, for inclusion in the virtual view, one or more regions of the primary subview having high confidence in the primary confidence map; and

selecting, for inclusion in the virtual view, one or more regions of the secondary subview having higher confidence in the secondary confidence map than corresponding regions of the primary confidence map.

36. The system of claim 35, wherein:

the plurality of cameras comprises;

a first plurality of cameras, each of which has a first resolution; and

a second plurality of cameras, each of which has a second resolution lower than the first resolution; and

the light-field camera system is further configured to capture the video data by;

capturing the primary subview with one of the first plurality of cameras; and

capturing the secondary subview with one of the second plurality of cameras.

37. The system of claim 36, wherein:

the plurality of cameras further comprises a third plurality of cameras, each of which has a third resolution lower than the second resolution;

the light-field camera system is further configured to capture the video data by capturing a tertiary subview with one of the third plurality of cameras;

the processor is further configured to generate the confidence map for each of the plurality of subviews by generating a tertiary confidence map for the tertiary subview; and

the processor is further configured to use the confidence maps and the subviews to generate the virtual view by selecting, for inclusion in the virtual view, one or more regions of the tertiary subview having higher confidence in the tertiary confidence map than corresponding regions of the secondary confidence map.