Image capture for virtual reality displays

• US 10,412,373 B2
• Filed: 04/28/2017
• Issued: 09/10/2019
• Est. Priority Date: 04/15/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Pitts, Colvin
• Primary Examiner(s): Chen, Yu
• Application Number:
  US15/582,237
• Publication Number:
  US 20170237971A1
• Time in Patent Office: 865 Days
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/388 Volumetric displays, i.e. systems where the image is built up from picture elements distributed through a volume
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules

Abstract

A light-field camera system such as a tiled camera array may be used to capture a light-field of an environment. The tiled camera array may be a tiered camera array with a first plurality of cameras and a second plurality of cameras that are arranged more densely, but have lower resolution, than those of the first plurality of cameras. The first plurality of cameras may be interspersed among the second plurality of cameras. The first and second pluralities may cooperate to capture the light-field. According to one method, a subview may be captured by each camera of the first and second pluralities. Estimated world properties of the environment may be computed for each subview. A confidence map may be generated to indicate a level of confidence in the estimated world properties for each subview. The confidence maps and subviews may be used to generate a virtual view of the environment.

Claims

1. A method for generating a virtual view of an environment, the method comprising:

with a light-field camera system, capturing video data depicting the environment, the video data comprising a plurality of subviews that provide a plurality of views of the environment;

computing estimated world properties of the environment;

for each of the subviews, generating a confidence map indicative of a level of confidence in the estimated world properties of each of a plurality of regions within the subview, wherein generating the confidence map comprises generating a tertiary confidence map for a tertiary subview; and

using the subviews and the confidence maps to generate the virtual view, wherein using the subviews and the confidence maps comprises selecting, for inclusion in the virtual view, one or more regions of the tertiary subview having higher confidence in the tertiary confidence map than corresponding regions of a secondary confidence map.

2. The method of claim 1, wherein computing the estimated world properties comprises ascertaining one or more shapes of one or more objects appearing in the environment.

3. The method of claim 2, wherein computing the estimated world properties further comprises, for each of the subviews, generating a depth map indicative of distance between one or more objects appearing in the subview, and the one of a plurality of cameras of the light-field cameras system, to which the subview pertains.

4. The method of claim 1, wherein computing the estimated world properties comprises ascertaining a reflectance of at least part of one or more objects appearing in the environment.

5. The method of claim 1, wherein computing the estimated world properties comprises ascertaining a specularity of at least part of one or more objects appearing in the environment.

6. The method of claim 1, wherein computing the estimated world properties comprises ascertaining at least one of a translucency of at least part of one or more objects appearing in the environment.

7. The method of claim 1, further comprising, for each of the subviews, prior to generating the confidence map, calculating a confidence level for each of the plurality of regions of the subview.

8. The method of claim 7, wherein calculating the confidence level for each of the plurality of regions comprises, for each region, comparing a first subset of the video data corresponding to the region with one or more other subsets of the video data from other subviews that depict a portion of the environment similar to that depicted in the region.

9. The method of claim 1, wherein:

the light-field camera system comprises a tiled camera array comprising a plurality of cameras; and

capturing the video data comprises capturing one of the subviews with each of the cameras.

10. The method of claim 9, wherein:

generating the confidence map for each of the plurality of subviews comprises;

generating a primary confidence map for a primary subview of the plurality of subviews; and

generating the secondary confidence map for a secondary subview of the plurality of subviews; and

using the confidence maps and the subviews to generate the virtual view comprises;

selecting, for inclusion in the virtual view, one or more regions of the primary subview having high confidence in the primary confidence map; and

selecting, for inclusion in the virtual view, one or more regions of the secondary subview having higher confidence in the secondary confidence map than corresponding regions of the primary confidence map.

11. The method of claim 10, wherein:

the plurality of cameras comprises;

a first plurality of cameras, each of which has a first resolution; and

a second plurality of cameras, each of which has a second resolution lower than the first resolution; and

capturing the video data comprises;

capturing the primary subview with one of the first plurality of cameras; and

capturing the secondary subview with one of the second plurality of cameras.

12. The method of claim 11, wherein:

the plurality of cameras further comprises a third plurality of cameras, each of which has a third resolution lower than the second resolution;

capturing the video data comprises capturing the tertiary subview with one of the third plurality of cameras.

13. A non-transitory computer-readable medium for generating a virtual view of an environment, the non-transitory computer-readable medium comprising instructions stored thereon, that when executed by a processor, perform the steps of:

receiving video data depicting the environment after capture of the video data by a light-field camera system, the video data comprising a plurality of subviews that provide a plurality of views of the environment;

computing estimated world properties of the environment;

for each of the subviews, generating a confidence map indicative of a level of confidence in the estimated world properties of each of a plurality of regions within the subview, wherein generating the confidence map comprises generating a tertiary confidence map for a tertiary subview; and

using the subviews and the confidence maps to generate the virtual view, wherein using the subviews and the confidence maps comprises selecting, for inclusion in the virtual view, one or more regions of the tertiary subview having higher confidence in the tertiary confidence map than corresponding regions of a secondary confidence map.

14. The non-transitory computer-readable medium of claim 13, wherein computing the estimated world properties comprises ascertaining one or more shapes of one or more objects appearing in the environment.

15. The non-transitory computer-readable medium of claim 13, wherein computing the estimated world properties further comprises, for each of the subviews, generating a depth map indicative of distance between one or more objects appearing in the subview, and the one of a plurality of cameras of the light-field camera system, to which the subview pertains.

16. The non-transitory computer-readable medium of claim 13, wherein computing the estimated world properties comprises at least one of:

ascertaining a reflectance of at least part of one or more objects appearing in the environment;

ascertaining a specularity of at least part of one or more objects appearing in the environment; and

ascertaining at least one of a translucency of at least part of one or more objects appearing in the environment.

17. The non-transitory computer-readable medium of claim 13, further comprising instructions stored thereon, that when executed by a processor, for each of the subviews, prior to generating the confidence map, calculate a confidence level for each of the plurality of regions of the subview by, for each region, comparing a first subset of the video data corresponding to the region with one or more other subsets of the video data from other subviews that depict a portion of the environment similar to that depicted in the region.

18. The non-transitory computer-readable medium of claim 13, wherein:

the light-field camera system comprises a tiled camera array comprising a plurality of cameras; and

the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, cause the light-field camera system to capture the video data by capturing one of the subviews with each of the cameras.

19. The non-transitory computer-readable medium of claim 18, wherein:

generating the confidence map for each of the plurality of subviews comprises;

generating a primary confidence map for a primary subview of the plurality of subviews; and

generating the secondary confidence map for a secondary subview of the plurality of subviews; and

using the confidence maps and the subviews to generate the virtual view comprises;

selecting, for inclusion in the virtual view, one or more regions of the primary subview having high confidence in the primary confidence map; and

selecting, for inclusion in the virtual view, one or more regions of the secondary subview having higher confidence in the secondary confidence map than corresponding regions of the primary confidence map.

20. The non-transitory computer-readable medium of claim 19, wherein:

the plurality of cameras comprises;

a first plurality of cameras, each of which has a first resolution; and

a second plurality of cameras, each of which has a second resolution lower than the first resolution; and

capturing the video data comprises;

capturing the primary subview with one of the first plurality of cameras; and

capturing the secondary subview with one of the second plurality of cameras.

21. The non-transitory computer-readable medium of claim 20, wherein:

the plurality of cameras further comprises a third plurality of cameras, each of which has a third resolution lower than the second resolution;

capturing the video data comprises capturing the tertiary subview with one of the third plurality of cameras.

22. A system for generating a virtual view of an environment, the system comprising:

a light-field camera system configured to capture video data depicting the environment, the video data comprising a plurality of subviews that provide a plurality of views of the environment; and

a processor configured to;

compute estimated world properties of the environment;

for each of the subviews, generate a confidence map indicative of a level of confidence in the estimated world properties of each of a plurality of regions within the subview, wherein the processor is further configured to generate the confidence map for each of the plurality of subviews by generating a tertiary confidence map for a tertiary subview;

use the subviews and the confidence maps to generate the virtual view, wherein the processor is further configured to use the confidence maps and the subviews to generate the virtual view by selecting, for inclusion in the virtual view, one or more regions of the tertiary subview having higher confidence in the tertiary confidence map than corresponding regions of a secondary confidence map.

23. The system of claim 22, wherein the processor is further configured to compute the estimated world properties by ascertaining one or more shapes of one or more objects appearing in the environment.

24. The system of claim 23, wherein the processor is further configured to compute the estimated world properties by, for each of the subviews, generating a depth map indicative of distance between one or more objects appearing in the subview, and the one of a plurality of cameras of the light-field camera system, to which the subview pertains.

25. The system of claim 22, wherein the processor is further configured to compute the estimated world properties by performing at least one of:

ascertaining a reflectance of at least part of one or more objects appearing in the environment;

ascertaining a specularity of at least part of one or more objects appearing in the environment; and

ascertaining at least one of a translucency of at least part of one or more objects appearing in the environment.

26. The system of claim 22, wherein the processor is further configured, for each of the subviews, prior to generating the confidence map, to calculate a confidence level for each of the plurality of regions of the subview by, for each region, comparing a first subset of the video data corresponding to the region with one or more other subsets of the video data from other subviews that depict a portion of the environment similar to that depicted in the region.

27. The system of claim 22, wherein:

the light-field camera system comprises a tiled camera array comprising a plurality of cameras; and

the light-field camera system is configured to capture the video data by capturing one of the subviews with each of the cameras.

28. The system of claim 27, wherein:

the processor is further configured to generate the confidence map for each of the plurality of subviews by;

generating a primary confidence map for a primary subview of the plurality of subviews; and

generating the secondary confidence map for a secondary subview of the plurality of subviews; and

the processor is further configured to use the confidence maps and the subviews to generate the virtual view by;

selecting, for inclusion in the virtual view, one or more regions of the primary subview having high confidence in the primary confidence map; and

selecting, for inclusion in the virtual view, one or more regions of the secondary subview having higher confidence in the secondary confidence map than corresponding regions of the primary confidence map.

29. The system of claim 28, wherein:

the plurality of cameras comprises;

a first plurality of cameras, each of which has a first resolution; and

a second plurality of cameras, each of which has a second resolution lower than the first resolution; and

the light-field camera system is further configured to capture the video data by;

capturing the primary subview with one of the first plurality of cameras; and

capturing the secondary subview with one of the second plurality of cameras.

30. The system of claim 29, wherein:

the plurality of cameras further comprises a third plurality of cameras, each of which has a third resolution lower than the second resolution;

the light-field camera system is further configured to capture the video data by capturing the tertiary subview with one of the third plurality of cameras.