Wedge-based light-field video capture

• US 10,275,898 B1
• Filed: 05/09/2017
• Issued: 04/30/2019
• Est. Priority Date: 04/15/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Song, Alex, Frank, Jonathan, Hernandez Zaragoza, Julio C., Green, Orin, Cooper, Steve, Braunstein, Ariel, Milliron, Tim, Pitts, Colvin, Yasui, Yusuke, Shahhosseini, Saeid, Zhang, Bipeng
• Primary Examiner(s): Du, Haixia
• Application Number:
  US15/590,951
• Time in Patent Office: 721 Days
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

A combined video of a scene may be generated for applications such as virtual reality or augmented reality. In one method, a camera system may be oriented at a first orientation and used to capture first video of a first portion of the scene. The camera system may then be rotated to a second orientation and used to capture second video of a second portion of the scene that is offset from the first portion such that the first video and the second video each have an overlapping video portion depicting an overlapping portion of the scene in which the first portion and the second portion of the scene overlap with each other. The first and second portions may be combined together to generate the combined video, which may depict the first and second portions substantially without duplicative inclusion of the overlapping video portion.

Claims

1. A method for generating a combined video of a scene, the method comprising:

orienting a camera system at a first orientation;

with the camera system at the first orientation, capturing first video of a first portion of a scene, wherein capturing the first video includes applying a lighting pattern to the scene, the lighting pattern comprising at least a first lighting change occurring at a first time within a first timeline in which the first video is captured;

rotating the camera system from the first orientation to a second orientation;

with the camera system at the second orientation, capturing second video of a second portion, offset from the first portion, of the scene such that the first video and the second video each comprise an overlapping video portion depicting an overlapping portion of the scene in which the first portion and the second portion of the scene overlap with each other, wherein capturing the second video includes applying the lighting pattern to the scene such that the first lighting change occurs at the first time within a second timeline in which the second video is captured; and

at a processor, combining the first video and the second video to generate a combined video depicting the first portion and the second portion of the scene substantially without duplicative inclusion of the overlapping video portion.

2. The method of claim 1, further comprising:

rotating the camera system from the second orientation to a third orientation;

with the camera system at the third orientation, capturing third video of a third portion, offset from the first portion and the second portion, of the scene;

rotating the camera system from the third orientation to a fourth orientation;

with the camera system at the fourth orientation, capturing fourth video of a fourth portion, offset from the first portion, the second portion, and the third portion, of the scene; and

rotating the camera system from the fourth orientation to a fifth orientation;

with the camera system at the fifth orientation, capturing fifth video of a fifth portion, offset from the first portion, the second portion, the third portion, and the fourth portion of the scene;

wherein generating the combined video further comprises combining the third video, the fourth video, and the fifth video with the first video and the second video to generate the combined video depicting the first portion, the second portion, the third portion, the fourth portion, and the fifth portion of the scene.

3. The method of claim 1, wherein generating the combined video further comprises adding at least a first computer-generated element into at least one of the first video and the second video.

4. The method of claim 3, further comprising generating a depth map indicating a relative depth of objects in the first portion and the second portion of the scene;

wherein adding at least the first computer-generated element into at least one of the first video and the second video comprises positioning the first computer-generated element at a first depth within the depth map.

5. The method of claim 1, further comprising:

positioning a calibration chart within the first portion of the scene; and

calibrating the camera system by capturing first calibration video depicting the first portion of the scene with the calibration chart positioned within the first portion.

6. The method of claim 5, further comprising:

positioning the calibration chart within the overlapping portion of the scene;

wherein calibrating the camera system further comprises capturing overlapping calibration video depicting the overlapping portion with the calibration chart positioned within the overlapping portion.

7. A method for generating a combined video of a scene, the method comprising:

calibrating a camera system comprising a camera array having a plurality of cameras arranged in a regular pattern to generate a virtual camera system comprising a plurality of virtual cameras, each of which has virtual characteristics that mimic operating characteristics of a corresponding one of the plurality of cameras;

orienting the camera system at a first orientation;

with the camera system at the first orientation, capturing first video of a first portion of a scene;

rotating the camera system from the first orientation to a second orientation;

with the camera system at the second orientation, capturing second video of a second portion, offset from the first portion, of the scene such that the first video and the second video each comprise an overlapping video portion depicting an overlapping portion of the scene in which the first portion and the second portion of the scene overlap with each other; and

at a processor, combining the first video and the second video to generate a combined video depicting the first portion and the second portion of the scene substantially without duplicative inclusion of the overlapping video portion.

8. The method of claim 7, wherein the regular pattern comprises a planar shape.

9. The method of claim 7, wherein each of the cameras comprises a light-field camera.

10. The method of claim 7, wherein:

the plurality of cameras comprise a central camera positioned at a center of the regular pattern; and

rotating the camera system from the first orientation to the second orientation comprises rotating the camera system about an axis passing through a nodal point of the central camera.

11. The method of claim 7, wherein combining the first video with the second video comprises using the virtual camera system.

12. The method of claim 7, wherein generating the combined video further comprises adding at least a first computer-generated element into at least one of the first video and the second video.

13. The method of claim 12, further comprising:

generating a depth map indicating a relative depth of objects in the first portion and the second portion of the scene; and

wherein adding at least the first computer-generated element into at least one of the first video and the second video comprises positioning the first computer-generated element at a first depth within the depth map.

14. A method for generating a combined video of a scene, the method comprising:

orienting a camera system at a first orientation;

with the camera system at the first orientation, capturing first video of a first portion of a scene;

rotating the camera system from the first orientation to a second orientation;

with the camera system at the second orientation, capturing second video of a second portion, offset from the first portion, of the scene such that the first video and the second video each comprise an overlapping video portion depicting an overlapping portion of the scene in which the first portion and the second portion of the scene overlap with each other;

at a processor, combining the first video and the second video to generate a combined video depicting the first portion and the second portion of the scene substantially without duplicative inclusion of the overlapping video portion; and

prior to capturing the first video, designating a subset of the first portion of the scene as a safe zone that is out-side the overlapping portion.

15. The method of claim 14, wherein designating the subset of the first portion of the scene as the safe zone comprises, with a laser, projecting a beam into the first portion of the scene, proximate a boundary of the safe zone.

16. A system for generating a combined video of a scene, the system comprising:

a camera system that is rotatable between at least a first orientation and a second orientation, wherein the camera system is configured to;

in the first orientation, capture first video of a first portion of a scene; and

in the second orientation, capture second video of a second portion, offset from the first portion, of the scene such that the first video and the second video each comprise an overlapping video portion depicting an overlapping portion of the scene in which the first portion and the second portion of the scene overlap with each other;

a processor, communicatively coupled to the camera system, configured to combine the first video and the second video to generate a combined video depicting the first portion and the second portion of the scene substantially without duplicative inclusion of the overlapping video portion; and

a laser configured to designate a subset of the first portion of the scene as a safe zone that is outside the overlapping portion by projecting a beam into the first portion of the scene, proximate a boundary of the safe zone.

17. The system of claim 16, wherein the processor is further configured to:

generate a depth map indicating a relative depth of objects in the first portion and the second portion of the scene; and

generate the combined video further by adding at least a first computer-generated element into at least one of the first video and the second video by positioning the first computer-generated element at a first depth within the depth map.

18. The system of claim 16, further comprising a calibration chart configured to be positioned within the first portion of the scene or within the overlapping portion of the scene;

wherein the camera system is further configured to undergo calibration by;

capturing first calibration video depicting the first portion of the scene with the calibration chart positioned within the first portion; and

capturing overlapping calibration video depicting the overlapping portion with the calibration chart positioned within the overlapping portion.

19. A system for generating a combined video of a scene, the system comprising:

a camera system that is rotatable between at least a first orientation and a second orientation, the camera system comprising a camera array having a plurality of cameras arranged in a regular pattern, and wherein the camera system is configured to;

in the first orientation, capture first video of a first portion of a scene; and

in the second orientation, capture second video of a second portion, offset from the first portion, of the scene such that the first video and the second video each comprise an overlapping video portion depicting an overlapping portion of the scene in which the first portion and the second portion of the scene overlap with each other; and

a processor, communicatively coupled to the camera system, configured to;

generate a virtual camera system via calibration of the camera system, the virtual camera system comprising a plurality of virtual cameras, each of which has virtual characteristics that mimic operating characteristics of a corresponding one of the plurality of cameras; and

combine the first video and the second video using the virtual camera system to generate a combined video depicting the first portion and the second portion of the scene substantially without duplicative inclusion of the overlapping video portion.

20. The system of claim 19, wherein the regular pattern comprises a planar shape.

21. The system of claim 19, wherein:

the plurality of cameras comprise a central camera positioned at a center of the regular pattern; and

the camera system is further configured such that rotating the camera system from the first orientation to the second orientation comprises rotating the camera system about an axis passing through a nodal point of the central camera.

22. The system of claim 19, wherein:

the camera system is further rotatable between the first orientation, the second orientation, a third orientation, a fourth orientation, and a fifth orientation, wherein the camera system is further configured to;

in the third orientation, capturing third video of a third portion, offset from the first portion and the second portion, of the scene;

in the fourth orientation, capturing fourth video of a fourth portion, offset from the first portion, the second portion, and the third portion, of the scene; and

in the fifth orientation, capturing fifth video of a fifth portion, offset from the first portion, the second portion, the third portion, and the fourth portion of the scene; and

the processor is further configured to generate the combined video by combining the third video, the fourth video, and the fifth video with the first video and the second video to generate the combined video depicting the first portion, the second portion, the third portion, the fourth portion, and the fifth portion of the scene.

23. The system of claim 19, further comprising a calibration chart configured to be positioned within the first portion of the scene or within the overlapping portion of the scene;

wherein the camera system is further configured to undergo calibration by;

capturing first calibration video depicting the first portion of the scene with the calibration chart positioned within the first portion; and

capturing overlapping calibration video depicting the overlapping portion with the calibration chart positioned within the overlapping portion.

24. The system of claim 19, wherein the processor is further configured to:

generate a depth map indicating a relative depth of objects in the first portion and the second portion of the scene; and

generate the combined video further by adding at least a first computer-generated element into at least one of the first video and the second video by positioning the first computer-generated element at a first depth within the depth map.