Apparatus and method for capturing a scene using staggered triggering of dense camera arrays

• US 8,027,531 B2
• Filed: 07/21/2005
• Issued: 09/27/2011
• Est. Priority Date: 07/21/2004
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Joshi, Neel, Horowitz, Mark, Levoy, Marc C., Wilburn, Bennett
• Primary Examiner(s): Dulaney; Kathleen Y
• Application Number:
  US11/187,699
• Publication Number:
  US 20070030342A1
• Time in Patent Office: 2,259 Days
• Field of Search: 382/154, 382/103, 382/312, 348/47, 348/48, 348/51, 345/419
• US Class Current: 382/154
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise

Abstract

This invention relates to an apparatus and a method for video capture of a three-dimensional region of interest in a scene using an array of video cameras. The video cameras of the array are positioned for viewing the three-dimensional region of interest in the scene from their respective viewpoints. A triggering mechanism is provided for staggering the capture of a set of frames by the video cameras of the array. The apparatus has a processing unit for combining and operating on the set of frames captured by the array of cameras to generate a new visual output, such as high-speed video or spatio-temporal structure and motion models, that has a synthetic viewpoint of the three-dimensional region of interest. The processing involves spatio-temporal interpolation for determining the synthetic viewpoint space-time trajectory. In some embodiments, the apparatus computes a multibaseline spatio-temporal optical flow.

Claims

1. An apparatus for high speed video capture of a three-dimensional region of interest in a scene, said apparatus comprising:

a) an array of video cameras for viewing said three-dimensional region of interest from a corresponding set of viewpoints, wherein each of said video cameras has a frame rate and a corresponding frame time, wherein said frame time is inversely related to said frame rate;

b) a triggering mechanism for staggering in time the capture of a set of frames by said array of video cameras, wherein said video cameras are triggered in a sequence whereby temporally successive frames in said set of frames is captured by two or more different video cameras of said array of video cameras, wherein said staggering introduces a time delay between said successive frames, wherein said time delay is the delay between triggering two of said different video cameras, and wherein said time delay is less than said frame time of each of said video cameras; and

c) a processing unit for combining and operating on said set of frames to generate a visual output having a synthetic viewpoint of said three-dimensional region of interest, wherein the processing unit solves for a flow field at a (x,y,t) location of the synthetic viewpoint in a (x,y,t) view space, wherein the processing unit decomposes an optical flow into a first component, being a scene motion per pixel, and a second component, being a scene depth per pixel, wherein said synthetic viewpoint is moveable in space and time, wherein said processing unit comprises a spatio-temporal interpolation unit for performing a spatio-temporal interpolation operation on said three-dimensional region of interest.

2. The apparatus of claim 1, wherein said visual output is a high-speed video of said three-dimensional region of interest.

3. The apparatus of claim 2, wherein said spatio-temporal interpolation unit calculates a space-time trajectory of said synthetic viewpoint.

4. The apparatus of claim 2, wherein said array of video cameras is a dense array of video cameras.

5. The apparatus of claim 1, wherein said visual output is a space-time structure and motion model of said three-dimensional region of interest.

6. The apparatus of claim 1, wherein said synthetic viewpoint is a virtual viewpoint interpolated in space-time by said spatio-temporal interpolation unit from said set of viewpoints.

7. The apparatus of claim 1, wherein said synthetic viewpoint corresponds to at least one viewpoint selected from said set of viewpoints.

8. The apparatus of claim 1, wherein said spatio-temporal interpolation unit further comprises an optical flow processor for obtaining a multibaseline spatio-temporal optical flow of said three-dimensional region of interest.

9. The apparatus of claim 8, wherein said multibaseline spatio-temporal optical flow comprises a scene depth per pixel and a scene motion per pixel.

10. The apparatus of claim 1, further comprising a frame adjustment module for controlling the frame capture of said array of video cameras.

11. The apparatus of claim 10, wherein said frame adjustment module comprises a rolling shutter canceller.

12. The apparatus of claim 10, wherein said frame adjustment module comprises a frame timing adjuster.

13. The apparatus of claim 1, wherein said synthetic viewpoint moves according to said triggering sequence.

14. A method of capturing high speed video of a three-dimensional region of interest in a scene, said method comprising:

a) positioning an array of video cameras for viewing said three-dimensional region of interest from a corresponding set of viewpoints, wherein each of said video cameras has a frame rate and a corresponding frame time, wherein said frame time is inversely related to said frame rate;

b) staggering in time the capture of a set of frames by said array of video cameras, wherein said video cameras are triggered in a sequence whereby temporally successive frames in said set of frames is captured by two or more different video cameras of said array of video cameras, wherein said staggering introduces a time delay between said successive frames, wherein said time delay is the delay between triggering two of said different video cameras, and wherein said time delay is less than said frame time period of each of said video cameras;

c) combining said set of frames;

d) operating on said set of frames to generate a visual output having a synthetic viewpoint of said three-dimensional area of interest, wherein said operating comprises solving for a flow field at a (x,y,t) location of the synthetic viewpoint in a (x,y,t) view space, wherein said operating comprises decomposing an optical flow into a first component, being a scene motion per pixel, and a second component, being a scene depth per pixel, wherein said synthetic viewpoint is moveable in space and time; and

e) spatio-temporally interpolating said visual output.

15. The method of claim 14, wherein said spatio-temporal interpolation comprises obtaining of a multibaseline spatio-temporal optical flow.

16. The method of claim 14, wherein said spatio-temporal interpolation comprises obtaining a spatio-temporal trajectory of said synthetic viewpoint.

17. The method of claim 14, wherein said visual output is a space-time structure and motion model of said three-dimensional region of interest.

18. The method of claim 14, wherein said staggering comprises delivering a trigger signal and a clock signal to said video cameras belonging to said array of video cameras.

19. The method of claim 14, wherein said time delay is programmed in each of said video cameras belonging to said array of video cameras.

20. A method of capturing high speed video of a three-dimensional region of interest in a scene, said method comprising:

a) positioning an array of video cameras for viewing said three-dimensional region of interest from a corresponding set of viewpoints, wherein each of said video cameras has a frame rate and a corresponding frame time, wherein said frame time is inversely related to said frame rate;

b) staggering in time the capture of a set of frames by said array of video cameras, wherein said video cameras are triggered in a sequence whereby temporally successive frames in said set of frames is captured by two or more different video cameras of said array of video cameras, wherein said staggering introduces a time delay between said successive frames, wherein said time delay is the delay between triggering two of said different video cameras, wherein said time delay is less than said frame time period of each of said video cameras, and wherein said time delay between triggering said different video cameras is a predetermined user-selected phase delay Δ

programmed into each of the video cameras;

c) combining said set of frames;

d) operating on said set of frames to generate a visual output having a synthetic viewpoint of said three-dimensional area of interest, wherein said operating comprises solving for a flow field at a (x,y,t) location of the synthetic viewpoint in a (x,y,t) view space, wherein said synthetic viewpoint is moveable in space and time; and

e) spatio-temporally interpolating said visual output.