Apparatus and method for capturing a scene using staggered triggering of dense camera arrays

• US 20070030342A1
• Filed: 07/21/2005
• Published: 02/08/2007
• Est. Priority Date: 07/21/2004
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford University)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford University)
• Inventors: Joshi, Neel, Horowitz, Mark, Levoy, Marc, Wilburn, Bennett
• Granted Patent: US 8,027,531 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise

Abstract

This invention relates to an apparatus and a method for video capture of a three-dimensional region of interest in a scene using an array of video cameras. The video cameras of the array are positioned for viewing the three-dimensional region of interest in the scene from their respective viewpoints. A triggering mechanism is provided for staggering the capture of a set of frames by the video cameras of the array. The apparatus has a processing unit for combining and operating on the set of frames captured by the array of cameras to generate a new visual output, such as high-speed video or spatio-temporal structure and motion models, that has a synthetic viewpoint of the three-dimensional region of interest. The processing involves spatio-temporal interpolation for determining the synthetic viewpoint space-time trajectory. In some embodiments, the apparatus computes a multibaseline spatio-temporal optical flow.

Claims

1. An apparatus for video capture of a three-dimensional region of interest in a scene, said apparatus comprising:

a) an array of video cameras for viewing said three-dimensional region of interest from a corresponding set of viewpoints;

b) a triggering mechanism for staggering the capture of a set of frames by said array of video cameras;

c) a processing unit for combining and operating on said set of frames to generate a visual output having a synthetic viewpoint of said three-dimensional region of interest.

2. The apparatus of claim 1, wherein said processing unit further comprises a spatio-temporal interpolation unit for performing a spatio-temporal interpolation operation on said three-dimensional region of interest.

3. The apparatus of claim 2, wherein said visual output is a high-speed video of said three-dimensional region of interest.

4. The apparatus of claim 3, wherein said spatio-temporal interpolation unit calculates a space-time trajectory of said synthetic viewpoint.

5. The apparatus of claim 3, wherein said array of video cameras is a dense array of video cameras.

6. The apparatus of claim 2, wherein said visual output is a space-time structure and motion model of said three-dimensional region of interest.

7. The apparatus of claim 2, wherein said synthetic viewpoint is a virtual viewpoint interpolated in space-time by said spatio-temporal interpolation unit from said set of viewpoints.

8. The apparatus of claim 2, wherein said synthetic viewpoint corresponds to at least one viewpoint selected from said set of viewpoints.

9. The apparatus of claim 2, wherein said spatio-temporal interpolation unit further comprises an optical flow processor for obtaining a multibaseline spatio-temporal optical flow of said three-dimensional region of interest.

10. The apparatus of claim 9, wherein said multibaseline spatio-temporal optical flow comprises a scene depth per pixel and a scene motion per pixel.

11. The apparatus of claim 1, further comprising a frame adjustment module for controlling the frame capture of said array of video cameras.

12. The apparatus of claim 11, wherein said frame adjustment module comprises a rolling shutter canceller.

13. The apparatus of claim 11, wherein said frame adjustment module comprises a frame timing adjuster.

14. The apparatus of claim 1, wherein each of said video cameras has a programmable phase delay Δ

.

15. A method of video capture of a three-dimensional region of interest in a scene, said method comprising:

a) positioning an array of video cameras for viewing said three-dimensional region of interest from a corresponding set of viewpoints;

b) staggering the capture of a set of frames by said array of video cameras;

c) combining said set of frames;

d) operating on said set of frames to generate a visual output having a synthetic viewpoint of said three-dimensional area of interest.

16. The method of claim 15, further comprising spatio-temporal interpolation of said visual output.

17. The method of claim 16, wherein said spatio-temporal interpolation comprises obtaining of a multibaseline spatio-temporal optical flow.

18. The method of claim 16, wherein said spatio-temporal interpolation comprises obtaining a spatio-temporal trajectory of said synthetic viewpoint.

19. The method of claim 16, wherein said visual output is a high-speed video of said three-dimensional region of interest and said method further comprises adjusting said staggering such that temporally successive frames in said set of frames are obtained from spatially neighboring video cameras of said array of video cameras.

20. The method of claim 16, wherein said visual output is a space-time structure and motion model of said three-dimensional region of interest.

21. The method of claim 15, wherein said staggering comprises delivering a trigger signal and a clock signal to said video cameras belonging to said array of video cameras.

22. The method of claim 15, wherein said staggering introduces a phase delay Δ

between successive frames of said set of frames.

23. The method of claim 22, wherein said phase delay Δ

is programmed in each of said video cameras belonging to said array of video cameras.