Automatic video system using multiple cameras

• US 7,015,954 B1
• Filed: 08/09/1999
• Issued: 03/21/2006
• Est. Priority Date: 08/09/1999
• Status: Expired due to Term

• Current Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Ahmad, Subutai, Boreczky, John, Foote, Jonathan
• Primary Examiner(s): Tran, Thai
• Assistant Examiner(s): NGUYEN, LUONG TRUNG
• Application Number:
  US09/370,406
• Time in Patent Office: 2,416 Days
• Field of Search: 348/36, 348/39, 348/42, 348/47, 348/48, 348/211.11, 348/211.12, 348/211.99, 348/218.1, 348/37, 348/38, 348/159, 382/254, 382/276
• US Class Current: 348/218.1
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

A camera array captures plural component images which are combined into a single scene from which “panning” and “zooming” within the scene are performed. In one embodiment, each camera of the array is a fixed digital camera. The images from each camera are warped and blended such that the combined image is seamless with respect to each of the component images. Warping of the digital images is performed via pre-calculated non-dynamic equations that are calculated based on a registration of the camera array. The process of registering each camera in the arrays is performed either manually, by selecting corresponding points or sets of points in two or more images, or automatically, by presenting a source object (laser light source, for example) into a scene being captured by the camera array and registering positions of the source object as it appears in each of the images. The warping equations are calculated based on the registration data and each scene captured by the camera array is warped and combined using the same equations determined therefrom. A scene captured by the camera array is zoomed, or selectively steered to an area of interest. This zooming- or steering, being done in the digital domain is performed nearly instantaneously when compared to cameras with mechanical zoom and steering functions.

Claims

1. A method, comprising:

providing a camera array, the camera array including a plurality of cameras;

providing a set of camera offset values for the camera array;

synchronously capturing a set of images from the camera array;

selecting pixels from at least one image of said set of images;

applying a bilinear transformation to the selected pixels to transform the selected pixels from a coordinate system of the at least one image to a common coordinate system of a composite image;

wherein applying a bilinear transformation includes using the set of camera offset values.

2. The method according to claim 1, wherein selecting pixels includes identifying contiguous patches in said set of images.

3. The method according to claim 2, wherein identifying contiguous patches, includes identifying pixels from the contiguous patches having registration points in common with said composite image.

4. The method according to claim 2, further comprising:

combining pixels from the contiguous patches having substantially similar registration points into a common location of said composite image.

5. The method according to claim 4, wherein combining pixels from the contiguous patches includes cross-fading pixels having substantially similar registration points having a substantially similar registration points into the common location of said composite image.

6. The method according to claim 4, wherein combining pixels from the contiguous patches includes at least one of cross-fading, blurring, averaging, and other image effects to seamlessly combine said pixels from different patches having substantially similar registration points into the common location of said composite image.

7. The method according to claim 2, wherein performing a transformation includes applying a predetermined bilinear transformation matrix to each patch, said predetermined bilinear transformation matrix representing an amount of warping required for each patch to transform each patch into said common coordinate system.

8. The method according to claim 1, further comprising

repeating said applying a transformative equation for each set of images synchronously captured by said camera array, each set of images representing one frame in a video stream of said scene.

9. The method according to claim 1, further comprising

selecting an area of interest from the composite image;

and

outputting the selected area of interest to a user.

10. The method according to claim 9, wherein said outputting comprises displaying said area of interest from the combined warped images.

11. The method according to claim 9, wherein said selecting comprises:

directing said area of interest to a predetermined area surrounding at least one of motion detected in said scene, audio sources detected within said scene, and proximity of objects detected in said scene.

12. A method, comprising:

synchronously capturing a set of images from a camera array;

selecting pixels from at least one image from the set of images;

identifying contiguous patches from the selected pixels;

transforming the selected pixels from a coordinate system of the at least one image to a common coordinate system;

determining overlap between the contiguous patches, wherein overlapping contiguous patches include pixels having substantially similar registration points;

combining overlapping contiguous patches into a common location of said a composite image;

wherein combining overlapping contiguous patches includes cross-fading pixels having substantially similar registration points into the common location of said composite image;

wherein cross-fading includes;

varying a parameter of the pixels having substantially similar registration points from a first patch from a minimum value at a first boundary of said first patch to a maximum value at an opposite boundary of said first patch;

varying said parameter of the pixels having substantially similar registration points from a second patch from said maximum value at a boundary of said second patch corresponding to said first boundary to said minimum value at a boundary of said second patch corresponding to said opposite boundary;

summing corresponding pixel values of said first patch and said second patch; and

placing the summed values in corresponding locations of said common coordinate system.

13. The method according to claim 12, wherein said parameter is at least one of brightness, contrast, and intensity.

14. A camera array, comprising:

a set of cameras mounted in an array;

a set of camera offset values for the set of cameras;

an image combining mechanism configured to combine at least two of images captured from said set of cameras into a composite image, the image combining mechanism including;

a warping device configured to warp patches of each image into a common coordinate system of said composite image by applying a bilinear transformation;

wherein the warping device applies the set of camera offset values to warp patches;

a view selection device configured to select a view from the composite image; and

an output mechanism configured to display the selected view.

15. The camera array according to claim 14, wherein said image combining mechanism includes,

a fading device configured to fade and combine patches having a same location in said composite image.

16. The camera array according to claim 15, wherein:

said array of cameras are immovably mounted on a firm fixed base; and

said warping device applies a pre-determined transformation to each of said patches.

17. The camera array according to claim 15, further comprising a registration mechanism configured to register each of said camera arrays by finding registration points in common with views of each camera and said composite image.

18. The camera array according to claim 14, wherein said view selection device includes:

at least one of,

a video motion detector configured to detect motion in the composite image,

a sound detection device configured to determine a location of sound originating within said composite image, and

a stereoscopic ranging mechanism configured to utilize at least two images from separate cameras of said camera array to determine a range of objects in said composite image; and

a detection mechanism configured to automatically detect any of a face, shape, color, and motion of a subject for inclusion in said selected frame based on at least one of ranging data from said stereoscopic ranging mechanism, location data from said sound detection device, and motion detected by said video motion detector.

19. A camera array comprising:

a plurality of cameras;

an image combining mechanism configured to combine at least two images captured from said plurality of cameras into a composite image, the image combining mechanism including;

a registration mechanism configured to register said plurality of cameras by finding registration points in common with views of the plurality of cameras and said composite image, the registration mechanism including;

a registration point source directable to locations within said views of the plurality of cameras, and

a detection mechanism configured to detect said registration point source and register the detected registration point source in said views of the plurality of cameras with respect to a coordinate system of said composite image.

20. The camera array according to claim 19, wherein said registration point source is at least one of a light source directed into view of said camera array and a grid placed in view of said camera array.

21. A method of registering a camera array, comprising:

placing at least one registration point in a field of view of at least two cameras of said camera array;

identifying a location of each registration point in a field of view of each camera of said array;

calculating a warped coordinate system for placing pixels of said cameras; and

maintaining a table identifying pixels of said cameras, an amount of fade, and a corresponding location in the warped coordinate system such that images can be combined in relation to said registration points.

22. The method according to claim 21, wherein said step placing comprises the steps of:

sweeping an beam light source across a field of view of at least two cameras of said camera array.

23. The method according to claim 21, wherein said step of placing comprises the step of:

placing a grid of known location in a field of view in at least two cameras of said array.

24. A method, comprising:

providing a camera array, the camera array including a plurality of cameras;

providing a set of camera offset values for the camera array;

synchronously capturing a set of images from the camera array;

selecting pixels from at least one image of said set of images; and

applying a transformative equation to the selected pixels to transform the selected pixels from a coordinate system of the at least one image to a common coordinate system of a composite image;

wherein said transformative equation is at least one of a bilinear transformation, a radial transformation, and an affine transformation;

wherein applying a transformative equation includes using the set of camera offset.