Method and system for compensating for parallax in multiple camera systems

• US 20020122113A1
• Filed: 11/20/2001
• Published: 09/05/2002
• Est. Priority Date: 08/09/1999
• Status: Active Grant

• Current Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Foote, Jonathan T.
• Granted Patent: US 7,277,118 B2
• Time in Patent Office: Days
• US Class Current: 348/48
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/006 Geometric correction correction of chromatic aberrations adapted to be used in an image pickup device containing an electronic image sensor H04N23/10; detecting, correcting, reducing or removing artefacts resulting only from the lens unit, e.g. flare, shading, vignetting or "cos4" H04N25/61
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

A camera array captures plural component images which are combined into a single scene. In one embodiment, each camera of the array is a fixed digital camera. The images from each camera are warped to a common coordinate system and the disparity between overlapping images is reduced using disparity estimation techniques.

Claims

1. A method for combining at least a portion of a plurality of images, comprising the steps of:

obtaining a first image and a second image, wherein at least a portion of said first image and said second image include a common field of view;

adjusting at least a portion of said first image to reduce an image disparity between said common field of view of said first image and said second image; and

,

combining at least a portion of said first image and at least a portion of said second image subsequent to said step of adjusting.

2. The method of claim 1, wherein said step of adjusting at least a portion of said first image includes shifting said first image a first distance with respect to said second image.

3. The method of claim 1 wherein said step of adjusting includes the steps of:

determining a plurality of disparity differences between at least a portion of said first image and at least a portion of said second image, each disparity difference corresponding to a distance of adjustment of said first image with respect to said second image; and

selecting a distance corresponding to a lowest disparity difference in said plurality of disparity differences as a first distance; and

,

shifting said first image said first distance with respect to said second image.

4. The method of claim 1, further including the step of:

adjusting said second image to reduce said image disparity between said common field of view of said first image and said second image.

5. The method of claim 4, wherein said step of adjusting said second image includes shifting said second image a second distance.

6. The method of claim 5 wherein said second distance is a distance where the image disparity is reduced.

7. The method of claim 1 further including the step of:

warping said first image and said second image into a common coordinate system of a composite image subsequent to said step of obtaining.

8. The method of claim 1 further including the step of:

cross-fading said common field of view of said first image and said second image, subsequent to said step of adjusting.

9. A method for combining frames of video from a plurality of cameras arranged in a camera array, comprising the steps of:

obtaining a first frame from a first camera, and a second frame from a second camera, wherein at least a portion of said first frame and said second frame include a common field of view;

stretching at least a portion of said first frame to reduce an image disparity between said common field of view of said first frame and said second frame; and

,

combining said common field of view of said first frame and said second frame subsequent to said step of stretching.

10. The method of claim 9, wherein said step of stretching at least a portion of said first frame includes stretching at least a portion of said first frame a first distance.

11. The method of claim 10 wherein said first distance is a distance where disparity is reduced.

12. The method of claim 9, further including the step of:

stretching said second frame to reduce said disparity between said common field of view of said first frame and said second frame.

13. The method of claim 12, wherein said step of stretching said second frame includes stretching said second frame a second distance.

14. The method of claim 9 wherein said step of stretching includes the steps of:

determining a plurality of disparity differences between at least a portion of said first frame and at least a portion of said second frame, each disparity difference corresponding to a distance of stretching of said first frame and said second frame; and

selecting a distance corresponding to a lowest disparity difference in said plurality of disparity differences as a stretching distance; and

,

stretching at least a portion of said first frame and said second frame such that a total stretching of said first and second frames approximately equals said stretching distance.

15. The method of claim 9 further including the step of:

warping said first frame and said second frame into a common coordinate system of a composite frame subsequent to said step of obtaining.

16. The method of claim 9 further including the step of:

cross-fading said common field of view of said first frame and said second frame, subsequent to said step of stretching.

18. The method of claim 17 further including the step of:

warping said first image, said second image, and said third image into a common coordinate system of a composite image subsequent to said step of obtaining.

19. The method of claim 17 further including the step of:

cross-fading said adjusted first portion of said first image with at least a portion of said second image, and

cross-fading said adjusted second portion of said first image with at least a portion of said third image.

20. The method of claim 17 wherein said step of adjusting said first portion of said first image includes stretching said first portion of said first image a first distance;

and

wherein said step of adjusting said second portion of said first image includes stretching said second portion of said first image a second distance.

21. The method of claim 20 further including the steps of:

stretching at least a portion of said second image a third distance; and

stretching at least a portion of said third image a fourth distance.

23. The apparatus of claim 21, wherein said image adjustor adjusts at least a portion of said first image by shifting said first image a first distance.

24. The apparatus of claim 21, wherein said image adjustor adjusts at least a portion of said first image by stretching said first image a first distance.

17. A method for combining a plurality of images captured from a plurality of cameras of a camera array into a panoramic image, comprising the steps of:

adjusting a first portion of a first image to reduce image disparity between said first portion of said first image and a second image;

adjusting a second portion of said first image to reduce image disparity between said second portion of said first image and a third image; and

,

combining said first image, said second image, and said third image into a panoramic image.

22. An apparatus for producing a panoramic video, comprising:

a camera array including a plurality of cameras;

an image obtaining device, wherein said image obtaining device obtains a first image from a first camera in said camera array and a second image from a second camera in said camera array, wherein said first image and said second image include a common field of view;

an image adjustor, wherein said first image adjustor adjusts at least a portion of said first image to reduce an image disparity between said common field of view of said first image and said second image; and

,

an image combiner, wherein said image combiner combines at least a portion of said first image and at least a portion of said second image.