Method and system for compensating for parallax in multiple camera systems

• US 7,710,463 B2
• Filed: 02/02/2006
• Issued: 05/04/2010
• Est. Priority Date: 08/09/1999
• Status: Expired due to Fees

• Current Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Foote, Jonathan T.
• Primary Examiner(s): NGUYEN, LUONG TRUNG
• Application Number:
  US11/345,708
• Publication Number:
  US 20060125921A1
• Time in Patent Office: 1,552 Days
• Field of Search: 348/218.1, 348/47, 348/48
• US Class Current: 348/218.1
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

A camera array captures plural component images which are combined into a single scene. In one embodiment, each camera of the array is a fixed digital camera. The images from each camera are warped to a common coordinate system and the disparity between overlapping images is reduced using disparity estimation techniques.

Claims

1. A method for combining frames of video from a plurality of cameras arranged in a camera array, comprising the steps of:

obtaining a first frame from a first camera, and a second frame from a second camera, wherein at least a portion of said first frame and said second frame include a common field of view;

stretching said at least a portion of said first frame to reduce an image disparity between said common field of view of said first frame and said second frame;

wherein the step of stretching further comprises the steps of;

determining a plurality of disparity differences between the at least a portion of said first frame and at least a portion of said second frame, each disparity difference corresponding to a distance of stretching of said first frame and said second frame;

selecting a distance corresponding to a lowest disparity difference in said plurality of disparity differences as a stretching distance;

stretching the at least a portion of said first frame and said second frame such that a total stretching of said first and second frames approximately equals said stretching distance; and

combining said common field of view of said first frame and said second frame subsequent to said step of stretching; and

wherein the method steps are done by at least one processor.

2. The method of claim 1, wherein said step of stretching the at least a portion of said first frame includes stretching the at least a portion of said first frame a first distance, wherein said first distance is a distance where the image disparity is reduced.

3. The method of claim 1, further including the step of:

stretching said second frame to reduce said image disparity between said common field of view of said first frame and said second frame.

4. The method of claim 3, wherein said step of stretching said second frame includes stretching said second frame a second distance.

5. The method of claim 1 further including the step of:

warping said first frame and said second frame into a common coordinate system of a composite frame subsequent to said step of obtaining.

6. A system for the combination of frames of video from a plurality of cameras arranged in a camera array, comprising:

a first frame from a first camera;

a second frame from a second camera, wherein at least a portion of said first frame and said second frame include a common field of view;

the common field of view, such that the at least a portion of said first frame is stretched to reduce an image disparity between said common field of view of said first frame and said second frame;

a plurality of disparity differences determined between the at least a portion of said first frame and the at least a portion of said second frame, each disparity difference corresponding to a distance of stretching of said first frame and said second frame;

a stretching distance corresponding to a lowest disparity difference in said plurality of disparity differences;

a stretching of the at least a portion of said first frame and a stretching of the at least a portion of said second frame such that a total stretching of said first and second frames approximately equals said stretching distance; and

a combination of said common field of view of said first frame and said second frame subsequent to said portion being stretched.

7. The system of claim 6, wherein said at least a portion of said first frame is stretched a first distance, wherein said first distance is a distance where the image disparity is reduced.

8. The system of claim 6, wherein the at least a portion of said second frame is stretched to reduce said image disparity between said common field of view of said first frame and said second frame.

9. The system of claim 8, wherein said at least a portion of said second frame is stretched a second distance.

10. The system of claim 6 further comprising:

a warping of said first frame and said second frame are warped into a common coordinate system of a composite frame.

11. A machine-readable storage medium having executable instructions stored thereon that performs the method of combining frames of video from a plurality of cameras arranged in a camera array, comprising the steps of:

obtaining a first frame from a first camera, and a second frame from a second camera, wherein at least a portion of said first frame and said second frame include a common field of view;

stretching the at least a portion of said first frame to reduce an image disparity between said common field of view of said first frame and said second frame;

wherein the step of stretching further comprises the steps of;

determining a plurality of disparity differences between the at least a portion of said first frame and the at least a portion of said second frame, each disparity difference corresponding to a distance of stretching of said first frame and said second frame;

selecting a distance corresponding to a lowest disparity difference in said plurality of disparity differences as a stretching distance;

stretching the at least a portion of said first frame and said second frame such that a total stretching of said first and second frames approximately equals said stretching distance; and

combining said common field of view of said first frame and said second frame subsequent to said step of stretching; and

wherein the method steps are done by at least one processor.