Image processing apparatus and method

• US 7,957,581 B2
• Filed: 11/24/2004
• Issued: 06/07/2011
• Est. Priority Date: 11/27/2003
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Wu, Weiguo
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): Allison, Andrae S
• Application Number:
  US10/996,521
• Publication Number:
  US 20050129325A1
• Time in Patent Office: 2,386 Days
• Field of Search: 382/154, 382162-167, 382/254, 345/419, 348/187, 348/207.99
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact

Abstract

Directions normal to images of an object captured by at least two cameras from different points of view are aligned with each other to form corrected object images, and a correspondence is determined between at least one pixel position on a horizontal line in one of the corrected object images and at least one pixel position on the same horizontal line in another of the corrected object images. The correspondence is determined by determining a similarity in brightness and color components at the pixel positions and a parallax between the corrected object images A robust, accurate image matching can be done by making dynamic matching between all pixels on a scan line in the images captured by the cameras.

Claims

1. An image processor, comprising:

means for providing control information;

an image correcting unit operable to align object images acquired by imaging an object by at least two cameras from different points of view based on said control information to form at least two corrected object images, the corrected object images each including a plurality of horizontal lines and a plurality of pixels on each of the horizontal lines, and the difference in brightness between the corrected object images being minimized through the use of projective transformation matrices corresponding to the object images, the correcting unit being operable to match images from the at least two cameras with a reference image through a projective transformation which is based on at least a chromatic aberration, a distortion, and an optical-axial misalignment of the lens for each of the cameras; and

a matching unit operable to determine a correspondence between at least one pixel position on a horizontal line in one of the corrected object images and at least one pixel position on the same horizontal line in another of the corrected object images by determining a similarity in a brightness component and a color component at the pixel positions and a parallax between the corrected object images,

in which said control information includes information indicative of a positional relationship of each camera to the object, the information indicative of a positional relationship being generated by using the object images to acquire the line of vision between the object and a display associated with the cameras and generating the information indicative of a positional relationship based on the acquired line of vision, and

in which, to determine the correspondence, the matching unit determines whether priority is to be given to the similarity or to the parallax, and the matching unit matches a plurality of pixel positions on the horizontal line in the one of the corrected object images with the same pixel positions on the same horizontal line in the another of the corrected object images when priority is given to the similarity, and the matching unit matches a plurality of pixel positions on the horizontal line in one of the corrected object images with one pixel position on the same horizontal line in the another of the corrected object images when priority is given to the parallax,

the control information including, for each camera, at least one of an indication of imaging direction and an indication of angle of view, and the correction unit using the at least one of the indication of imaging direction and the indication of angle of view for performing a geometric normalization of the corrected object images.

2. The image processor according to claim 1, wherein the matching unit weights either the brightness component or the color component at the pixel positions to determine the similarity.

3. The image processor according to claim 1, wherein the matching unit determines the similarity in reflecting the correlation between the brightness component and the color component in pixel positions above and below the horizontal line.

4. The image processor according to claim 1, wherein the matching unit determines the parallax based on a distance from each of the cameras to the object.

5. The image processor according to claim 1, wherein the image correcting unit aligns a line normal to each of the images acquired by the cameras with the direction of a line normal to a virtual plane set on a virtual basis.

6. The image processor according to claim 5, wherein the image correcting unit determines a projective transformation matrix intended for projection of each of the images acquired by the cameras onto the virtual plane and aligns the direction of the line normal to each of the images with the direction of the line normal to the virtual plane based on the projective transformation matrix.

7. The image processor according to claim 1, further comprising:

an information generating unit operable to generate relative-position information indicative of a positional relationship between an optical center of a virtual camera placed on a virtual basis and each of the cameras; and

an image generating unit operable to determine positions of pixels forming a virtual viewpoint image to be produced by a virtual camera and the brightness components at the pixel positions in the virtual viewpoint image from the pixel positions matched by the matching unit and the brightness component and the color component in the pixel positions matched by the matching unit based on the relative-position information.

8. An image processing method of using a processor to perform steps comprising:

providing control information;

aligning object images acquired by imaging an object by at least two cameras from different points of view based on said control information to form at least two corrected object images, the corrected object images each including a plurality of horizontal lines and a plurality of pixels on each of the horizontal lines, and the difference in brightness between the corrected object images being minimized through the use of projective transformation matrices corresponding to the object images, a correcting unit being used to match images from the at least two cameras with a reference image through a projective transformation which is based on at least a chromatic aberration, a distortion, and an optical-axial misalignment of the lens for each of the cameras; and

using a matching unit for determining a correspondence between at least one pixel position on a horizontal line in one of the corrected object images and at least one pixel position on the same horizontal line in another of the corrected object images by determining a similarity in a brightness component and a color component at the pixel positions and a parallax between the corrected object images,

in which said control information includes information indicative of a positional relationship of each camera to the object, the information indicative of a positional relationship being generated by using the object images to acquire the line of vision between the object and a display associated with the cameras and generating the information indicative of a positional relationship based on the acquired line of vision,

in which the step of determining the correspondence includes determining whether priority is to be given to the similarity or to the parallax, and

in which a plurality of pixel positions on the horizontal line in the one of the corrected object images is matched with the same pixel positions on the same horizontal line in the another of the corrected object images when priority is given to the similarity, and a plurality of pixel positions on the horizontal line in the one of the corrected object images is matched with one pixel position on the same horizontal line in the another of the corrected object images when priority is given to the parallax,

the control information including, for each camera, at least one of an indication of imaging direction and an indication of angle of view, and the method further comprising using the at least one of the indication of imaging direction and the indication of angle of view for performing a geometric normalization of the corrected object images.

9. The method according to claim 8, wherein the step of determining the correspondence includes weighting either the brightness component or the color component at the pixel positions to determine the similarity.

10. The method according to claim 8, wherein the step of determining the correspondence includes determining the similarity in reflecting the correlation between the brightness component and the color component in pixel positions above and below the horizontal line.

11. The method according to claim 8, wherein the step of determining the correspondence includes determining the parallax based on a distance from each of the cameras to the object.

12. The method according to claim 8, wherein the aligning step includes aligning a line normal to each of the images acquired by the cameras with the direction of a line normal to a virtual plane set on a virtual basis.

13. The method according to claim 12, wherein the aligning step includes determining a projective transformation matrix intended for projection of each of the images acquired by the cameras onto the virtual plane and aligning the direction of the line normal to each of the images with the direction of the line normal to the virtual plane based on the projective transformation matrix.

14. The method according claim 8, further comprising:

generating relative-position information indicative of a positional relationship between an optical center of a virtual camera placed on a virtual basis and each of the cameras; and

determining positions of pixels forming a virtual viewpoint image to be produced by a virtual camera and the brightness components at the pixel positions in the virtual viewpoint image from the pixel positions matched in the step of determining the correspondence and the brightness component and the color component in the pixel positions matched in the step of determining the correspondence based on the relative-position information.