Image processing apparatus and method

• US 20050129325A1
• Filed: 11/24/2004
• Published: 06/16/2005
• Est. Priority Date: 11/27/2003
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Wu, Weiguo
• Granted Patent: US 7,957,581 B2
• Time in Patent Office: Days
• US Class Current: 382/254
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact

Abstract

Directions normal to images of an object captured by at least two cameras from different points of view are aligned with each other to form corrected object images, and a correspondence is determined between at least one pixel position on a horizontal line in one of the corrected object images and at least one pixel position on the same horizontal line in another of the corrected object images. The correspondence is determined by determining a similarity in brightness and color components at the pixel positions and a parallax between the corrected object images A robust, accurate image matching can be done by making dynamic matching between all pixels on a scan line in the images captured by the cameras.

Claims

1. An image processor, comprising:

an image correcting unit operable to align object images acquired by imaging an object by at least two cameras from different points of view to thereby form at least two corrected object images, the corrected object images each including a plurality of horizontal lines and a plurality of pixels on each of the horizontal lines; and

a matching unit operable to determine a correspondence between at least one pixel position on a horizontal line in one of the corrected object images and at least one pixel position on the same horizontal line in another of the corrected object images by determining a similarity in a brightness component and a color component at the pixel positions and a parallax between the corrected object images.

2. The image processor according to claim 1, wherein, to determine the correspondence, the matching unit determines whether priority is to be given to the similarity or to the parallax, and

if priority is given to the similarity, the matching unit matches a plurality of pixel positions on the horizontal line in the one of the corrected object images with the same pixel positions on the same horizontal lines in the another of the corrected object images, and if priority is given to the parallax, the matching unit matches a plurality of pixel positions on the horizontal line in one of the corrected object images with one pixel position on the same horizontal line in the another of the corrected object images.

3. The image processor according to claim 1, wherein the matching unit weights either the brightness component or the color component at the pixel positions to determine the similarity.

4. The image processor according to claim 1, wherein the matching unit determines the similarity in reflecting the correlation between the brightness component and the color component in pixel positions above and below the horizontal line.

5. The image processor according to claim 1, wherein the matching unit determines the parallax based on a distance from each of the cameras to the object.

6. The image processor according to claim 1, wherein the image correcting unit aligns a line normal to each of the images acquired by the cameras with the direction of a line normal to a virtual plane set on a virtual basis.

7. The image processor according to claim 6, wherein the image correcting unit determines a projective transformation matrix intended for projection of each of the images acquired by the cameras onto the virtual plane and aligns the direction of the line normal to each of the images with the direction of the line normal to the virtual plane based on the projective transformation matrix.

8. The image processor according to claim 1, further comprising:

an information generating unit operable to generate relative-position information indicative of a positional relationship between an optical center of a virtual camera placed on a virtual basis and each of the cameras; and

an image generating unit operable to determine positions of pixels forming a virtual viewpoint image to be produced by a virtual camera and the brightness components at the pixel positions in the virtual viewpoint image from the pixel positions matched by the matching unit and the brightness component and the color component in the pixel positions matched by the matching unit based on the relative-position information.

9. An image processing method, comprising:

aligning object images acquired by imaging an object by at least two cameras from different points of view to thereby form at least two corrected object images, the corrected object images each including a plurality of horizontal lines and a plurality of pixels on each of the horizontal lines; and

determining a correspondence between at least one pixel position on a horizontal line in one of the corrected object images and at least one pixel position on the same horizontal line in another of the corrected object images by determining a similarity in a brightness component and a color component at the pixel positions and a parallax between the corrected object images.

10. The method according to claim 9, wherein the step of determining the correspondence includes determining whether priority is to be given to the similarity or to the parallax, and

if priority is given to the similarity, a plurality of pixel positions on the horizontal line in the one of the corrected object images is matched with the same pixel positions on the same horizontal line in the another of the corrected object images, and if priority is given to the parallax, a plurality of pixel positions on the horizontal line in the one of the corrected object images is matched with one pixel position on the same horizontal line in the another of the corrected object images.

11. The method according to claim 9, wherein the step of determining the correspondence includes weighting either the brightness component or the color component at the pixel positions to determine the similarity.

12. The method according to claim 9, wherein the step of determining the correspondence includes determining the similarity in reflecting the correlation between the brightness component and the color component in pixel positions above and below the horizontal line.

13. The method according to claim 9, wherein the step of determining the correspondence includes determining the parallax based on a distance from each of the cameras to the object.

14. The method according to claim 9, wherein the aligning step includes aligning a line normal to each of the images acquired by the cameras with the direction of a line normal to a virtual plane set on a virtual basis.

15. The method according to claim 14, wherein the aligning step includes determining a projective transformation matrix intended for projection of each of the images acquired by the cameras onto the virtual plane and aligning the direction of the line normal to each of the images with the direction of the line normal to the virtual plane based on the projective transformation matrix.

16. The method according to claim 9, further comprising:

generating relative-position information indicative of a positional relationship between an optical center a virtual camera placed on a virtual basis and each of the cameras; and

determining positions of pixels forming a virtual viewpoint image to be produced by a virtual camera and the brightness components at the pixel positions in the virtual viewpoint image from the pixel positions matched in the step of determining the correspondence and the brightness component and the color component in the pixel positions matched in the step of determining the correspondence based on the relative-position information.