Image processing apparatus and method

• US 20060083421A1
• Filed: 10/14/2004
• Published: 04/20/2006
• Est. Priority Date: 10/14/2004
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Yoshigahara, Takayuki, Tabatabai, Ali, Weiguo, Wu, Vedula, Sundar
• Granted Patent: US 7,330,584 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact

Abstract

An image processing apparatus and an image processing method, applied to e.g. a TV conference system or a TV telephone system, in which an image to be transmitted or received is captured and reconstructed into a virtual view point image which may appear as if it is captured by a virtual camera. The correspondence between pixels may be taken with higher robustness and higher accuracy, while mismatch in such correspondence, otherwise caused due to disparity, may be reduced to as small a value as possible. To this end, the disparity between the images, obtained on imaging an object from different view points by at least two cameras 11a, 12a, are detected and, based on the detected disparity, each of the images is separated into plural image regions. The relation of correspondence is found of the separated each image region, in association with the object, from one pixel position on the same horizontal lines to the next. From the pixel positions and the associated luminance components, for which the relation of correspondence has been found, a virtual view point image, to be generated by a virtual camera, mounted imaginarily, is constructed.

Claims

1. An image processing apparatus comprising:

disparity detection means for detecting the disparity between images obtained on imaging an object by at least two cameras from different view points;

image region separating means for separating said each image into plural image regions, based on the disparity detected by said disparity detection means;

matching means for finding the relation of correspondence of said image regions, separated by said image separating means, as said image regions are coordinated with said object, from one pixel position on the same horizontal line to another; and

image generating means for finding pixel positions, forming a virtual view point image, to be generated by a virtual camera, mounted imaginarily, and luminance components thereof, from pixel positions coordinated with one another by said matching means, and luminance components thereof;

said matching means comparing the luminance components and chroma components, for each pixel position, the relation of correspondence of which is to be found, to calculate the matching score, said matching means finding the relation of correspondence, responsive to the matching score as calculated, to discriminate the disparity between the images of said object.

2. The image processing apparatus according to claim 1 further comprising

image rectification means for finding a projective transformation matrix for projecting each image captured by each camera onto a virtual plane and for bringing the normal line direction of each image into coincidence with the normal line direction of said virtual plane, based on the projective transformation matrix as found.

3. The image processing apparatus according to claim 1 wherein said disparity detection means estimates the distance of said object to each camera to detect said disparity based on the estimated distance.

4. The image processing apparatus according to claim 3 wherein said image region separating means separates each image into plural image regions based on the distance to each camera of said object estimated by said disparity detection means.

5. The image processing apparatus according to claim 3 wherein said image generating means forms said virtual view point image so that the image at a closer distance to said camera will be at a more forward position.

6. The image processing apparatus according to claim 1 further comprising

information generating means for generating the relative position information representing the relative position relationship to each camera of the optical center of said virtual camera;

said image generating means finding pixel positions forming said virtual view point image, and luminance components thereof, based on the relative position information generated by said information generating means.

7. The image processing apparatus according to claim 1 wherein

said matching means in finding the relation of correspondence between said image determines which of the matching score as calculated and the disparity is to be prioritized and wherein

if the matching score is to be prioritized, correspondence is taken between the same pixel positions on said horizontal line and, if the disparity is to be prioritized, a plurality of pixel positions lying on the same horizontal line are coordinated with one pixel position.

8. The image processing apparatus according to claim 1 wherein

said matching means weights luminance components or chroma components to be compared in calculating said matching score.

9. The image processing apparatus according to claim 1 wherein

said matching means calculates said matching score as the relation of correspondence with respect to the luminance and the chroma components in each pixel position lying above and below said horizontal line.

10. An image processing method comprising:

a disparity detection step of detecting the disparity between images obtained on imaging an object by at least two cameras from different view points;

an image region separating step of separating each of said images into plural image regions, based on the disparity detected in said disparity detection step;

a matching step of finding the relation of correspondence of said image regions, separated in said image region separating step, as said image regions are coordinated with said object, from one pixel position on the same horizontal line to another; and

an image generating step of finding pixel positions, forming a virtual view point image, to be generated by a virtual camera, mounted imaginarily, and luminance components thereof, from pixel positions coordinated with one another in said matching step, and luminance components thereof;

said matching step comparing the luminance components and chroma components, for each pixel position, the relation of correspondence of which is to be found, to calculate the matching score, said matching step finding the relation of correspondence, responsive to the matching score as calculated, to discriminate the disparity between the images of said object.