Image processing apparatus, image processing method and transmission medium

• US 6,625,301 B2
• Filed: 12/16/2002
• Issued: 09/23/2003
• Est. Priority Date: 08/01/1997
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Hayashi, Kazuyoshi
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): Choobin, Barry
• Application Number:
  US10/320,114
• Publication Number:
  US 20030086589A1
• Time in Patent Office: 281 Days
• Field of Search: 382/106, 382/278, 382/103, 382/154, 382/254, 382/255, 382/275, 382/192, 382/107, 356/3, 356/608, 356/337, 348/229.1, 348/625, 348/148, 396/104, 702/142, 345/173, 250/559.05
• US Class Current: 382/106
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

The invention intends to achieve higher accuracy and higher speeds in image processing based on a stereo method. Images from a plurality cameras are applied to respective image input portions and stored in built-in memories. The stored images are then read out in a certain sequence to cancel lens aberrations of the cameras, and output to respective SAD circuits after being compensated for the differences of intensity values due to variations in the characteristics of the cameras. The SAD circuit calculates an absolute value of the difference for each pixel value between a reference image output from a reference camera and two compared images output from another two cameras, and outputs the calculated value to an SSAD circuit. The SSAD circuit performs a block matching process upon outputs from the corresponding SAD circuit, and outputs the obtained results to a second-order portion. The second-order portion performs second-order interpolation upon the data output from the corresponding SSAD circuit and outputs an obtained result. A minimum value selecting portion and a result memory cooperate to select a minimum value from among the data output from the second-order portions.

Claims

1. An apparatus for calculating a distance to a target point based on images from a plurality of image pickup devices, said apparatus comprising:

first storage means for storing pixel data of respective images from each of said plurality of image pickup devices responding to respective scene patterns,

second storage means for storing sequence data for reading out said pixel data,

reading means for reading out said pixel data stored in said first storage means in accordance with said sequence data stored in said second storage means, such that lens aberrations of said image pickup devices are cancelled, and

means for modifying gradations of said respective pixel data read out by said reading means to provide compensation for variations in characteristics between said image pickup devices such that respective modified pixel data are substantially identical when said respective scene patterns are substantially identical.

2. An apparatus according to claim 1, wherein said sequence data stored in said second storage means are predetermined based on said lens aberrations of each of said image pickup devices.

3. An apparatus according to claim 1, wherein said converting means is operable such that gradations of said plurality of image pickup devices are equal to one another.

4. A method for calculating a distance to a target point based on images from a plurality of image pickup devices, said method comprising:

storing pixel data of respective images from each of said plurality of image pickup devices responding to respective scene patterns,

reading out pixel data of said image stored in said storing step in accordance with a stored reading sequence such that lens aberrations of said image pickup devices are cancelled, and

modifying gradations of said respective pixel data read out in said reading step to provide compensation for variations in characteristics between said image pickup devices such that respective modified pixel data are substantially identical when said respective scene patterns are substantially identical.

5. A method according to claim 4, wherein said reading sequence is predetermined based on said lens aberrations of each of said image pickup devices.

6. A method according to claim 4, wherein said converting step is operable such that gradations of said plurality of image pickup devices are equal to one another.

7. A transmission medium for transmitting a computer program for use in an apparatus, the apparatus for calculating a distance to a target point based on images from a plurality of image pickup devices, said computer program being capable of causing the apparatus to perform steps comprising:

storing pixel data of respective images from each of said plurality of image pickup devices responding to respective scene patterns,

reading out pixel data of said image stored in said storing step in accordance with a stored reading sequence such that lens aberrations of said image pickup devices are cancelled, and

modifying gradations of said respective pixel data read out in said reading step to provide compensation for variations in characteristics between said image pickup devices such that respective modified pixel data are substantially identical when said respective scene patterns are substantially identical.

8. An apparatus for calculating a distance to a target point based on images from a plurality of image pickup devices, said apparatus comprising:

first input means for receiving a reference image from one of said plurality of image pickup devices,

second input means for receiving a comparison image from another of said image pickup devices,

searching means for searching an area of said comparison image that corresponds to a given area of said reference image using a matching-process,

detecting means for detecting if said area of said comparison image is outside a valid region of the comparison image, and

control means for controlling said matching process performed by said searching means in accordance with a detection result from said detecting means.

9. An apparatus according to claim 8, further comprising storage means for storing data related to a correlation between said reference image and said comparison image, wherein said detecting means detects if said area is outside said valid region of said comparison, image by referring to an effective bit of said data stored in said storage means.

10. An apparatus according to claim 8, wherein said searching means performs said matching process between said reference image and each of first and second comparison images having respective valid regions, and said control means controls said searching means such that:

when said detecting means detects that said search area is within both valid regions of said first and second comparison images, said control means controls said searching means to execute said matching process for both said comparison images,

when said detecting means detects that said search area is outside one of said valid regions of said first and second comparison images, said control means controls said searching means to execute said matching process for only said comparison image for which said search area is within said valid region, and

when said detecting means detects that said search area is outside both valid regions of said first and second comparison images, said control means controls said searching means to output a predetermined value as a result of said matching process.

11. An apparatus according to claim 8, wherein said matching process calculates absolute values of differences in respective pixel values between said reference image and said comparison image, and determines a minimum value of said calculated absolute values.

12. A method for calculating a distance to a target point based on images from a plurality of image pickup devices, said method comprising:

receiving a reference image from one of said plurality of image pickup devices,

receiving a comparison image from another of said image pickup devices,

searching an area of said comparison image that corresponds to a given area of said reference image using a matching process,

detecting if said area of said comparison image is outside a valid region of said comparison image, and

a control step of controlling said matching process performed in said searching step in accordance with a detection result of said detecting step.

13. A method according to claim 12, wherein said detecting step detects if said search area is outside said valid region of said comparison image by referring to an effective bit of said data that is stored and related to a correlation between said reference image and said comparison image.

14. An image processing method according to claim 12, wherein said searching step performs said matching process between said reference image and each of first and second comparison images having respective valid regions, and said control step controls said searching step such that:

when said detecting step detects that said search area is within both valid regions of said first and second comparison images, said control step controls said searching step to execute said matching process for both said comparison images,

when said detecting step detects that said search area is outside one of said valid regions of said first and second comparison images, said control step controls said searching step to execute said matching process for only said comparison image for which said search area is within said valid region, and

when said detecting step detects that said search area is outside both valid regions of said first and second comparison images, said control step controls said searching step to output a predetermined value as a result of said matching process.

15. An image processing method according to claim 12, wherein said matching process calculates absolute values of differences in respective pixel values between said reference image and said comparison image, and determines a minimum value of said calculated absolute values.

16. A transmission medium for transmitting a computer program for use in an apparatus, the apparatus for calculating a distance to a target point based on images from a plurality of image pickup devices, said computer program being capable of causing the apparatus to perform steps comprising:

receiving a reference image from one of said plurality of image pickup devices,

receiving a comparison image from another of said image pickup devices,

searching an area of said comparison image that corresponds to a given area of said reference image using a matching process,

detecting if said area of said comparison image is outside a valid region of said comparison image, and a control step of controlling said matching process performed in said searching step in accordance with a detection result of said detecting step.