Image processing apparatus, image processing method and transmission medium

• US 6,526,157 B2
• Filed: 07/27/1998
• Issued: 02/25/2003
• Est. Priority Date: 08/01/1997
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Hayashi, Kazuyoshi
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Choobin, Barry
• Application Number:
  US09/123,089
• Publication Number:
  US 20010043720A1
• Time in Patent Office: 1,674 Days
• Field of Search: 382/106, 382/154, 382/107, 382/100, 395/127, 364/561, 73/17.9, 396/189, 356/3, 235/414, 348/139
• US Class Current: 382/106
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

The invention intends to achieve higher accuracy and higher speeds in image processing based on a stereo method. Images from a plurality cameras are applied to respective image input portions and stored in built-in memories. The stored images are then read out in a certain sequence to cancel lens aberrations of the cameras, and output to respective SAD circuits after being compensated for the differences of intensity values due to variations in the characteristics of the cameras. The SAD circuit calculates an absolute value of the difference for each pixel value between a reference image output from a reference camera and two compared images output from another two cameras, and outputs the calculated value to an SSAD circuit. The SSAD circuit performs a block matching process upon outputs from the corresponding SAD circuit, and outputs the obtained results to a second-order portion. The second-order portion performs second-order interpolation upon the data output from the corresponding SSAD circuit and outputs an obtained result. A minimum value selecting portion and a result memory cooperate to select a minimum value from among the data output from the second-order portions.

Claims

1. An apparatus for calculating a distance to a target point based on images from a plurality of image pickup devices, said apparatus comprising:

first input means for receiving a reference image from one of said image pickup devices,

second input means for receiving a first comparison image from another of said image pickup devices,

third input means for receiving a second comparison image from yet another of said image pickup devices,

matching means for calculating absolute values of differences for each pixel value between said reference image and said first and second comparison images to produce difference data,

interpolating means for interpolating said difference data, to produce interpolation difference data, and

detecting means for detecting a minimum value of said interpolation difference data.

2. An apparatus according to claim 1, wherein said interpolating means approximates said interpolation difference data using a second-order curve.

3. An apparatus according to claim 1, wherein:

said second input means receives a plurality of comparison images from two or more of said image pickup devices,

said matching means performs said matching process between said reference image and each of said comparison images to produce said difference data for each of said comparison images,

said interpolation means performs interpolation on said difference data to produce said interpolation difference data, and

said detecting means detects a minimum value of said interpolation difference data based on said comparison images.

4. An apparatus according to claim 1, wherein:

said second input means receives a plurality of comparison images from two or more of said image pickup devices,

said matching means combines two of said comparison images to create a pair of comparison images, and performs said matching process between said reference image and each of said pair of comparison images to produce said difference data,

said interpolating means performs interpolation on said difference data obtained for each pair of said comparison images to produce said interpolation difference data, and

said detecting means detects a minimum value of said interpolation difference data based on said pair of comparison images.

5. A method for calculating a distance to a target point based on images from a plurality of image pickup devices, said method comprising:

receiving a reference image from one of said plurality of image pickup devices,

receiving first and second comparison images from others of said image pickup devices,

performing a matching process to calculate values of differences for each pixel value between said reference image and said first and second comparison images to produce difference data,

performing interpolation on said difference data produced during said matching step to produce interpolation difference data, and

detecting a minimum value of said interpolation difference data.

6. A method according to claim 5, wherein said interpolation step produces said interpolation difference data using a second-order curve.

7. A method according to claim 5, wherein:

said step of receiving said first and second comparison images includes receiving a plurality of comparison images from two or more of said image pickup devices,

said matching step performs said matching process between said reference image and each of said comparison images,

said interpolation step performs interpolation on said difference data obtained for each of said comparison images to produce said interpolation difference data, and

said detecting step detects a minimum value of said interpolation difference data based on said comparison images.

8. A method according to claim 5, wherein:

said step of receiving said first and second comparison images includes receiving a plurality of comparison images from two or more of said image pickup devices,

said matching step combines two of said comparison images to create a pair of comparison images, and performs said matching process between said reference image and each of said pair of comparison images to produce said difference data,

said interpolation step performs interpolation on said difference data obtained for each pair of said comparison images to produce said interpolation difference data, and

said detecting step detects a minimum value of said interpolation difference data based on said pair of comparison images.

9. A transmission medium for transmitting a computer program for use in an image processing apparatus, the image processing apparatus for calculating a distance to a target point based on images from a plurality of image pickup devices, said computer program being capable of causing the image processing apparatus to perform steps comprising:

receiving a reference image from one of said plurality of image pickup devices,

receiving first and second comparison images from others of said image pickup devices,

performing a matching process to calculate absolute values of differences for each pixel value between said reference image and said first and second comparison images to produce difference data,

performing interpolation on said difference data produced during said matching step to produce interpolation difference data, and

detecting a minimum value of said interpolation difference data.

10. An apparatus for calculating a distance to a target based on images from a plurality of image pickup devices, said apparatus comprising:

a reference input circuit operable to receive a reference image from one of said image pickup devices;

a first comparison input circuit operable to receive a first comparison image from another of said image pickup devices;

a second comparison input circuit operable to receive a second comparison image from yet another of said image pickup devices;

at least one sum of differences circuit operable to produce difference data based on pixel information of said reference and first and second comparison images;

at least one interpolation circuit operable to produce interpolation difference data from said difference data; and

a detector circuit operable to detect a minimum value of said interpolation difference data.

11. An apparatus according to claim 10, wherein said interpolation circuit is operable to produce said interpolation difference data by fitting a second order curve to said difference data.

12. An apparatus according to claim 11, wherein said detector circuit is operable to detect said minimum value of said interpolation difference data by detecting a minimum value of said second order curve.

13. An apparatus according to claim 10, wherein said reference input circuit includes:

a memory address generator operable to produce memory storing addresses and memory reading addresses, said memory storing addresses being a plurality of sequential addresses and said memory reading addresses being a plurality of substantially non-sequential addresses based on lens aberrations of one or more of said image pickup devices; and

a memory operable to store said reference image in accordance with said memory storing addresses and to output said reference image in accordance with said memory reading addresses.

14. An apparatus according to claim 13, wherein said reference input circuit further includes a lookup table operable to receive said reference image from said memory and to convert a gradation of said reference image such that compensation for variations in characteristics between said image pickup devices is obtained.

15. An apparatus according to claim 10, wherein said at least one of said first and second comparison input circuits includes:

a memory address generator operable to produce memory storing addresses and memory reading addresses, said memory storing addresses being a plurality of sequential addresses and said memory reading addresses being a plurality of substantially non-sequential addresses based on lens aberrations of one or more of said image pickup devices; and

a memory operable to store said comparison image in accordance with said memory storing addresses and to output said comparison image in accordance with said memory reading addresses.

16. An apparatus according to claim 15, wherein said at least one of said first and second comparison input circuits further includes a lookup table operable to receive said comparison image from said memory and to convert a gradation of said comparison image such that compensation for variations in characteristics between said image pickup devices is obtained.

17. An apparatus according to claim 10, wherein said at least one sum of differences circuit includes:

a first subtraction circuit operable to produce absolute values of differences in pixel information between said reference image and said first comparison image;

a second subtraction circuit operable to produce absolute values of differences in pixel information between said reference image and said second comparison image;

a first selector circuit operable to output one of said absolute values from said first and second subtraction circuits based on a first effective bit signal;

a second selector circuit operable to output one of said absolute values from said first and second subtraction circuits based on a second effective bit signal;

an adder circuit operable to produce an aggregate of said outputs of said first and second selector circuits; and

a third selector circuit operable to output one of said aggregate from said adder circuit and a predetermined value based on said first and second effective bit signals.

18. An apparatus according to claim 17, wherein said first and second effective bit signals provide an indication of whether said pixel information from said first and second comparison images is within respective predetermined image regions.

19. An apparatus according to claim 18, wherein said respective predetermined image regions exist along respective epipolar lines from respective image pickup devices to said target point.

20. An apparatus according to claim 18, wherein:

said first selector circuit is operable to output said absolute value from said first subtraction circuit when said first effective bit signal indicates that said pixel information from said first comparison image is within its predetermined image region and to output said absolute value from said second subtraction circuit when said first effective bit signal indicates that said pixel information from said first comparison image is outside its predetermined image region; and

said second selector circuit is operable to output said absolute value from said second subtraction circuit when said second effective bit signal indicates that said pixel information from said second comparison image is within its predetermined image region and to output said absolute value from said first subtraction circuit when said second effective bit signal indicates that said pixel information from said second comparison image is outside its predetermined image region.

21. An apparatus according to claim 20, wherein:

said third selector circuit is operable to output said aggregate from said adder circuit when at least one of said first and second effective bit signals indicate that said pixel information from at least one of said first and second comparison images is outside its predetermined image region; and

said third selector circuit is operable to output said predetermined value when said first and second effective bit signals indicate that said pixel information from said first and second comparison images are outside their predetermined image regions.