Distance estimation apparatus, distance estimation method, storage medium storing program, integrated circuit, and camera

• US 8,159,598 B2
• Filed: 08/18/2009
• Issued: 04/17/2012
• Est. Priority Date: 08/19/2008
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Watanabe, Tatsumi, Kuwahara, Yasuhiro, Ito, Takeshi, Toji, Bumpei, Sato, Daisuke, Kiuchi, Shinya, Owaki, Yoshiaki
• Primary Examiner(s): DINH, KHANH Q
• Application Number:
  US12/542,886
• Publication Number:
  US 20100046802A1
• Time in Patent Office: 973 Days
• Field of Search: 348/270, 348/345, 348/349, 348/E5.042
• US Class Current: 348/345
• CPC Class Codes: G01S 17/36 with phase comparison between the received signal and the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/564 from contours
• CPC Class Codes: H04N 2213/006 Pseudo-stereoscopic systems, i.e. systems wherein a stereoscopic effect is obtained without sending different images to the viewer's eyes

Abstract

Attempts to achieve a higher resolution and a higher frame rate of a distance image when a distance to an object within a target space is estimated using the TOF method would cause CCD saturation due to shot noise or environment light, and lower distance precision. A distance estimation apparatus illuminates an object with illumination light for distance estimation emitted from a light source that can emit light (electromagnetic wave) having a predetermined illumination frequency, receives reflected light of the illumination light, obtains information about the distance from the apparatus to the object, generates distance image data based on the distance information, extracts edge information of a color image formed using a visible light component obtained in synchronization with the reflected light, and corrects distance information of a target part of the distance image using distance information of a neighboring part of the target part based on the edge information.

Claims

1. A distance estimation apparatus that illuminates an object with light having a modulated light intensity and estimates a distance to an object using reflected light from the object, the apparatus comprising:

a light source operable to emit light whose light intensity can be modulated;

a light source control unit operable to control the light source;

a light receiving optical system operable to focus light from the object;

a color separation unit operable to separate light received by the light receiving optical system into a first light component and a second light component based on a frequency of the light;

a light receiving unit operable to convert the second light component separated by the color separation unit to obtain charge;

a charge accumulation unit operable to accumulate the charge obtained by the light receiving unit and obtain a charge signal;

a signal processing unit operable to calculate distance information based on the charge signal;

a distance image generation unit operable to generate a distance image having a first number of pixels based on the distance information;

an imaging unit operable to convert the second light component separated by the color separation unit to an image generation charge signal;

a high resolution image generation unit operable to generate a high resolution image having pixels more than the first number of pixels based on the image generation charge signal obtained by the imaging unit;

an edge information extraction unit operable to extract edge information from the high resolution image generated by the high resolution image generation unit; and

a distance image correction unit operable to correct distance information of a target pixel of the distance image based on a neighboring pixel of the target pixel in the distance image and based on the edge information extracted by the edge information extraction unit.

2. The distance estimation apparatus according to claim 1, wherein when an image area of the high resolution image corresponding to the target pixel of the distance image includes edge information, the distance image correction unit corrects the distance information of the target pixel of the distance image by specifying position information Pos(x, y) of the edge information in the distance image, weighting distance information of the neighboring pixel based on a positional relationship between the position information Pos(x, y) and the neighboring pixel, calculating distance information of the position information Pos(x, y) using the weighted distance information of the neighboring pixel, and replacing the calculated distance information of the position information Pos(x, y) with the distance information of the target pixel.

3. The distance estimation apparatus according to claim 1, wherein when an image area of the high resolution image corresponding to the target pixel of the distance image includes edge information, the distance image correction unit corrects the distance information of the target pixel of the distance image by dividing the target pixel into pixel parts based on the edge information in the distance image, and calculating distance information of each pixel part using distance information of a neighboring pixel of the target pixel that is near the pixel part in the distance image.

4. A distance estimation apparatus that illuminates an object with light having a modulated light intensity and estimates a distance to an object using reflected light from the object, the apparatus comprising:

a light source operable to emit light whose light intensity can be modulated;

a light source control unit operable to control the light source;

a light receiving optical system operable to focus light from the object;

a color separation unit operable to separate light received by the light receiving optical system into a first light component and a second light component based on a frequency of the light;

a light receiving unit operable to convert the second light component separated by the color separation unit to obtain charge;

a charge accumulation unit operable to accumulate the charge obtained by the light receiving unit and obtain a charge signal;

a signal processing unit operable to calculate distance information based on the charge signal;

a distance image generation unit operable to generate a distance image having a first number of pixels based on the distance information;

a distance image storage unit operable to store the distance image;

an imaging unit operable to convert the second light component separated by the color separation unit to an image generation charge signal;

a high resolution image generation unit operable to generate a high resolution image having pixels more than the first number of pixels based on the image generation charge signal obtained by the imaging unit;

a high resolution image storage unit operable to store the high resolution image;

a motion vector detection unit operable to detect a motion vector by comparing a high resolution image Img_color(t) that is obtained at a predetermined unit timing t by the high resolution image generation unit and a high resolution image Img_color(t−

α

) that is obtained at a timing (t−

α

) preceding the predetermined unit timing t and stored in the high resolution image storage unit; and

a distance image correction unit operable to correct distance information of a target pixel of the distance image based on a distance image Img_distance(t) that is obtained at the predetermined unit timing t by the distance image generation unit, a distance image Img_distance(t−

α

) that is obtained at the timing (t−

α

) preceding the predetermined unit timing t and stored in the distance image storage unit, and a motion vector that is detected by the motion vector detection unit.

5. A distance estimation apparatus that illuminates an object with light having a modulated light intensity and estimates a distance to an object using reflected light from the object, the apparatus comprising:

a light source operable to emit light whose light intensity can be modulated;

a light source control unit operable to control the light source;

a light receiving optical system operable to focus light from the object;

a light receiving unit operable to convert the light received by the light receiving optical system to obtain charge;

a charge accumulation unit operable to accumulate the charge obtained by the light receiving unit and obtain a charge signal;

a signal processing unit operable to calculate distance information based on the charge signal;

a distance image generation unit operable to generate a distance image based on the distance information;

a distance reliability calculation unit operable to calculate a reliability of a pixel of the distance image based on at least one of the distance information and the charge signal; and

a distance image correction unit operable to correct distance information of the pixel of the distance image based on the reliability calculated by the distance reliability calculation unit.

6. The distance estimation apparatus according to claim 5, wherein the reliability calculation unit calculates the reliability in a manner that a value of the reliability is smaller as a value of distance information of a target pixel of the distance image is larger.

7. The distance estimation apparatus according to claim 5, wherein the reliability calculation unit calculates the reliability in a manner that a value of the reliability is smaller as an amplitude value of the charge signal corresponding to a target pixel of the distance image is smaller.

8. A distance estimation method for illuminating an object with light having a modulated light intensity and estimating a distance to an object using reflected light from the object, the method being used by a distance estimation apparatus including a light source operable to emit light whose light intensity can be modulated, a light receiving optical system operable to focus light from the object, a color separation unit operable to separate light received by the light receiving optical system into a first light component and a second light component based on a frequency of the light, a light receiving unit operable to convert the second light component separated by the color separation unit to obtain charge, a charge accumulation unit operable to accumulate the charge obtained by the light receiving unit and obtain a charge signal, and an imaging unit operable to convert the second light component separated by the color separation unit to an image generation charge signal, the method comprising:

controlling the light source;

calculating distance information based on the charge signal;

generating a distance image having a first number of pixels based on the distance information;

generating a high resolution image having pixels more than the first number of pixels based on the image generation charge signal obtained by the imaging unit;

extracting edge information from the high resolution image generated in the high resolution image generation step; and

correcting distance information of a target pixel of the distance image based on a neighboring pixel of the target pixel in the distance image and based on the edge information extracted in the edge information extraction step.

9. A distance estimation method for illuminating an object with light having a modulated light intensity and estimating a distance to an object using reflected light from the object, the method being used by a distance estimation apparatus including a light source operable to emit light whose light intensity can be modulated, a light receiving optical system operable to focus light from the object, a color separation unit operable to separate light received by the light receiving optical system into a first light component and a second light component based on a frequency of the light, a light receiving unit operable to convert the second light component separated by the color separation unit to obtain charge, a charge accumulation unit operable to accumulate the charge obtained by the light receiving unit and obtain a charge signal, and an imaging unit operable to convert the second light component separated by the color separation unit to an image generation charge signal,

the method comprising:

controlling the light source;

calculating distance information based on the charge signal;

generating a distance image having a first number of pixels based on the distance information;

storing the distance image;

generating a high resolution image having pixels more than the first number of pixels based on the image generation charge signal obtained by the imaging unit;

storing the high resolution image;

detecting a motion vector by comparing a high resolution image Img_color(t) that is obtained at a predetermined unit timing t in the high resolution image generation step and a high resolution image Img_color(t−

α

) that is obtained at a timing (t−

α

) preceding the predetermined unit timing t and stored in the high resolution image storage step; and

correcting distance information of a target pixel of the distance image based on a distance image Img_distance(t) that is obtained at the predetermined unit timing t in the distance image generation step, a distance image Img_distance(t−

α

) that is obtained at the timing (t−

α

) preceding the predetermined unit timing t and stored in the distance image storage step, and a motion vector that is detected in the motion vector detection step.

10. A distance estimation method for illuminating an object with light having a modulated light intensity and estimating a distance to an object using reflected light from the object, the method being used by a distance estimation apparatus including a light source operable to emit light whose light intensity can be modulated, a light receiving optical system operable to focus light from the object, a light receiving unit operable to convert the light received by the light receiving optical system to obtain charge, and a charge accumulation unit operable to accumulate the charge obtained by the light receiving unit and obtain a charge signal,

the method comprising:

controlling the light source;

calculating distance information based on the charge signal;

generating a distance image based on the distance information;

calculating a reliability of a pixel of the distance image based on at least one of the distance information and the charge signal; and

correcting distance information of the pixel of the distance image based on the reliability calculated in the distance reliability calculation step.

11. A storage medium storing a distance estimation program for illuminating an object with light having a modulated light intensity and estimating a distance to an object using reflected light from the object, the program enabling a computer to implement a distance estimation method used by a distance estimation apparatus including a light source operable to emit light whose light intensity can be modulated, a light receiving optical system operable to focus light from the object, a color separation unit operable to separate light received by the light receiving optical system into a first light component and a second light component based on a frequency of the light, a light receiving unit operable to convert the second light component separated by the color separation unit to obtain charge, a charge accumulation unit operable to accumulate the charge obtained by the light receiving unit and obtain a charge signal, and an imaging unit operable to convert the second light component separated by the color separation unit to an image generation charge signal,

the storage medium storing the program enabling a computer to implement the method comprising:

controlling the light source;

calculating distance information based on the charge signal;

generating a distance image having a first number of pixels based on the distance information;

generating a high resolution image having pixels more than the first number of pixels based on the image generation charge signal obtained by the imaging unit;

extracting edge information from the high resolution image generated in the high resolution image generation step; and

correcting distance information of a target pixel of the distance image based on a neighboring pixel of the target pixel in the distance image and based on the edge information extracted in the edge information extraction step.

12. A storage medium storing a distance estimation program for illuminating an object with light having a modulated light intensity and estimating a distance to an object using reflected light from the object, the program enabling a computer to implement a distance estimation method used by a distance estimation apparatus including a light source operable to emit light whose light intensity can be modulated, a light receiving optical system operable to focus light from the object, a color separation unit operable to separate light received by the light receiving optical system into a first light component and a second light component based on a frequency of the light, a light receiving unit operable to convert the second light component separated by the color separation unit to obtain charge, a charge accumulation unit operable to accumulate the charge obtained by the light receiving unit and obtain a charge signal, and an imaging unit operable to convert the second light component separated by the color separation unit to an image generation charge signal,

the storage medium storing the program enabling a computer to implement the method comprising:

controlling the light source;

calculating distance information based on the charge signal;

generating a distance image having a first number of pixels based on the distance information;

storing the distance image;

generating a high resolution image having pixels more than the first number of pixels based on the image generation charge signal obtained by the imaging unit;

storing the high resolution image;

detecting a motion vector by comparing a high resolution image Img_color(t) that is obtained at a predetermined unit timing t in the high resolution image generation step and a high resolution image Img_color(t−

α

) that is obtained at a timing (t−

α

) preceding the predetermined unit timing t and stored in the high resolution image storage step; and

correcting distance information of a target pixel of the distance image based on a distance image Img_distance(t) that is obtained at the predetermined unit timing t in the distance image generation step, a distance image Img_distance(t−

α

) that is obtained at the timing (t−

α

) preceding the predetermined unit timing t and stored in the distance image storage step, and a motion vector that is detected in the motion vector detection step.

13. A storage medium storing a distance estimation program for illuminating an object with light having a modulated light intensity and estimating a distance to an object using reflected light from the object, the program enabling a computer to implement a distance estimation method used by a distance estimation apparatus including a light source operable to emit light whose light intensity can be modulated, a light receiving optical system operable to focus light from the object, a light receiving unit operable to convert the light received by the light receiving optical system to obtain charge, and a charge accumulation unit operable to accumulate the charge obtained by the light receiving unit and obtain a charge signal,

the storage medium storing the program enabling a computer to implement the method comprising:

controlling the light source;

calculating distance information based on the charge signal;

generating a distance image based on the distance information;

calculating a reliability of a pixel of the distance image based on at least one of the distance information and the charge signal; and

correcting distance information of the pixel of the distance image based on the reliability calculated in the distance reliability calculation step.

14. An integrated circuit used in a distance estimation apparatus that illuminates an object with light having a modulated light intensity and estimates a distance to an object using reflected light from the object, wherein the distance estimation apparatus includes a light source operable to emit light whose light intensity can be modulated, a light receiving optical system operable to focus light from the object, a color separation unit operable to separate light received by the light receiving optical system into a first light component and a second light component based on a frequency of the light, a light receiving unit operable to convert the second light component separated by the color separation unit to obtain charge, a charge accumulation unit operable to accumulate the charge obtained by the light receiving unit and obtain a charge signal, and an imaging unit operable to convert the second light component separated by the color separation unit to an image generation charge signal,

the integrated circuit comprising:

a light source control unit operable to control the light source;

a signal processing unit operable to calculate distance information based on the charge signal;

a distance image generation unit operable to generate a distance image having a first number of pixels based on the distance information;

a high resolution image generation unit operable to generate a high resolution image having pixels more than the first number of pixels based on the image generation charge signal obtained by the imaging unit;

an edge information extraction unit operable to extract edge information from the high resolution image generated by the high resolution image generation unit; and

a distance image correction unit operable to correct distance information of a target pixel of the distance image based on a neighboring pixel of the target pixel in the distance image and based on the edge information extracted by the edge information extraction unit.

15. An integrated circuit used in a distance estimation apparatus that illuminates an object with light having a modulated light intensity and estimates a distance to an object using reflected light from the object, wherein the distance estimation apparatus includes a light source operable to emit light whose light intensity can be modulated, a light receiving optical system operable to focus light from the object, a color separation unit operable to separate light received by the light receiving optical system into a first light component and a second light component based on a frequency of the light, a light receiving unit operable to convert the second light component separated by the color separation unit to obtain charge, a charge accumulation unit operable to accumulate the charge obtained by the light receiving unit and obtain a charge signal, and an imaging unit operable to convert the second light component separated by the color separation unit to an image generation charge signal,

the integrated circuit comprising:

a light source control unit operable to control the light source;

a signal processing unit operable to calculate distance information based on the charge signal;

a distance image generation unit operable to generate a distance image having a first number of pixels based on the distance information;

a distance image storage unit operable to store the distance image;

a high resolution image generation unit operable to generate a high resolution image having pixels more than the first number of pixels based on the image generation charge signal obtained by the imaging unit;

a high resolution image storage unit operable to store the high resolution image;

a motion vector detection unit operable to detect a motion vector by comparing a high resolution image Img_color(t) that is obtained at a predetermined unit timing t by the high resolution image generation unit and a high resolution image Img_color(t−

α

) that is obtained at a timing (t−

α

) preceding the predetermined unit timing t and stored in the high resolution image storage unit; and

a distance image correction unit operable to correct distance information of a target pixel of the distance image based on a distance image Img_distance(t) that is obtained at the predetermined unit timing t by the distance image generation unit, a distance image Img_distance(t−

α

) that is obtained at the timing (t−

α

) preceding the predetermined unit timing t and stored in the distance image storage unit, and a motion vector that is detected by the motion vector detection unit.

16. An integrated circuit used in a distance estimation apparatus that illuminates an object with light having a modulated light intensity and estimates a distance to an object using reflected light from the object, wherein the distance estimation apparatus includes a light source operable to emit light whose light intensity can be modulated, a light receiving optical system operable to focus light from the object, a light receiving unit operable to convert the light received by the light receiving optical system to obtain charge, and a charge accumulation unit operable to accumulate the charge obtained by the light receiving unit and obtain a charge signal,

the integrated circuit comprising:

a light source control unit operable to control the light source;

a signal processing unit operable to calculate distance information based on the charge signal;

a distance image generation unit operable to generate a distance image based on the distance information;

a distance reliability calculation unit operable to calculate a reliability of a pixel of the distance image based on at least one of the distance information and the charge signal; and

a distance image correction unit operable to correct distance information of the pixel of the distance image based on the reliability calculated by the distance reliability calculation unit.