METHOD OF GENERATING RANGE IMAGES AND APPARATUS THEREFOR

• US 20080106620A1
• Filed: 10/30/2007
• Published: 05/08/2008
• Est. Priority Date: 11/02/2006
• Status: Active Grant

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Inventors: SAWACHI, Youichi
• Granted Patent: US 7,911,496 B2
• Time in Patent Office: Days
• US Class Current: 348/262
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/257 Colour aspects

Abstract

According to an aspect of the invention, since two range images (first and second range images) are individually generated by a first range image generating device and a second range image generating device by different methods, and one range image (a third range image) is generated on the basis of those range images, highly accurate range images are generated. Furthermore, as the first range image generating device and the second range image generating device commonly use image sensors, the increase in hardware size and in cost due to the use of two devices for generating range images can be restrained.

Claims

1. A range image generating apparatus comprising:

a plurality of image pickup units having an optical system which forms images of subjects and image sensors which capture the images of the subjects formed by the optical system with two-dimensionally arrayed light receiving elements as electric signals;

a first image capturing device which captures from each of the plurality of image pickup units an image of the subject formed by light in a first wavelength range as a first image;

a second image capturing device which captures from at least one of the plurality of image pickup units an image of the subject formed by light in a second wavelength range as a second image;

a first range image generating device which generates a first range image having values of pixels as distance data indicating a distance on the basis of a plurality of first images captured by the first image capturing device;

a second range image generating device which generates a second range image having values of pixels as distance data indicating a distance on the basis of a second image captured by the second image capturing device;

a third range image generating device which generates a third range image on the basis of the first range image and the second range image; and

a recording device which records the third range image onto a recording medium.

2. The range image generating apparatus according to claim 1, further comprising a light radiating device which irradiates the subject with light in the second wavelength range, wherein

the second image capturing device irradiates the subject with light in the second wavelength range by using the light radiating device and captures an image of the subject formed by the reflected light thereof as the second image.

3. The range image generating apparatus according to claim 1, wherein the first range image generating device is a range image generating device using a stereo matching technique which detects, from the plurality of first images, corresponding points indicating image points for the same object point, figures out the distance to the object point on the basis of the parallax of the corresponding points, and generates the first range image by using distance data indicating the distance as values of the pixels of the corresponding points.

4. The range image generating apparatus according to claim 1, wherein the second range image generating device is a range image generating device using a TOF technique which figures out the distance to the object point having each pixel as the image point on the basis of the second image representing pixel values according to the length of time taken by the light of the second wavelength range to irradiate the subject and reach the image pickup units, and generates the second range image by using the distance data indicating the distance as the value of each pixel.

5. The range image generating apparatus according to claim 1, wherein the third range image generating device generates the third range image by interpolating the first range image with the second range image.

6. The range image generating apparatus according to claim 1, wherein the third range image generating device generates the third range image by interpolating the distance data of pixels around the corresponding points detected for the first range image by the first range image generating device with the second range image.

7. The range image generating apparatus according to claim 1, wherein the first wavelength range is a visible wavelength range and the second wavelength range is a wavelength range including invisible light.

8. The range image generating apparatus according to claim 1, wherein the second wavelength range is an infrared wavelength range.

9. The range image generating apparatus according to claim 1,

wherein in the image pickup unit which captures the first images with the first image capturing device, first pixel filters of a single type which transmit light of the first wavelength range or first pixel filters comprising a plurality of types which transmit a light in each of a plurality of wavelength ranges into which the first wavelength range is divided are arranged in the light receiving elements of the image sensors, and the first image capturing device captures the first images with the light receiving elements in which the first pixel filters are arranged, and

wherein in the image pickup unit which captures the second images with the second image capturing device, second pixel filters which transmit light of the second wavelength range are arranged in the light receiving elements of the image sensors, and the second image capturing device captures the second images with the light receiving elements in which the second pixel filters are arranged.

10. The range image generating apparatus according to claim 9, wherein

in the image pickup unit which captures the second images with the second image capturing device, a second wavelength range cut filter which cuts off light in the second wavelength range is detachably arranged on an optical path where light coming incident on the image sensors passes,

the first image capturing device captures the first images in a state in which the second wavelength range cut filter is inserted into the optical path, and

the second image capturing device captures the second images in a state in which the second wavelength range cut filter is kept away from the optical path.

11. The range image generating apparatus according to claim 9, wherein in the image pickup unit which captures the second images with the second image capturing device, the second pixel filters arranged in the light receiving elements of the image sensors are first pixel filters of a prescribed type having filtering characteristics to transmit light of the second wavelength range.

12. The range image generating apparatus according to claim 11, wherein

the first wavelength range is a visible wavelength range, and

the first pixel filters comprise pixel filters transmitting light of the red wavelength range, pixel filters transmitting light of the green wavelength range and pixel filters transmitting light of the blue wavelength range.

13. The range image generating apparatus according to claim 11, wherein the prescribed type of first pixel filters having filtering characteristics to transmit light of the second wavelength range are pixel filters which transmit light of the red wavelength range.

14. The range image generating apparatus according to claim 11, wherein the prescribed type of first pixel filters having filtering characteristics to transmit light of the second wavelength range are all the types of first pixel filters.

15. The range image generating apparatus according to claim 11, further comprising a second image correcting device which corrects the second images by subtracting, from the values of the pixels of the second images captured by the second image capturing device, values according to the values of corresponding pixels of the first images captured by the first image capturing device from the same image pickup unit as the captured second images.

16. The range image generating apparatus according to claim 15, wherein the second image correcting device corrects the second images by subtracting the product of multiplying the value of the corresponding pixel in the first images by a prescribed value from the value of each pixel in the second images.

17. The range image generating apparatus according to claim 16, wherein the prescribed value is set according to the ratio of the level of exposure at the time of picking up the second images and the level of exposure at the time of picking up the first images.

18. The range image generating apparatus according to claim 9, wherein in the image pickup unit into which the second images are captured by the second image capturing device, the second pixel filters are pixel filters arranged in different light receiving elements from the light receiving elements in which the first pixel filters are arranged.

19. The range image generating apparatus according to claim 18, wherein the first wavelength range is the green wavelength range and the first pixel filters are pixel filters which transmit light in the green wavelength range.

20. The range image generating apparatus according to claim 18, wherein the second wavelength range is the infrared wavelength range and the second pixel filters are pixel filters which transmit light in the infrared wavelength range.

21. The range image generating apparatus according to claim 18, wherein in the image pickup unit into which the second images are captured by the second image capturing device, the first pixel filters and the second pixel filters are arranged alternately.

22. The range image generating apparatus according to claim 18, wherein in the image pickup unit into which the second images are captured by the second image capturing device, the image sensors are so configured that each of the light receiving elements is connected changeably between the electric charge discharging side and the electric charge transferring side via electrical switches.

23. The range image generating apparatus according to claim 22, wherein the electrical switches are independently controlled by the light receiving elements in which the first pixel filters are arranged and the light receiving elements in which the second pixel filters are arranged.

24. The range image generating apparatus according to claim 18, wherein the image pickup unit into which the second images are captured by the second image capturing device is so configured as to prevent the light receiving elements in which the first pixel filters are arranged and the light receiving elements in which the second pixel filters are arranged from overlapping each other in the period of exposure.

25. The range image generating apparatus according to claim 24, further comprising a light radiating device which irradiates the subject with light in the second wavelength range, wherein

the subject is irradiated by the light radiating device with light in the second wavelength range during the exposure period of the light receiving elements in which the second pixel filters are arranged, and

the irradiation with light by the light radiating device is suspended during the exposure period of the light receiving elements in which the first pixel filters are arranged.

26. The range image generating apparatus according to claim 25, wherein exposure of the light receiving elements in which the first pixel filters are arranged is not executed in the period in which the light radiating device is radiating light and in the period over which the reflected light of the light from the light radiating device arrives.

27. The range image generating apparatus according to claim 1, wherein the recording device records the first image captured from one image pickup unit out of the plurality of image pickup units onto the recording medium together with the third range image.

28. The range image generating apparatus according to claim 27, wherein a first camera unit having an image pickup unit into which first images to be recorded onto the recording medium are captured and a second camera unit having other image pickup units than this image pickup unit are made separable.

29. A method of generating range images comprising:

a first image capturing step of capturing, from each of a plurality of image pickup units provided with an optical system which forms images of subjects and image sensors which capture the images of the subjects formed by the optical system with two-dimensionally arrayed light receiving elements as electric signals, an image of the subject formed by light in a first wavelength range as a first image;

a second image capturing step of capturing from at least one of the plurality of image pickup units an image of the subject formed by light in a second wavelength range as a second image;

a first range image generating step of generating a first range image having the values of pixels as distance data indicating a distance on the basis of a plurality of first images captured at the first image capturing step;

a second range image generating step of generating a second range image having the values of pixels as distance data indicating a distance on the basis of second images captured at the second image capturing step;

a third range image generating step of generating a third range image on the basis of the first range image and the second range image; and

a recording step of recording the third range image onto a recording medium.

30. The method of generating range images according to claim 29, wherein in the second image capturing step, the subject is irradiated with light in the second wavelength range by a light radiating device which irradiates the subject with light in the second wavelength range, and an image of the subject formed by the reflected light thereof is captured as the second image.

31. The method of generating range images according to claim 29, wherein the first range image generating step is a range image generating step using a stereo matching technique which detects from the plurality of first images corresponding points indicating image points for the same object point, figures out the distance to the object point on the basis of the parallax of the corresponding points, and generates the first range image by using distance data indicating the distance as values of the pixels of the corresponding points.

32. The method of generating range images according to claim 29, wherein the second range image generating step is a range image generating step using a TOF technique which figures out the distance to the object point having each pixel as the image point on the basis of the second image representing pixel values according to the length of time taken by the light of the second wavelength range to irradiate the subject and reach the image pickup units, and generates the second range image by using the distance data indicating the distance as the value of each pixel.

33. The method of generating range images according to claim 29, wherein in the third range image generating step, the third range image is generated by interpolating the first range image with the second range image.

34. The method of generating range images according to claim 29, wherein in the third range image generating step, the third range image is generated by interpolating the distance data of pixels around the corresponding points detected for the first range image at the first range image generating step with the second range image.

35. The method of generating range images according to claim 29, wherein the first wavelength range is a visible wavelength range and the second wavelength range is a wavelength range including invisible light.

36. The method of generating range images according to claims 29, wherein the second wavelength range is an infrared wavelength range.

37. The method of generating range images according to claim 29,

wherein in the image pickup unit which captures the first images in the first image capturing step, first pixel filters of a single type which transmit light of the first wavelength range or first pixel filters comprising a plurality of types which transmit each of a plurality of wavelength ranges into which the first wavelength range is divided are arranged in the light receiving elements of the image sensors, and the first images are captured with the light receiving elements in which the first pixel filters are arranged in the first image capturing step, and

wherein in the image pickup unit which captures the second images in the second image capturing step, second pixel filters which transmit light of the second wavelength range are arranged in the light receiving elements of the image sensors, and the second images are captured with the light receiving elements in which the second pixel filters are arranged in the second image capturing step.

38. The method of generating range images according to claim 37,

wherein in the image pickup unit which captures the second images in the second image capturing step, a second wavelength range cut filter which cuts off light in the second wavelength range is detachably arranged on an optical path where light coming incident on the image sensors passes,

in the first image capturing step, the first images are captured in a state in which the second wavelength range cut filter is inserted into the optical path, and

in the second image capturing step, the second images are captured in a state in which the second wavelength range cut filter is kept away from the optical path.

39. The method of generating range images according to claim 37, wherein in the image pickup unit which captures the second images in the second image capturing step, the second pixel filters arranged in the light receiving elements of the image sensors are first pixel filters of a prescribed type having filtering characteristics to transmit light of the second wavelength range.

40. The method of generating range images according to claim 39, wherein the first wavelength range is a visible wavelength range and the first pixel filters comprise pixel filters transmitting light of the red wavelength range, pixel filters transmitting light of the green wavelength range and pixel filters transmitting light of the blue wavelength range.

41. The method of generating range images according to claim 39, wherein the prescribed type of first pixel filters having filtering characteristics to transmit light of the second wavelength range are pixel filters which transmit light of the red wavelength range.

42. The method of generating range images according to claim 39, wherein the prescribed type of first pixel filters having filtering characteristics to transmit light of the second wavelength range are all the types of first pixel filters.

43. The method of generating range images according to claim 39, further comprising a second image correcting step of correcting the second images by subtracting, from the values of the pixels of the second images captured in the second image capturing step, values according to the values of corresponding pixels of the first images captured at the first image capturing step from the same image pickup unit as the second images.

44. The method of generating range images according to claim 43, wherein in the second image correcting step, the second images are corrected by subtracting the product of multiplying the value of the corresponding pixel in the first images by a prescribed value from the value of each pixel in the captured second images.

45. The method of generating range images according to claim 44, wherein the prescribed value is set according to the ratio of the level of exposure at the time of capturing the second images and the level of exposure at the time of capturing the first images.

46. The method of generating range images according to claim 37, wherein in the image pickup unit into which the second images are captured in the second image capturing step, the second pixel filters are pixel filters arranged in different light receiving elements from the light receiving elements in which the first pixel filters are arranged.

47. The method of generating range images according to claim 46, wherein the first wavelength range is the green wavelength range and the first pixel filters are pixel filters which transmit light in the green wavelength range.

48. The method of generating range images according to claim 46, wherein the second wavelength range is the infrared wavelength range and the second pixel filters are pixel filters which transmit light in the infrared wavelength range.

49. The method of generating range images according to claim 46, wherein in the image pickup unit into which the second images are captured by the second image capturing device, the first pixel filters and the second pixel filters are arranged alternately.

50. The method of generating range images according to claim 46, wherein in the image pickup unit into which the second images are captured by the second image capturing step, the image sensors are so configured that each of the light receiving elements is connected switchably to the electric charge discharging side and the electric charge transferring side via electrical switches.

51. The method of generating range images according to claim 50, wherein the electrical switches are independently controlled by the light receiving elements in which the first pixel filters are arranged and the light receiving elements in which the second pixel filters are arranged.

52. The method of generating range images according to claim 46, wherein the image pickup unit into which the second images are captured by the second image capturing step is so configured as to prevent the light receiving elements in which the first pixel filters are arranged and the light receiving elements in which the second pixel filters are arranged from overlapping each other in the period of exposure.

53. The method of generating range images according to claim 52, wherein

the subject is irradiated with light in the second wavelength range by a light radiating device, which irradiates the subject with light in the second wavelength range during the exposure period of the light receiving elements in which the second pixel filters are arranged, and

the irradiation with light by the light radiating device is suspended during the exposure period of the light receiving elements in which the first pixel filters are arranged.

54. The method of generating range images according to claim 53, wherein exposure of the light receiving elements in which the first pixel filters are arranged is not executed in the period in which the light radiating device is radiating light and in the period over which the reflected light of the light from the light radiating device arrives.

55. The method of generating range images according to claim 29, wherein in the recording step, the first image captured from one image pickup unit out of the plurality of image pickup units is recorded onto the recording medium together with the third range image.