Image capturing device, method of searching for occlusion region, and program

• US 8,446,492 B2
• Filed: 12/09/2010
• Issued: 05/21/2013
• Est. Priority Date: 12/10/2009
• Status: Expired due to Fees

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Nagai, Takayuki, Nakano, Mikio, Mizutani, Akira, Nakamura, Tomoaki, Funakoshi, Kotaro
• Primary Examiner(s): Ho, Tuan
• Assistant Examiner(s): Nazrul, Shahbaz
• Application Number:
  US12/964,116
• Publication Number:
  US 20110141306A1
• Time in Patent Office: 894 Days
• Field of Search: 348234-236, 348/148, 348/43, 382/154
• US Class Current: 348/236
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor

Abstract

An image capturing device is equipped with an external parameter estimating unit for estimating external parameters using a distance image obtained by a TOF camera and a luminance image obtained by a CCD camera, a corresponding pixel determining unit for determining a correspondence relationship between pixel positions in the distance image and pixel positions in the luminance image, using previously stored internal parameters of the TOF camera and the CCD camera or the CCD camera, and the external parameters, and an occlusion searching unit for searching for an occlusion region in the distance image, using the correspondence relationship between pixel positions in the distance image and pixel positions in the luminance image.

Claims

1. An image capturing device comprising:

a distance image sensor;

at least two luminance image sensors;

an external parameter estimating unit for estimating external parameters using a distance image obtained by the distance image sensor and a luminance image obtained by the luminance image sensor;

a corresponding pixel determining unit for determining a correspondence relationship between pixel positions in the distance image and pixel positions in the luminance image, using previously stored internal parameters of the distance image sensor and the luminance image sensor, and the external parameters; and

an occlusion searching unit for searching for an occlusion region in the distance image, using the correspondence relationship between pixel positions in the distance image and pixel positions in the luminance image determined by the corresponding pixel determining unit.

2. The image capturing device according to claim 1, wherein the occlusion searching unit scans the pixels in the distance image in one direction, and sets, as a pixel position at which the occlusion starts, a pixel position of the pixels in the distance image at a time when a movement direction of pixels in the luminance image, which correspond to the pixels in the distance image, becomes opposite to the one direction.

3. The image capturing device according to claim 2, wherein the occlusion searching unit sets, as a pixel position at which the occlusion in the one direction ends, a pixel position of pixels in the distance image immediately before the pixel position of pixels in the luminance image, which correspond to the pixels in the distance image, become on the side of the one direction from a pixel position of pixels in the luminance image immediately before the movement direction changes.

4. The image capturing device according to claim 1, further comprising a mapping unit for mapping luminance information of pixels of the luminance image, which correspond to pixels in the distance image, to distance information of pixels in the distance image outside of the occlusion region, using the correspondence relationship between pixel positions in the distance image and pixel positions in the luminance image.

5. The image capturing device according to claim 4, comprising at least two of the luminance image sensors, wherein the mapping unit maps to distance information of the pixels of the occlusion region in the distance image, which have been searched for by the occlusion searching unit, using the luminance image captured by one of the luminance image sensors, luminance information of corresponding pixels of the luminance image captured by the other luminance image sensor.

6. An occlusion region searching method implemented by a computer having a distance image sensor and at least two luminance image sensors, comprising the steps of:

estimating external parameters using a distance image obtained by the distance image sensor and a luminance image obtained by the luminance image sensor;

determining a correspondence relationship between pixel positions in the distance image and pixel positions in the luminance image, using previously stored internal parameters of the distance image sensor and the luminance image sensor, and the external parameters; and

searching for an occlusion region in the distance image, using the determined correspondence relationship between pixel positions in the distance image and pixel positions in the luminance image.

7. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to enable a computer having a distance image sensor and at least two luminance image sensors to function as:

an external parameter estimating unit for estimating external parameters using a distance image obtained by the distance image sensor and a luminance image obtained by the luminance image sensor;

a corresponding pixel determining unit for determining a correspondence relationship between pixel positions in the distance image and pixel positions in the luminance image, using previously stored internal parameters of the distance image sensor and the luminance image sensor, and the external parameters; and

an occlusion searching unit for searching for an occlusion region in the distance image, using the correspondence relationship between pixel positions in the distance image and pixel positions in the luminance image determined by the corresponding pixel determining unit.