OBSTACLE DETECTION DEVICE AND METHOD AND OBSTACLE DETECTION SYSTEM

• US 20120140988A1
• Filed: 08/06/2010
• Published: 06/07/2012
• Est. Priority Date: 08/12/2009
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Takahashi, Katsuhiko
• Granted Patent: US 8,755,634 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

An obstacle region candidate point relating unit assumes that a pixel in an image corresponds to a point on a road surface, and associates pixels between images at two times on the basis of the amount of movement of a vehicle in question, a road plane, and a flow of the image estimated. When a pixel corresponds to a shadow of the vehicle in question or the moving object therearound appearing on the road surface, the ratio of intensities of the pixel values of the spectral images between two images should be approximately the same as the ratio of the spectral characteristics of the sunshine in the sun and the shade. Therefore, when the ratio of intensities is approximately the same as the ratio of the spectral characteristics, the obstacle determining unit does not determine that the pixel in question is a point corresponding to the obstacle. Only when the ratio of intensities is not approximately the same as the ratio of the spectral characteristics, the obstacle determining unit determines that the pixel in question is a point corresponding to the obstacle.

Claims

1. An obstacle detection device for detecting an obstacle around a vehicle from a spectral image obtained by an imaging unit mounted on the vehicle, the obstacle detection device comprising:

an image storage unit for storing an image provided by the imaging unit;

a flow extraction unit for extracting a flow of a pixel from images stored at two different times in the image storage unit, the two different times being a current time and an earlier time than the current time;

a vehicle movement amount estimation unit for estimating an amount of movement of the vehicle in a real world based on the extracted flow;

a road plane estimation unit for estimating a position of a road plane based on the extracted flow and the amount of movement of the vehicle;

an obstacle candidate point relating unit relating pixels in images at the two times by using a pixel in an image at an earlier time as a reference, by assuming that the pixel corresponds to the road plane in the real world, and by extracting a corresponding pixel at the current time based on the amount of movement of the vehicle;

a non-matching range identifying unit for comparing, pixel by pixel, partial images having centers at obstacle candidate points related between the pixels in the images at the two times, and identifying a range in which pixel values are different as a non-matching range; and

an obstacle determining unit for comparing the pixel values in the spectral images at the two times corresponding to the non-matching range, and determining whether the obstacle candidate point comprises a point corresponding to a shadow on the road plane or a point corresponding to an actual obstacle.

2. The obstacle detection device according to claim 1, wherein an obstacle candidate point relating unit relates a pixel producing a flow which is not consistent with the amount of movement of the vehicle and the road plane.

3. The obstacle detection device according to claim 1, wherein the obstacle candidate point relating unit relates a pixel at which the flow does not satisfy epipolar constraint when the amount of movement of the vehicle is assumed.

4. The obstacle detection device according to claim 1, further comprising:

a current time obtaining unit for obtaining a current time; and

a sunshine spectral characteristic database for storing relationship between a time and spectral intensities of sunshine in the sun and the shade,

wherein the obstacle determining unit obtains the spectral intensities of sunshine in the sun and the shade from the sunshine spectral characteristic database corresponding to the current time of the current time obtaining unit, and determines whether the obstacle candidate point corresponds to the obstacle or not, based on spectral characteristic thereof.

9. An obstacle detection system comprising:

a vehicle-mounted apparatus having the obstacle detection device according to claim 1 mounted thereon; and

an information center internal apparatus for transmitting and receiving data to/from the vehicle-mounted apparatus,

the vehicle-mounted apparatus including;

a vehicle-side data transmission unit for transmitting current position information to the information center internal apparatus; and

a vehicle-side data reception unit for receiving spectral characteristic data from the information center internal apparatus,

the information center internal apparatus including;

a detailed sunshine spectral characteristic database for storing relationship between spectral characteristics of sunshine in the sun and the shade in association with position information and a time;

a center-side data reception unit for receiving the current position information from the vehicle-side data transmission unit of the vehicle-mounted apparatus;

a sunshine spectral characteristic obtaining unit for searching spectral characteristic data of sunshine in the sun and the shade corresponding to the position information and the current time from the detailed sunshine spectral characteristic database; and

a center-side data transmission unit for transmitting the spectral characteristic data obtained by the sunshine spectral characteristic obtaining unit to the vehicle-mounted apparatus,

wherein the obstacle determining unit determines whether the obstacle candidate point corresponds to an obstacle or not, based on the spectral characteristic data received by the vehicle-side data reception unit.

11. The obstacle detection device according to claim 2, further comprising:

a current time obtaining unit for obtaining a current time; and

a sunshine spectral characteristic database for storing relationship between a time and spectral intensities of sunshine in the sun and the shade,

wherein the obstacle determining unit obtains the spectral intensities of sunshine in the sun and the shade from the sunshine spectral characteristic database corresponding to the current time of the current time obtaining unit, and determines whether the obstacle candidate point corresponds to the obstacle or not, based on spectral characteristic thereof.

12. The obstacle detection device according to claim 3, further comprising:

a current time obtaining unit for obtaining a current time; and

a sunshine spectral characteristic database for storing relationship between a time and spectral intensities of sunshine in the sun and the shade,

wherein the obstacle determining unit obtains the spectral intensities of sunshine in the sun and the shade from the sunshine spectral characteristic database corresponding to the current time of the current time obtaining unit, and determines whether the obstacle candidate point corresponds to the obstacle or not, based on spectral characteristic thereof.

15. An obstacle detection system comprising:

a vehicle-mounted apparatus having the obstacle detection device according to claim 2 mounted thereon; and

an information center internal apparatus for transmitting and receiving data to/from the vehicle-mounted apparatus, the vehicle-mounted apparatus including;

a vehicle-side data transmission unit for transmitting current position information to the information center internal apparatus; and

a vehicle-side data reception unit for receiving spectral characteristic data from the information center internal apparatus,

the information center internal apparatus including;

a detailed sunshine spectral characteristic database for storing relationship between spectral characteristics of sunshine in the sun and the shade in association with position information and a time;

a center-side data reception unit for receiving the current position information from the vehicle-side data transmission unit of the vehicle-mounted apparatus;

a sunshine spectral characteristic obtaining unit for searching spectral characteristic data of sunshine in the sun and the shade corresponding to the position information and the current time from the detailed sunshine spectral characteristic database; and

a center-side data transmission unit for transmitting the spectral characteristic data obtained by the sunshine spectral characteristic obtaining unit to the vehicle-mounted apparatus,

wherein the obstacle determining unit determines whether the obstacle candidate point corresponds to an obstacle or not, based on the spectral characteristic data received by the vehicle-side data reception unit.

16. An obstacle detection system comprising:

a vehicle-mounted apparatus having the obstacle detection device according to claim 3 mounted thereon; and

an information center internal apparatus for transmitting and receiving data to/from the vehicle-mounted apparatus, the vehicle-mounted apparatus including;

a vehicle-side data transmission unit for transmitting current position information to the information center internal apparatus; and

a vehicle-side data reception unit for receiving spectral characteristic data from the information center internal apparatus,

the information center internal apparatus including;

a detailed sunshine spectral characteristic database for storing relationship between spectral characteristics of sunshine in the sun and the shade in association with position information and a time;

a center-side data reception unit for receiving the current position information from the vehicle-side data transmission unit of the vehicle-mounted apparatus;

a sunshine spectral characteristic obtaining unit for searching spectral characteristic data of sunshine in the sun and the shade corresponding to the position information and the current time from the detailed sunshine spectral characteristic database; and

a center-side data transmission unit for transmitting the spectral characteristic data obtained by the sunshine spectral characteristic obtaining unit to the vehicle-mounted apparatus,

wherein the obstacle determining unit determines whether the obstacle candidate point corresponds to an obstacle or not, based on the spectral characteristic data received by the vehicle-side data reception unit.

17. An obstacle detection system comprising:

a vehicle-mounted apparatus having the obstacle detection device according to claim 4 mounted thereon; and

an information center internal apparatus for transmitting and receiving data to/from the vehicle-mounted apparatus, the vehicle-mounted apparatus including;

a vehicle-side data transmission unit for transmitting current position information to the information center internal apparatus; and

a vehicle-side data reception unit for receiving spectral characteristic data from the information center internal apparatus,

the information center internal apparatus including;

a detailed sunshine spectral characteristic database for storing relationship between spectral characteristics of sunshine in the sun and the shade in association with position information and a time;

a center-side data reception unit for receiving the current position information from the vehicle-side data transmission unit of the vehicle-mounted apparatus;

a sunshine spectral characteristic obtaining unit for searching spectral characteristic data of sunshine in the sun and the shade corresponding to the position information and the current time from the detailed sunshine spectral characteristic database; and

a center-side data transmission unit for transmitting the spectral characteristic data obtained by the sunshine spectral characteristic obtaining unit to the vehicle-mounted apparatus,

wherein the obstacle determining unit determines whether the obstacle candidate point corresponds to an obstacle or not, based on the spectral characteristic data received by the vehicle-side data reception unit.

5. An obstacle detection method for detecting an obstacle around a vehicle from a spectral image obtained by an imaging unit mounted on the vehicle, the obstacle detection method comprising:

causing an image storage unit to store a image provided by the imaging unit;

extracting a flow of a pixel from images stored at two different times in the image storage unit, the two different times being a current time and an earlier time than the current time;

estimating an amount of movement of the vehicle in a real world based on the extracted flow;

estimating a position of road plane based on the extracted flow and the amount of movement of the vehicle;

relating pixels in images at the two times by using a pixel in an image at an earlier time as a reference, assuming that the pixel corresponds to the road plane in the real world, and extracting a corresponding pixel at the current time based on the amount of movement of the vehicle;

comparing, pixel by pixel, partial images having centers at obstacle candidate points related between the pixels in the images at the two times, and identifying a range in which pixel values are different as a non-matching range; and

comparing the pixel values in the spectral images at the two times corresponding to the non-matching range, and determining whether the obstacle candidate point comprises a point corresponding to a shadow on the road plane or a point corresponding to an actual obstacle.

6. The obstacle detection method according to claim 5, wherein when in relating the pixels in the images at the two times, a pixel producing a flow not in consistent with the amount of movement of the vehicle and the road plane is related.

7. The obstacle detection method according to claim 5, wherein in relating the pixels in the images at the two times, a pixel is associated at which the flow does not satisfy epipolar constraint when the amount of movement of the vehicle is assumed.

8. The obstacle detection method according to claim 5, wherein in determining whether the obstacle candidate point comprises a point corresponding to the shadow on the road plane or a point corresponding to the actual obstacle, spectral intensities of sunshine in the sun and the shade corresponding to a current time are obtained from a sunshine spectral characteristic database storing relationship between a time and the spectral intensities of sunshine in the sun and the shade, and a determination is made as to whether the obstacle candidate point corresponds to the obstacle or not, based on spectral characteristic thereof.

13. The obstacle detection method according to claim 6, wherein in determining whether the obstacle candidate point comprises a point corresponding to the shadow on the road plane or a point corresponding to the actual obstacle, spectral intensities of sunshine in the sun and the shade corresponding to a current time are obtained from a sunshine spectral characteristic database storing relationship between a time and the spectral intensities of sunshine in the sun and the shade, and a determination is made as to whether the obstacle candidate point corresponds to the obstacle or not, based on spectral characteristic thereof.

14. The obstacle detection method according to claim 7, wherein in determining whether the obstacle candidate point comprises a point corresponding to the shadow on the road plane or a point corresponding to the actual obstacle, spectral intensities of sunshine in the sun and the shade corresponding to a current time are obtained from a sunshine spectral characteristic and the shade corresponding to a current time are obtained from a sunshine spectral characteristic database storing relationship between a time and the spectral intensities of sunshine in the sun and the shade, and a determination is made as to whether the obstacle candidate point corresponds to the obstacle or not, based on spectral characteristic thereof.

10. A computer-readable information recording medium storing a program for causing a computer to function as a flow extraction unit for extracting a flow of a pixel from images stored at two different times in an image storage unit, the two different times being a current time and an earlier time than the current time;

a vehicle movement amount estimation unit for estimating an amount of movement of the vehicle in a real world based on the extracted flow;

a road plane estimation unit for estimating a position of road plane based on the extracted flow and the amount of movement of the vehicle;

an obstacle candidate point relating unit relating pixels in images at the two times by using a pixel in an image at an earlier time as a reference, assuming that the pixel corresponds to the road plane in the real world, and extracting a corresponding pixel at the current time based on the amount of movement of the vehicle;

a non-matching range identifying unit for comparing, pixel by pixel, partial images having centers at obstacle candidate points relater between the pixels in the images at the two times, and identifying a range in which pixel values are different as a non-matching range; and

an obstacle determining unit for comparing the pixel values in the spectral images at the two times corresponding to the non-matching range, and determining whether the obstacle candidate point comprises a point corresponding to a shadow on the road plane or a point corresponding to an actual obstacle.