Three-dimensional object detection device

• US 9,349,057 B2
• Filed: 07/27/2012
• Issued: 05/24/2016
• Est. Priority Date: 09/12/2011
• Status: Active Grant

• Current Assignee: Nissan Motor Co., Ltd.
• Original Assignee: Nissan Motor Co., Ltd.
• Inventors: Tsuchiya, Chikao, Hayakawa, Yasuhisa, Tanaka, Shinya, Fukata, Osamu
• Primary Examiner(s): Shibru, Helen
• Application Number:
  US14/235,132
• Publication Number:
  US 20140168440A1
• Time in Patent Office: 1,397 Days
• Field of Search: 348/148, 382/103
• US Class Current: 1/1
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: B60R 2300/8066 for monitoring rearward traffic
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A three-dimensional object detection device basically includes a three-dimensional object detection unit, a natural object assessment unit and a control unit. The three-dimensional object detection unit detects three-dimensional objects based on image information of a rear of a vehicle from a camera. The natural object assessment unit assesses that a detected three-dimensional object is a natural object, such as a plant or snow, based on an irregularity evaluation value calculated based on a first pixel number of first pixels representing a first predetermined differential in the differential image containing the detected three-dimensional object and a second pixel number of second pixels corresponding to the three-dimensional object and representing a second predetermined differential greater than the first predetermined differential.

Claims

1. A three-dimensional object detection device comprising:

an imaging unit configured to be installed on a vehicle to capture images of a region at a rear of the vehicle;

an image conversion unit configured to convert a viewpoint of the images obtained by the imaging unit to bird'"'"'s-eye view images;

a differential waveform information creating unit configured to create differential waveform information by positionally aligning the bird'"'"'s-eye view images of different times obtained by the image conversion unit within a bird'"'"'s-eye view, the differential waveform information creating unit being further configured to create a frequency distribution of a number of pixels by counting the number of pixels representing a predetermined differential in a differential image of the bird'"'"'s-eye view images that were positionally aligned;

a three-dimensional object detection unit configured to detect three-dimensional objects which are included in the differential waveform information and which are present in detection areas set in left and right rear sides of the vehicle, based on the frequency distribution of the number of pixels representing the predetermined differential in the differential image along a direction in which the three-dimensional object falls when the bird'"'"'s-eye view images are viewpoint-converted; and

a natural object assessment unit programmed to calculate an irregularity evaluation value for evaluating an irregularity of the differential waveform information based on a first pixel number of first pixels representing a first predetermined differential in the differential image containing the detected three-dimensional object that was detected, and a second pixel number of second pixels that have been extracted along the direction in which the three-dimensional object falls when the bird'"'"'s-eye view images are viewpoint-converted and that represent a second predetermined differential greater than the first predetermined differential in the differential image, and the natural object assessment unit configured to assess that the three-dimensional object detected by the three-dimensional object detection unit is a natural object including plants or snow present along a lane traveled by the vehicle when the calculated irregularity evaluation value is equal to or greater than a predetermined irregularity evaluation threshold set in advance.

3. The three-dimensional object detection device according to claim 1, wherein

the natural object assessment unit is further programmed to:

calculate a third pixel number by subtracting the second pixel number from the first pixel number; and

assess that the irregularity evaluation value is equal to or greater than the predetermined irregularity evaluation threshold when the second pixel number is less than a first predetermined value and the third pixel number is equal to or greater than a second predetermined value.

4. The three-dimensional object detection device according to claim 1, wherein

the natural object assessment unit is further programmed to:

calculate a third pixel number by subtracting the second pixel number from the first pixel number; and

assess that the irregularity evaluation value is equal to or greater than the predetermined irregularity evaluation threshold when the value of subtracting the second pixel number from the third pixel number is equal to or greater than a third predetermined value.

5. The three-dimensional object detection device according to claim 1, wherein

the natural object assessment unit is further programmed to:

calculate a third pixel number by subtracting the second pixel number from the first pixel number;

calculate an abundance rate of the third pixel number relative to the first pixel number or the second pixel number; and

assess that the irregularity evaluation value is equal to or greater than the predetermined irregularity evaluation threshold when the abundance rate of the third pixel number relative to either the first pixel number or the second pixel number is equal to or greater than a predetermined rate.

6. The three-dimensional object detection device according to claim 1, wherein

the natural object assessment unit is further programmed to:

designate, as second pixels, pixels that are extracted along the direction in which the three-dimensional object falls when the bird'"'"'s-eye view images are viewpoint-converted, and that, among pixels representing the second predetermined differential, constitute a pixel group aligned in at least a predetermined density, and

find the number of these second pixels as the second pixel number.

8. The three-dimensional object detection device according to claim 1, further comprising:

a three-dimensional object assessment unit programmed to assess whether or not the three-dimensional object detected by the three-dimensional object detection unit is another vehicle in the detection areas; and

a control unit programmed to perform suppress an assessment that the three-dimensional object detected by the three-dimensional object detection unit is another vehicle when the natural object assessment unit has assessed that the three-dimensional object that was detected is a natural object including plants or snow present along the lane traveled by the vehicle.

9. The three-dimensional object detection device according to claim 8, wherein

the three-dimensional object detection unit detects a three-dimensional object based on the differential waveform information and a first threshold α

; and

the control unit outputs to the three-dimensional object detection unit a control command for raising the first threshold α

so as to impede detection of the three-dimensional object when the three-dimensional object that was detected is assessed by the natural object assessment unit to be a natural object including plants or snow present along the lane traveled by the vehicle.

10. The three-dimensional object detection device according to claim 8, wherein

the three-dimensional object detection unit detects a three-dimensional object based on a pixel number of the differential waveform information and a first threshold α

; and

the control unit creates a control command for lowering a value of the number of pixels representing a predetermined differential in the differential image of the bird'"'"'s-eye view images, and outputs the control command to the three-dimensional object detection unit when the three-dimensional object that was detected is assessed by the natural object assessment unit to be a natural object including plants or snow present along the lane traveled by the vehicle.

13. The three-dimensional object detection device according to claim 8, wherein

the three-dimensional object assessment unit positionally aligns a position of a first bird'"'"'s-eye view image of the bird'"'"'s-eye view images obtained at a first time at which the three-dimensional object is detected and a position of a second bird'"'"'s-eye view image of the bird'"'"'s-eye view images obtained at a second time after the first time within a bird'"'"'s-eye view,

the three-dimensional object assessment unit finds a first integrated value of first differential waveform information created by counting and creating a frequency distribution of a number of pixels representing a predetermined differential in the differential image of the positionally aligned bird'"'"'s-eye view images,

the three-dimensional object assessment unit finds a second integrated value of second differential waveform information created by counting and creating a frequency distribution of a number of pixels representing a predetermined differential in the differential image of the first bird'"'"'s-eye view image obtained at the first time and the second bird'"'"'s-eye view image obtained at the second time after the first time, and

the three-dimensional object assessment unit assesses that the three-dimensional object detected by the three-dimensional object detection unit is a moving object when the evaluation value corresponding to a number of times the first integrated value is assessed to be greater than the second integrated value is equal to or greater than a predetermined evaluation threshold;

and

the control unit halts suppressing the assessment of the three-dimensional object that was detected as another vehicle when the three-dimensional object is assessed by the three-dimensional object assessment unit to be a moving object.

15. The three-dimensional object detection device according to claim 13, wherein

the three-dimensional object assessment unit adds a first counted value to calculate the evaluation value when the first integrated value is assessed to be greater than the second integrated value;

and

the three-dimensional object assessment unit sets a higher value for the first counted value in response to a greater number of continuous assessments when the assessment of the first integrated value being greater than the second integrated value continues.

16. The three-dimensional object detection device according to claim 13, wherein

the three-dimensional object assessment unit calculates the evaluation value by adding a first counted value when the first integrated value is assessed to be greater than the second integrated value, and subtracting a second counted value when the first integrated value is assessed to be less than the second integrated value;

and

the three-dimensional object assessment unit sets a higher value for the second counted value when an assessment is made that the first integrated value is greater than the second integrated value, an assessment is then made that the first integrated value is less than the second integrated value, and an assessment is thereafter made that the first integrated value is greater than the second integrated value.

17. The three-dimensional object detection device according to claim 13, wherein

the three-dimensional object assessment unit is further programmed to:

subtracts a second counted value to calculate the evaluation value when the first integrated value is assessed to be less than the second integrated value; and

sets a higher value for the second counted value when the assessment of the first integrated value being less than the second integrated value continues for at least a predetermined number of times.

2. A three-dimensional object detection device comprising:

an imaging unit configured to be installed on a vehicle to capture images of a region at a of the vehicle;

an image conversion unit configured to convert a viewpoint of the images obtained by the imaging unit to bird'"'"'s-eye view images;

an edge information creating unit configured to create edge information by extracting pixels in which a luminance difference of adjacent image areas is equal to or greater than a predetermined threshold in the bird'"'"'s-eye view images obtained by the image conversion unit;

a three-dimensional object detection unit configured to detect three-dimensional objects which are included in the edge information and which are present in detection areas set in left and right rear sides of the vehicle, based on the edge information including the pixels that are extracted along the direction in which the three-dimensional object falls when the bird'"'"'s-eye view image is viewpoint-converted and that have a luminance difference of adjacent image areas equal to or greater than the predetermined threshold; and

a natural object assessment unit programmed to calculate an irregularity evaluation value for evaluating an irregularity of the edge information based on a first pixel number of first pixels in which the luminance difference of the adjacent image areas in the bird'"'"'s-eye images containing the three-dimensional object that was detected is equal to or greater than a first predetermined threshold, and a second pixel number of second pixels that have been extracted along the direction in which the three-dimensional object falls when the bird'"'"'s-eye view images are viewpoint-converted and in which the luminance difference of adjacent image areas in the bird'"'"'s-eye images is equal to or greater than a second predetermined threshold greater than the first predetermined threshold, and the natural object assessment unit configured to assess that the three-dimensional object detected by the three-dimensional object detection unit is a natural object including plants or snow present along a lane traveled by the vehicle when the calculated irregularity evaluation value is equal to or greater than a predetermined irregularity evaluation threshold set in advance.

7. The three-dimensional object detection device according to claim 2, wherein

the natural object assessment unit is further programmed to:

designate, as second pixels, pixels that are extracted along the direction in which the three-dimensional object falls when the bird'"'"'s-eye view image is viewpoint-converted, and that, among pixels in which the luminance difference of adjacent image areas in the bird'"'"'s-eye images are equal to or greater than a second predetermined threshold, constitute a pixel group aligned in at least a predetermined density, and

find the number of these second pixels as the second pixel number.

11. The three-dimensional object detection device according to claim 2, further comprising:

a three-dimensional object assessment unit programmed to assess whether or not the three-dimensional object detected by the three-dimensional object detection unit is another vehicle in the detection areas; and

a control unit programmed to perform suppress an assessment that the three-dimensional object detected by the three-dimensional object detection unit is another vehicle when the natural object assessment unit has assessed that the three-dimensional object that was detected is a natural object including plants or snow present along the lane traveled by the vehicle.

12. The three-dimensional object detection device according to claim 11, wherein

the three-dimensional object detection unit detects a three-dimensional object based on an amount of edge information and a second threshold β

; and

the control unit outputs to the three-dimensional object detection unit a control command for outputting a lower amount of detected edge information when the three-dimensional object that was detected is assessed by the natural object assessment unit to be a natural object including plants or snow present along the lane traveled by the vehicle.

14. The three-dimensional object detection device according to claim 11, wherein

the three-dimensional object assessment unit positionally aligns a position of a first bird'"'"'s-eye view image of the bird'"'"'s-eye view images obtained at a first time at which the three-dimensional object is detected and a position of a second bird'"'"'s-eye view image of the bird'"'"'s-eye view images obtained at a second time after the first time within a bird'"'"'s-eye view,

the three-dimensional object assessment unit finds a first integrated value of first luminance distribution information created by counting and creating a frequency distribution of a number of pixels in which the luminance difference of adjacent image areas is equal to or greater than a predetermined threshold in the differential image of the positionally aligned bird'"'"'s-eye view images,

the three-dimensional object assessment unit finds a second integrated value of second luminance distribution information created by counting and creating a frequency distribution of a number of pixels in which the luminance difference of adjacent image areas is equal to or greater than a predetermined threshold in the differential image of the first bird'"'"'s-eye view image obtained at the first time and the second bird'"'"'s-eye view image obtained at the second time after the first time, and

the three-dimensional object assessment unit assesses that the three-dimensional object detected by the three-dimensional object detection unit is a moving object when the evaluation value corresponding to a number of times the first integrated value is assessed to be greater than the second integrated value is equal to or greater than a predetermined evaluation threshold.

20. The three-dimensional object detection device according to claim 2, wherein

the natural object assessment unit is further programmed to:

calculate a third pixel number by subtracting the second pixel number from the first pixel number; and

assess that the irregularity evaluation value is equal to or greater than the predetermined irregularity evaluation threshold when the second pixel number is less than a first predetermined value and the third pixel number is equal to or greater than a second predetermined value.

21. The three-dimensional object detection device according to claim 2, wherein

the natural object assessment unit is further programmed to:

calculate a third pixel number by subtracting the second pixel number from the first pixel number; and

assess that the irregularity evaluation value is equal to or greater than the predetermined irregularity evaluation threshold when the value of subtracting the second pixel number from the third pixel number is equal to or greater than a third predetermined value.

22. The three-dimensional object detection device according to claim 2, wherein

the natural object assessment unit is further programmed to:

calculate a third pixel number by subtracting the second pixel number from the first pixel number;

calculate an abundance rate of the third pixel number relative to the first pixel number or the second pixel number; and

assess that the irregularity evaluation value is equal to or greater than the predetermined irregularity evaluation threshold when the abundance rate of the third pixel number relative to either the first pixel number or the second pixel number is equal to or greater than a predetermined rate.

23. The three-dimensional object detection device according to claim 11, wherein

the three-dimensional object detection unit detects a three-dimensional object based on the edge information and a second threshold β

; and

the control unit outputs to the three-dimensional object detection unit a control command for raising the second threshold β

so as to impede detection of the three-dimensional object when the three-dimensional object that was detected is assessed by the natural object assessment unit to be a natural object including plants or snow present along the lane traveled by the vehicle.

24. The three-dimensional object detection device according to claim 14, wherein

the three-dimensional object assessment unit adds a first counted value to calculate the evaluation value when the first integrated value is assessed to be greater than the second integrated value;

and

the three-dimensional object assessment unit sets a higher value for the first counted value in response to a greater number of continuous assessments when the assessment of the first integrated value being greater than the second integrated value continues.

25. The three-dimensional object detection device according to claim 14, wherein

the three-dimensional object assessment unit calculates the evaluation value by adding a first counted value when the first integrated value is assessed to be greater than the second integrated value, and subtracting a second counted value when the first integrated value is assessed to be less than the second integrated value;

and

the three-dimensional object assessment unit sets a higher value for the second counted value when an assessment is made that the first integrated value is greater than the second integrated value, an assessment is then made that the first integrated value is less than the second integrated value, and an assessment is thereafter made that the first integrated value is greater than the second integrated value.

26. The three-dimensional object detection device according to claim 14, wherein

the three-dimensional object assessment unit is further programmed to:

subtracts a second counted value to calculate the evaluation value when the first integrated value is assessed to be less than the second integrated value; and

sets a higher value for the second counted value when the assessment of the first integrated value being less than the second integrated value continues for at least a predetermined number of times.

18. A three-dimensional object detection method comprising:

capturing viewpoint images of a region rearward of a vehicle with an imaging device;

converting the viewpoint images to bird'"'"'s-eye view images;

creating differential waveform information by aligning the bird'"'"'s-eye view images of different times within a bird'"'"'s-eye view, and creating a frequency distribution of a number of pixels by counting the number of pixels representing a predetermined differential in a differential image of the bird'"'"'s-eye view images that were positionally aligned;

detecting three-dimensional objects which are included in the differential waveform information and which are present in detection areas set in left and right rear sides of the vehicle, based on the frequency of the number of pixels representing the predetermined differential in the differential image along a direction in which the three-dimensional object falls when the bird'"'"'s-eye view images are viewpoint-converted; and

calculating an irregularity evaluation value for evaluating an irregularity of the differential waveform information based on a first pixel number of first pixels representing a first predetermined differential in the differential image containing the three-dimensional object that was detected, and a second pixel number of second pixels that have been extracted along the direction in which the three-dimensional object falls when the bird'"'"'s-eye view an viewpoint-converted and that represent a second predetermined differential greater than the first predetermined differential in the differential image, and outputting a natural object assessment result based on the irregularity evaluation value to a control unit, the natural object assessment result indicating that the three-dimensional object that was detected is assessed to be a natural object including plants or snow present along the lane traveled by the vehicle when the calculated irregularity evaluation value is equal to or greater than a predetermined irregularity evaluation threshold set in advance.

19. A three-dimensional object detection method comprising:

capturing viewpoint images of a region rearward of a vehicle with an imaging device;

converting the viewpoint images to bird'"'"'s-eye view images;

creating edge information by extracting pixels in which a luminance difference of adjacent image areas is equal to or greater than a predetermined threshold in the bird'"'"'s-eye view images that were obtained;

detecting three-dimensional objects which are included in the edge information and which are present in detection areas set in left and right rear sides of the vehicle, based on the edge information including the pixels that are extracted along the direction in which the three-dimensional object falls when the bird'"'"'s-eye view image is viewpoint-converted and that have a luminance difference of adjacent image areas equal to or greater than the predetermined threshold; and

calculating an irregularity evaluation value for evaluating an irregularity of the edge information based on a first pixel number of first pixels in which the luminance difference of adjacent image areas in the bird'"'"'s-eye image containing the three-dimensional object that was detected is equal to or greater than a first predetermined threshold, and a second pixel number of second pixels that have been extracted along the direction in which the three-dimensional object falls when the bird'"'"'s-eye view images are is viewpoint-converted and in which the luminance difference of adjacent image areas in the bird'"'"'s-eye image is equal to or greater than a second predetermined threshold greater than the first predetermined threshold, and outputting a natural object assessment result based on the irregularity evaluation value to a control unit, the natural object assessment result indicating that the three-dimensional object that was detected is a natural object including plants or snow present along the lane traveled by the vehicle when the calculated irregularity evaluation value is equal to or greater than a predetermined irregularity evaluation threshold set in advance.