Water droplet detection device, and three-dimensional object detection device using water droplet detection device

• US 10,096,124 B2
• Filed: 07/24/2013
• Issued: 10/09/2018
• Est. Priority Date: 07/27/2012
• Status: Active Grant

• Current Assignee: Nissan Motor Co., Ltd.
• Original Assignee: Clarion Co., Ltd. (Faurecia SA), Nissan Motor Co., Ltd.
• Inventors: Hayakawa, Yasuhisa, Fukata, Osamu, Takemura, Masayuki, Utagawa, Akira, Muramatsu, Shoji, Irie, Kota
• Primary Examiner(s): An, Shawn S
• Application Number:
  US14/410,585
• Publication Number:
  US 20150325005A1
• Time in Patent Office: 1,903 Days
• CPC Class Codes: B60S 1/0844 including a camera
• CPC Class Codes: B60S 1/0848 Cleaning devices for cameras on vehicle
• CPC Class Codes: B60S 1/56 specially adapted for cleaning other parts or devices than front windows or windscreens rear-view mirror arrangements mounted on vehicle exterior including cleaning devices B60R1/0602
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/42 using transform domain methods
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection
• CPC Class Codes: H04N 25/625 for the control of smear
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A water droplet detection device has an image capturing unit and a water droplet detection unit. The image capturing unit has a photographic optical system that an area captures an image of a predetermined area. The water droplet detection unit sets an arbitrary attention point in the captured image, a plurality of first reference points inside an imaginary circle of a predetermined radius having the attention point as a center the imaginary circle, and a plurality of second reference points corresponding to the first reference points outside the imaginary circle. The water droplet detection unit detects edge information between the first reference points and second reference points, and assesses a circularity strength of the edge information to detect a water droplet attached to the photographic optical system. The water droplet detection device can be used with an image conversion unit to form a three-dimensional object detection device.

Claims

1. A water droplet detection device comprising:

an image capturing unit configured to capture an area an image of a predetermined area, the image capturing unit having a photographic optical system; and

a water droplet detection unit programmed to set an arbitrary attention point in the image obtained by the image capturing unit, a plurality of first reference points inside an imaginary circle of a predetermined radius having the attention point as a center the imaginary circle, and a plurality of second reference points corresponding to the first reference points outside the imaginary circle, each of the second reference points being set on a straight line connecting the attention point and the corresponding one of the first reference points, the water droplet detection unit being further programmed to detect edge information between the first reference points and second reference points, and the water droplet detection unit being further programmed to assess a circularity strength of the edge information, thereby detecting a water droplet attached to the photographic optical system,

the circularity strength being determined as a ratio of a number of pairs of the corresponding first and second reference points between which the edge information was detected to a total number of the corresponding first and second reference points, and

the water droplet attached to the photographic optical system being detected when the circularity strength is equal to or greater than a prescribed ratio.

2. The water droplet detection device according to claim 1, wherein

the water droplet detection unit is further programmed to set, as the first reference point, at least one of an upper center part, an upper left part, an upper right part, a lower left part, and a lower right part inside the imaginary circle.

3. The water droplet detection device according to claim 1, wherein

the water droplet detection is further programmed to assess that the circularity strength is greater when there is a higher proportion at which edge information is detected between the plurality of first reference points and the plurality of second reference points.

4. The water droplet detection device according to claim 1, wherein

the edge information is detected between a corresponding pair of the first and second reference points when a luminance difference equal to or greater than a prescribed value exists between the first reference point and the second reference point of the corresponding pair.

5. A three-dimensional object detection device comprising:

an image capturing unit configured to capture an image of a predetermined area, the image capturing unit having a photographic optical system;

an image conversion unit programmed to perform viewpoint conversion of the image obtained by the image capturing unit to a bird'"'"'s-eye-view image;

a water droplet detection unit programmed to set an arbitrary attention point in the image obtained by the image capturing unit, a plurality of first reference points inside an imaginary circle of a predetermined radius having the attention point as a center the imaginary circle, and a plurality of second reference points corresponding to the first reference points outside the imaginary circle, the water droplet detection unit being further programmed to detect edge information between the first reference points and second reference points, and the water droplet detection unit being further programmed to assess a circularity strength of the edge information, thereby detecting a water droplet attached to the photographic optical system;

a three-dimensional object detection unit programmed to detect a presence of the three-dimensional object based on the pixel distribution data by detecting information on pixel distribution in the bird'"'"'s-eye-view image obtained by the image conversion unit, for which the luminance difference is equal to or greater than a predetermined first threshold value in a direction in which a three-dimensional object collapses upon viewpoint conversion to the bird'"'"'s-eye-view image, and, detects the three-dimensional object when a degree of distribution of the pixels is equal to or greater than a predetermined second threshold value in the direction in which the three-dimensional object collapses;

a three-dimensional object assessment unit programmed to access whether or not a three-dimensional object in a detection area that has been detected by the three-dimensional object detection unit is another vehicle; and

a controller programmed to control the vehicle in accordance with a state of attachment of water droplets in a detection area detected by the water droplet detection unit.

6. The three-dimensional object detection device according to claim 5, further comprising

a water droplet removal device configured to remove water droplets that have attached to the photographic optical system,

the controller being programmed to operate the water droplet removal device in accordance with the state of water droplet attachment in the detection area detected by the water droplet detection unit.

7. The three-dimensional object detection device according to claim 6, wherein

the controller is further programmed to set a longer operating time for the water droplet removal device as the number of water droplets detected by the water droplet detection unit increases.

8. The three-dimensional object detection device according to claim 6, wherein

the controller is further programmed to operate the water droplet removal device when water droplets are detected in the predetermined area in the image.

9. The three-dimensional object detection device according to claim 6, wherein

the controller is further programmed to operate the water droplet removal device when the brightness of the environment is equal to or less than a predetermined value.

10. The three-dimensional object detection device according to claim 5, wherein

the controller is further programmed to suppress detection of a three-dimensional object by the three-dimensional object detection unit, or suppress assessment by the three-dimensional object assessment unit that the three-dimensional object is another vehicle, in accordance with the water droplet attachment state detected by the water droplet detection unit.

11. The three-dimensional object detection device according to claim 5, wherein

the water droplet detection unit is further programmed to assess that the circularity strength is greater when there is an increasing proportion at which edge information is detected between the plurality of first reference points and plurality of second reference points, and

the controller is further programmed to increase the detection time of the three-dimensional object detection unit longer correspondingly with respect to increased circularity strength detected by the water droplet detection unit.

12. The three-dimensional object detection device according to claim 5, wherein

the three-dimensional object detection unit comprises a first three-dimensional object detection unit programmed to detect a presence of the three-dimensional object based on differential waveform information by aligning bird'"'"'s-eye viewpoint positions of bird'"'"'s-eye-view images obtained at different points in time by the image conversion unit;

counting a number of pixels exhibiting a predetermined difference in a preset detection area in a predetermined area on a differential image of the bird'"'"'s-eye-view images that were aligned;

producing a frequency distribution to generate the differential waveform information;

the three-dimensional object detection unit detects the three-dimensional object when the differential waveform information is equal to or greater than a predetermined first threshold value α

; and

the controller is further programmed to;

generate a first control command for upwardly adjusting the predetermined first threshold value α

so that the three-dimensional object is not easily detected when water droplets have been detected by the water droplet detection unit, and

output the first control command to the first three-dimensional object detection unit.

13. The three-dimensional object detection device according to claim 12, wherein

the first three-dimensional object detection unit is programmed to detect the three-dimensional object when the differential waveform information is equal to or greater than the predetermined first threshold value α

, and

the controller is further programmed to;

generate a second control command to reduce the value obtained by counting the number of pixels exhibiting the predetermined difference in the differential image of the bird'"'"'s-eye-view images and producing the frequency distribution when water droplets have been detected by the water droplet detection unit; and

output the second control command to the first three-dimensional object detection unit.

14. The three-dimensional object detection device according to claim 12, wherein

the first three-dimensional object detection unit is programmed to extract the number of pixels exhibiting pixel values that are equal to or greater than a threshold value p as the number of pixels exhibiting the predetermined difference, and

the controller is further programmed to:

generate a second control command for upwardly adjusting the threshold value p so that the three-dimensional object is not easily detected when water droplets have been detected by the water droplet detection unit, and

output the second control command to the first three-dimensional object detection unit.

15. The three-dimensional object detection device according to claim 12, wherein

the first three-dimensional object detection unit is programmed to extract the number of pixels exhibiting pixel values that are equal to or greater than a threshold value p as the number of pixels exhibiting the predetermined difference, and

the controller is further programmed to:

generate a second control command for outputting a downwardly adjusted number of pixels that are extracted in the differential image along the direction in which the three-dimensional object collapses upon viewpoint conversion of the bird'"'"'s-eye-view image when water droplets have been detected by the water droplet detection unit, and

output the second control command to the first three-dimensional object detection unit.

16. The three-dimensional object detection device according to claim 5, wherein

the three-dimensional object detection unit comprises a second three-dimensional object detection unit programmed to detect a presence of the three-dimensional object based on edge information in a preset detection area within the predetermined area from the bird'"'"'s-eye-view images obtained by the image conversion unit,

the three-dimensional object detection unit programmed to extract edge lines based on the pixels exhibiting a luminance difference that is equal to or greater than a predetermined threshold value t;

and

the controller is further programmed to

generate a first control command for upwardly adjusting the predetermined threshold value t so that the three-dimensional object is not easily detected when water droplets have been detected by the water droplet detection unit, and

output the first control command to the second three-dimensional object detection unit.

17. The three-dimensional object detection device according to claim 16, wherein

the second three-dimensional object detection unit is programmed to extract edge lines based on pixels exhibiting the luminance difference that is equal to or greater than the predetermined threshold value t;

and

the controller is further programmed to

generate a second control command that reduces the luminance value of the pixels when water droplets have been detected by the water droplet detection unit, and

output the second control command to the second three-dimensional object detection unit.

18. The three-dimensional object detection device according to claim 16, wherein

the second three-dimensional object detection unit is programmed to detect a three-dimensional object based on edge lines having a length equal to or greater than a threshold value θ

contained in the edge information, and

the controller is further programmed to generate a control command for upwardly adjusting the threshold value θ

so that the three-dimensional object is not easily detected when water droplets have been detected by the water droplet detection unit, and outputs the control command to the second three-dimensional object detection unit.

19. The three-dimensional object detection device according to claim 16, wherein

the second three-dimensional object detection unit is programmed to detect a three-dimensional object based on edge lines having a length equal to or greater than a threshold value θ

contained in the edge information, and

the controller is further programmed to;

generate a second control command for outputting a downwardly adjusted value for the length of the edges contained in the detected edge information when water droplets have been detected by the water droplet detection unit, and

output the second control command to the second three-dimensional object detection unit.

20. The three-dimensional object detection device according to claim 16, wherein

the second three-dimensional object detection unit is programmed to detect the three-dimensional object based on assessing whether the number of edge lines that are equal to or greater than a predetermined length contained in the edge information is equal to or greater than a second threshold value β

, and

the controller is further programmed to

generate a second control command for upwardly adjusting the second threshold value β

so that the three-dimensional object is not readily detected when water droplets have been detected by the water droplet detection unit, and

output the second control command to the second three-dimensional object detection unit.

21. The three-dimensional object detection device according to claim 16, wherein

the second three-dimensional object detection unit is programmed to detect the three-dimensional object based on assessing whether the number of edge lines that are equal to or greater than a predetermined length contained in the edge information is equal to or greater than a second threshold value β

, and

the controller is further programmed to

generate a second control command for outputting a downwardly adjusted number of detected edge lines that are equal to or greater than the predetermined length when water droplets have been detected by the water droplet detection unit, and

output the second control command to the second three-dimensional object detection unit.

22. The three-dimensional object detection device according to claim 5, wherein

the controller is further programmed to assess that the three-dimensional object is another vehicle when water droplets have been detected in the predetermined area in the image.

23. The three-dimensional object detection device according to claim 5, wherein

the controller is further programmed to increase a degree of suppression of the assessment that the three-dimensional object is another vehicle in correspondence to an increase in the number of water droplets detected by the water droplet detection unit.

24. The three-dimensional object detection device according to claim 5, wherein

the controller is further programmed to suppress the assessment that the three-dimensional object is another vehicle when a brightness of an environment is equal to or less than a predetermined value.

25. The three-dimensional object detection device according to claim 5, wherein

the three-dimensional object assessment unit is further programmed to assess that the three-dimensional object is another vehicle when a traveling speed of the three-dimensional object that has been detected is equal to or greater than a preset predetermined value,

the controller, when water droplets have been detected by the water droplet detection unit, is further programmed to execute out at least one of the following:

a) generating a first control command for increasing a predetermined speed that serves as a lower limit when an assessment is made that the three-dimensional object is another vehicle, and outputs the first control command to the three-dimensional object assessment unit,

b) generating a second control command for outputting a downwardly adjusted a three-dimensional object traveling speed that is compared with the predetermined speed that serves as the lower limit when an assessment is made that the three-dimensional object is another vehicle and outputs the second control command to the three-dimensional object assessment unit,

c) generating a third control command for downwardly adjusting a predetermined speed that serves as an upper limit when an assessment is made that the three-dimensional object is another vehicle, and outputs the third control command to the three-dimensional object assessment unit, and

d) generating a fourth control command for upwardly adjusting the traveling speed of the three-dimensional object that is compared with the predetermined speed that serves as the upper limit when an assessment is made that the three-dimensional object is another vehicle,

and outputs the fourth control command to the three-dimensional object assessment unit.

26. A water droplet detection method for detecting water droplets attached to a photographic optical system of an image capturing unit comprising a photographic optical system, the water droplet detection method comprising:

setting an arbitrary attention point in the captured image that has been acquired by the image capturing unit, a plurality of first reference points inside an imaginary circle of a predetermined radius having the attention point as a center the imaginary circle, and a plurality of second reference points corresponding to the first reference points outside the imaginary circle, each of the second reference points being set on a straight line connecting the attention point and the corresponding one of the first reference points;

detecting edge information between the first reference points and the second reference points;

determining a circularity strength of the edge information as a ratio of a number of pairs of the corresponding first and second reference points between which the edge information was detected to a total number of the corresponding first and second reference points;

assessing that a water droplet is attached to the photographic optical system at a location corresponding to the imaginary circle when the circularity strength is equal to or greater than a prescribed ratio.

27. The water droplet detection method according to claim 26, wherein

the edge information is detected between a corresponding pair of the first and second reference points when a luminance difference equal to or greater than a prescribed value exists between the first reference point and the second reference point of the corresponding pair.