Real-time road flare detection using templates and appropriate color spaces

• US 9,286,520 B1
• Filed: 07/16/2013
• Issued: 03/15/2016
• Est. Priority Date: 07/16/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lo, Wan-Yen, Ferguson, David Ian Franklin, Ogale, Abhijit
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Beck, Leron
• Application Number:
  US13/942,747
• Time in Patent Office: 973 Days
• Field of Search: 348/113
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/126 Evolutionary algorithms, e.g. genetic algorithms or genetic programming
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness

Abstract

Methods and systems for real-time road flare detection using templates and appropriate color spaces are described. A computing device of a vehicle may be configured to receive an image of an environment of the vehicle. The computing device may be configured to identify a given pixels in the plurality of pixels having one or more of: (i) a red color value greater than a green color value, and (ii) the red color value greater than a blue color value. Further, the computing device may be configured to make a comparison between one or more characteristics of a shape of an object represented by the given pixels in the image and corresponding one or more characteristics of a predetermined shape of a road flare; and determine a likelihood that the object represents the road flare.

Claims

1. A method comprising:

receiving, at a computing device of a vehicle, an image of an environment of the vehicle, wherein the image includes a plurality of pixels having respective red color values, green color values, and blue color values;

identifying given pixels in the plurality of pixels having one or more of;

(i) a red color value greater than a green color value, and (ii) the red color value greater than a blue color value;

making a comparison between one or more characteristics of a shape of an object represented by the given pixels in the image and corresponding one or more characteristics of a predetermined shape of a road flare;

determining one or more motion characteristics of the object represented by the given pixels;

determining, using the computing device, a likelihood that the object represents the road flare based at least on (i) the comparison, and (ii) the one or more motion characteristics of the object, such that the likelihood is increased when the one or more motion characteristics indicate that the object remains stationary;

modifying, using the computing device, a control strategy associated with a driving behavior of the vehicle, based on the likelihood; and

providing, using the computing device, instructions to control the vehicle based on the modified control strategy.

2. The method of claim 1, wherein the computing device is configured to control the vehicle in an autonomous operation mode.

3. The method of claim 1, wherein identifying the given pixels comprises:

converting the image into a monochrome space image, wherein in the monochrome space image;

any pixel having a difference between a red color value and a green color value less than zero, or a difference between the red color value and a blue color value less than zero, is assigned a zero intensity value, and

remaining pixels are assigned an intensity value equal to a maximum of the difference between the red color value and the green color, or the difference between the red color value and the blue color.

4. The method of claim 1, identifying the given pixels comprises:

making a comparison of the red color value of the given pixels to a predetermined red color value threshold,

wherein determining the likelihood comprises determining the likelihood further based on an output of the comparison of the red color value of the given pixels to the predetermined red color value threshold.

5. The method of claim 1, wherein making the comparison between the one or more characteristics of the shape of the object represented by the given pixels and the corresponding one or more characteristics of the predetermined shape of the road flare comprises:

applying a predetermined shape template, associated with the road flare, to the object to determine whether one or more pixels surrounding the object have respective color values different from predetermined color values of the road flare.

6. The method of claim 1, further comprising:

determining a first likelihood that the object represents the road flare based on the red color value of the given pixels; and

determining a second likelihood that the object represents the road flare based on an output of the comparison between the one or more characteristics of the shape of the object and the corresponding one or more characteristics of the predetermined shape of the road flare,

wherein determining the likelihood that the object represents the road flare is based on a combination of the first likelihood and the second likelihood.

7. The method of claim 6, further comprising:

determining a third likelihood that the object represents the road flare based on one or more motion characteristics of the object,

wherein determining the likelihood comprises determining the likelihood based on a combination of the first likelihood, the second likelihood, and the third likelihood.

8. The method of claim 1, further comprising:

detecting, using the computing device, based on the road flare, a hazard condition in the environment of the vehicle, wherein modifying the control strategy is based on the detected hazard condition.

9. The method of claim 1, wherein providing instructions to control the vehicle comprises providing instructions for one or more of:

(i) utilizing sensor information received from on-board or off-board sensors in making a navigation decision rather than preexisting map information, (ii) utilizing the sensor information to estimate lane boundaries rather than the preexisting map information, (iii) determining locations of hazard condition markers rather than lane markers on a road to estimate and follow the lane boundaries, (iv) activating one or more sensors for detection of humans and making the navigation decision based on the detection, (v) following another vehicle, (vi) maintaining a predetermined safe distance with other vehicles, (vii) turning-on lights, and (viii) reducing a speed of the vehicle.

10. A non-transitory computer readable medium having stored thereon instructions that, when executed by a computing device of a vehicle, cause the computing device to perform functions comprising:

receiving an image of an environment of the vehicle, wherein the image includes a plurality of pixels having respective red color values, green color values, and blue color values;

identifying given pixels in the plurality of pixels having one or more of;

(i) a red color value greater than a green color value, and (ii) the red color value greater than a blue color value;

making a comparison between one or more characteristics of a shape of an object represented by the given pixels in the image and corresponding one or more characteristics of a predetermined shape of a road flare;

determining one or more motion characteristics of the object represented by the given pixels;

determining a likelihood that the object represents the road flare based at least on (i) the comparison, and (ii) the one or more motion characteristics of the object, such that the likelihood is increased when the one or more motion characteristics indicate that the object remains stationary;

modifying a control strategy associated with a driving behavior of the vehicle, based on the likelihood; and

providing instructions to control the vehicle based on the modified control strategy.

11. The non-transitory computer readable medium of claim 10, wherein the function of identifying the given pixels comprises:

assigning to any pixel having a difference between a red color value and a green color value less than zero, or a difference between the red color value and a blue color value less than zero, a zero intensity value; and

assigning to remaining pixels an intensity value equal to a maximum of the difference between the red color value and the green color, or the difference between the red color value and the blue color.

12. The non-transitory computer readable medium of claim 10, wherein the function of making the comparison between the one or more characteristics of the shape of the object represented by the given pixels and the corresponding one or more characteristics of the predetermined shape of the road flare comprises:

applying a predetermined shape template, associated with the road flare, to the object to determine whether one or more pixels surrounding the object have respective color values different from predetermined color values of the road flare.

13. The non-transitory computer readable medium of claim 10, wherein the function of providing instructions to control comprises providing instruction for one or more of:

(i) utilizing sensor information received from on-board or off-board sensors in making a navigation decision rather than preexisting map information, (ii) utilizing the sensor information to estimate lane boundaries rather than the preexisting map information, (iii) determining locations of hazard condition markers rather than lane markers on a road to estimate and follow the lane boundaries, (iv) activating one or more sensors for detection of humans and making the navigation decision based on the detection, (v) following another vehicle, (vi) maintaining a predetermined safe distance with other vehicles, (vii) turning-on lights, and (viii) reducing a speed of the vehicle.

14. A system, comprising:

at least one processor; and

data storage comprising instructions that, when executed by the at least one processor, cause the system to perform functions comprising;

receiving an image of an environment of a vehicle, wherein the image includes a plurality of pixels having respective red color values, green color values, and blue color values;

identifying given pixels in the plurality of pixels having one or more of;

(i) a red color value greater than a green color value, and (ii) the red color value greater than a blue color value;

making a comparison between one or more characteristics of a shape of an object represented by the given pixels in the image and corresponding one or more characteristics of a predetermined shape of a road flare;

determining one or more motion characteristics of the object represented by the given pixels;

determining a likelihood that the object represents the road flare based at least on (i) the comparison, and (ii) the one or more motion characteristics of the object, such that the likelihood is increased when the one or more motion characteristics indicate that the object remains stationary;

modifying a control strategy associated with a driving behavior of the vehicle, based on the likelihood; and

providing instructions to control the vehicle based on the modified control strategy.

15. The system of claim 14, wherein the functions further comprise:

determining a first likelihood that the object represents the road flare based on the red color value of the given pixels; and

determining a second likelihood that the object represents the road flare based on an output of the comparison between the one or more characteristics of the shape of the object and the corresponding one or more characteristics of the predetermined shape of the road flare,

wherein determining the likelihood that the object represents the road flare is based on a combination of the first likelihood and the second likelihood.

16. The system of claim 15, wherein the functions further comprise:

determining a third likelihood that the object represents the road flare based on one or more motion characteristics of the object,

wherein determining the likelihood comprises determining the likelihood based on a combination of the first likelihood, the second likelihood, and the third likelihood.

17. The system of claim 14, wherein the function of determining the likelihood comprises determining the likelihood in real-time.