Systems, methods and apparatuses are provided for enhanced surface condition detection based on image scene and ambient light analysis

• US 10,678,255 B2
• Filed: 02/14/2018
• Issued: 06/09/2020
• Est. Priority Date: 02/14/2018
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Zhao, Qingrong, Khalili, Mohsen, Wang, Jinsong, Litkouhi, Bakhtiar B.
• Primary Examiner(s): Rodden, Joshua E
• Application Number:
  US15/896,104
• Publication Number:
  US 20190250630A1
• Time in Patent Office: 846 Days
• Field of Search: 701 28
• CPC Class Codes: G05D 1/0238 using obstacle or wall sensors G05D1/0246 and G05D1/0289 take precedence; lidar systems designed for anti-collision purposes G01S17/93
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

Systems, Methods and Apparatuses are provided for detecting surface conditions, which includes: an image scene captured by a camera wherein the image scene includes: a set of a plurality of regions of interest (ROIs); and a processor configured to receive the image scene to: extract at least a first and a second ROI from the set of the plurality of ROIs of the image scene; associate the first ROI with an above-horizon region and associate the second ROI with a surface region; analyze the first ROI and the second ROI in parallel for a condition related to an ambient lighting in the first ROI and for an effect related to the ambient lighting in the second ROI; and extract from the first ROI features of the condition of the ambient lighting and extract from the second ROI features of the effect of the ambient lighting on a surface region.

Claims

1. A surface detection system for detecting surface conditions comprising:

an image scene captured by a camera wherein the image scene comprises;

a set of a plurality of regions of interest (ROIs); and

a processor configured to receive the image scene to;

extract at least a first and a second ROI from the set of the plurality of ROIs of the image scene;

associate the first ROI with an above-horizon region and associate the second ROI with a surface region;

analyze the first ROI and the second ROI in parallel for a condition related to an ambient lighting in the first ROI and for an effect related to the ambient lighting in the second ROI; and

extract from the first ROI features of the condition of the ambient lighting and extract from the second ROI features of the effect of the ambient lighting on a surface region.

2. The surface detection system of claim 1, further comprising:

analyzing the second ROI to extract the features of the effects of the ambient lighting on the surface region in common with the features of the first ROI of the conditions of ambient lighting based on a plurality of image attributes wherein a set of extracted features from the first and second ROIs comprise;

a common set of features for both ROIs.

3. The surface detection system of claim 2, wherein the image attributes comprise:

a set of one or more of a texture, a brightness and a color information.

4. The surface detection system of claim 3, further comprising:

generating the common set of features for both the ROIs based on combining the plurality of image attributes of the second ROI with that of the first ROI.

5. The surface detection system of claim 4, further comprising:

classifying by a classifier each of the common set of features to a particular surface type.

6. The surface detection system of claim 1, further comprising:

an autonomous, semi-autonomous, or non-autonomous vehicle system, or non-vehicular applications.

7. The surface detection system of claim 1, wherein

the processor is configured to:

determine, by thresholds related to values of pixels of the first ROI, features of the condition of the ambient lighting and, by thresholds related to values of pixels of the second ROI, features of the effect of the ambient lighting on the surface region.

8. A method for detecting a surface condition, the method comprising:

capturing an image scene by a camera wherein the image scene comprises;

a set of a plurality of regions of interest (ROIs);

extracting, by a processor associated with the vehicle, at least a first and a second ROI from the set of the plurality of ROIs of the image scene;

associating, by the processor, the first ROI with an above-horizon region and associating the second ROI with a surface region;

analyzing, by the processor, the first ROI and the second ROI in parallel for a condition related to an ambient lighting in the first ROI and for an effect related to the ambient lighting in the second ROI; and

extracting, by the processor from the first ROI, features of the condition of the ambient lighting and extracting, from the second ROI, features of the effect of the ambient lighting on a surface region.

9. The method of claim 8, further comprising:

an autonomous, semi-autonomous, non-autonomous driving method, or non-driving use.

10. The method of claim 8, further comprising:

taking an action by the processor based on a feature of the surface condition or the effect on a surface related to vehicle control.

11. The method of claim 8, further comprising:

taking an action by the processor based on the surface condition or the effect on a surface of sending a notification for notifying of a particular surface condition or effect on the surface.

12. The method of claim, 8, further comprising:

analyzing, by the processor, the second ROI to extract the features of the effects of the ambient lighting on the surface region in common with the features of the first ROI of the conditions of ambient lighting based on a plurality of image attributes wherein a set of extracted features from the first and second ROIs comprise;

a common set of features for both ROIs.

13. The method of claim 12, wherein the image attributes comprise:

a set of one or more of a texture, a brightness and a color information.

14. The method of claim 13, further comprising:

generating, by the processor, the common set of features for both the ROIs based on combining the plurality of image attributes of the second ROI with that of the first ROI.

15. A surface detection apparatus, comprising:

a camera for generating images wherein the images comprise;

regions of interest (ROI);

a processor, configured to receive the images to;

extract at least a first and a second ROI from the set of the plurality of ROIs of the image scene;

associate the first ROI with an above-horizon region and associate the second ROI with a surface region;

analyze the first ROI and the second ROI in parallel for a condition related to an ambient lighting in the first ROI and for an effect related to the ambient lighting in the second ROI; and

extract from the first ROI features of the condition of the ambient lighting and extract from the second ROI features of the effect of the ambient lighting on a surface region.

16. The apparatus of claim 15, further comprising:

an autonomous, semi-autonomous or conventional driving apparatus.

17. The apparatus of claim 15, further comprising:

a cloud connected apparatus to send messages of surface conditions to third party networks.

18. The apparatus of claim 15, further comprising:

the processor is configured to;

determine, by thresholds related to values of pixels of the first ROI, features of the condition of the ambient lighting and, by thresholds related to values of pixels of the second ROI, features of the effect of the ambient lighting on the surface region.

19. The apparatus of claim 15, further comprising:

the processor is configured to;

analyze the second ROI to extract the features of the effects of the ambient lighting on the surface region in common with the features of the first ROI of the conditions of ambient lighting based on a plurality of image attributes wherein a set of extracted features from the first and second ROIs comprise;

a common set of features for both ROIs;

generate the common set of features for both the ROIs based on combining the plurality of image attributes of the second ROI with that of the first ROI; and

classify, by a classifier, each of the common set of features to a particular surface type.