Environmental scene condition detection

• US 9,734,425 B2
• Filed: 02/11/2015
• Issued: 08/15/2017
• Est. Priority Date: 02/11/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Xu, Shili, Biswas, Mainak, Zhong, Xin, Guo, Feng, Ren, Jianfeng, Yang, Yang, Gao, Dashan, Yang, Ruiduo
• Primary Examiner(s): Harandi, Siamak
• Application Number:
  US14/619,354
• Publication Number:
  US 20160232423A1
• Time in Patent Office: 916 Days
• CPC Class Codes: B60H 1/00735 Control systems or circuits characterised by their input, i.e. by the detection, measurement or calculation of particular conditions, e.g. signal treatment, dynamic models
• CPC Class Codes: B60Q 1/20 Fog lights
• CPC Class Codes: B60S 1/0844 including a camera
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A method of processing data includes receiving, at a computing device, data representative of an image captured by an image sensor. The method also includes determining a first scene clarity score. The method further includes determining whether the first scene clarity score satisfies a threshold, and if the first scene clarity score satisfies the threshold, determining a second scene clarity score based on second data extracted from the data.

Claims

1. A method for processing data at a computing device associated with a vehicle, the method comprising:

receiving, by the computing device, the data representative of at least a portion of an image captured by an image sensor, the image depicting a scene;

calculating, by the computing device, a first scene clarity score based on first data extracted from the data;

performing, by the computing device, a comparison based on the first scene clarity score and a threshold;

in response to a result of the comparison, calculating, by the computing device, a second scene clarity score based on second data extracted from the data, the second data including the first data and additional data; and

initiating, by the computing device, an action by the vehicle in response to an identification of an environmental condition based on the second scene clarity score.

2. The method of claim 1, further comprising:

extracting the first data and the second data from the data; and

determining whether the scene includes the environmental condition based on the second scene clarity score.

3. The method of claim 1, wherein the additional data includes gradient feature data.

4. The method of claim 1, wherein initiating the action comprises activating a heating, ventilation, and air conditioning (HVAC) system of the vehicle, and further comprising determining whether the environmental condition is present on a window of the vehicle.

5. The method of claim 1, wherein initiating the action comprises activating a windshield wiper of the vehicle.

6. The method of claim 1, wherein the second scene clarity score is calculated in response to the first scene clarity score exceeding the threshold, and wherein the first scene clarity score and the second scene clarity score are indicative of contrast in the image.

7. The method of claim 1, wherein initiating the action includes activating a fog light of the vehicle, providing an audio warning, providing a visual warning, or a combination thereof.

8. The method of claim 1, wherein the action includes processing the data to produce clarified image data.

9. The method of claim 8, wherein producing the clarified image data includes using local tone mapping (LTM) to boost local contrast in high intensity regions of the image.

10. The method of claim 8, wherein intensity values of one or more pixels in the image are modified by amounts selected based on one or both of the first scene clarity score and the second scene clarity score.

11. The method of claim 8, further comprising performing object detection based on the clarified image data and providing an output to an output device of a vehicle based on the object detection.

12. The method of claim 1, wherein the first data includes dark feature data that corresponds to pixels in the image that have a high intensity value in at least one of a red channel, a green channel, and a blue channel of the image.

13. The method of claim 1, wherein calculating the first scene clarity score includes comparing the first data to a model generated using machine learning.

14. The method of claim 1, further comprising:

comparing the second scene clarity score to a second threshold; and

identifying the environmental condition based on the second scene clarity score satisfying the second threshold.

15. The method of claim 1, wherein the image is included in a plurality of images received from the image sensor, and further comprising selecting a particular image of the plurality of images and selectively processing the particular image to calculate a corresponding scene clarity score.

16. An apparatus comprising:

a memory; and

a processor coupled to the memory and configured to;

receive data representative of at least a portion of an image captured by an image sensor, the image depicting a scene associated with a vehicle;

calculate a first scene clarity score of the image based on first data extracted from the data;

perform a comparison based on the first scene clarity score and a threshold;

in response to a result of the comparison, calculate a second scene clarity score based on second data extracted from the data, the second data including the first data and additional data; and

in response to an identification of an environmental condition based on the second scene clarity score, initiate an action by the vehicle.

17. The apparatus of claim 16, further comprising the image sensor, wherein the image sensor is located within a passenger compartment of the vehicle.

18. The apparatus of claim 16, wherein the processor includes a local tone mapping (LTM) module, and wherein the action includes using the LTM module to generate clarified image data.

19. The apparatus of claim 18, wherein the LTM module is configured to generate the clarified image data based on second data representative of at least a portion of a second image, the second image captured by the image sensor after the image sensor captured the image.

20. The apparatus of claim 18, wherein the LTM module is configured to modify intensity values of one or more pixels in the image by amounts selected based on one or both of the first scene clarity score and the second scene clarity score, and wherein the amounts are positively correlated with one or both of the first scene clarity score and the second scene clarity score.

21. The apparatus of claim 16, further comprising a body control unit configured to control one or more electronic systems of the vehicle, wherein the memory is configured to store the data.

22. The apparatus of claim 16, further comprising a speaker configured to output an audio warning associated with the action.

23. The apparatus of claim 16, wherein the processor is further configured to perform the action to generate clarified image data, and further comprising a display device onboard the vehicle, the display device configured to display a clarified image represented by the clarified image data.

24. The apparatus of claim 16, wherein the processor is further configured to perform the action to generate clarified image data and to detect an objects based on the clarified image data.

25. The apparatus of claim 16, wherein the action includes activating a heating, ventilation, and air conditioning (HVAC) system, activating a windshield wiper, activating a fog light of a vehicle, providing an audio warning, providing a visual warning, or a combination thereof.

26. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations including:

receiving data representative of at least a portion of an image captured by an image sensor, the image depicting a scene associated with a vehicle;

calculating a first scene clarity score based on first data extracted from the data;

performing a comparison based on the first scene clarity score and a threshold; and

in response to a result of the comparison;

calculating a second scene clarity score based on second data extracted from the data, the second data including the first data and additional data; and

in response to an identification of an environmental condition based on the second scene clarity score, initiating performance of an action by the vehicle.

27. The non-transitory computer-readable storage medium of claim 26, wherein the first scene clarity score is calculated by comparing a first linear support vector model to the first data, and wherein the second scene clarity score is calculated by comparing a second linear support vector model to the second data.

28. The non-transitory computer-readable storage medium of claim 27, wherein each of the first linear support vector model and the second linear support vector model is generated using sample videos.

29. An apparatus comprising:

means for receiving data representative of at least a portion of an image; and

means for image processing configured to;

calculate a first scene clarity score of the image based on first data extracted from the data, the first scene clarity score indicative of contrast in the image;

perform a comparison based on the first scene clarity score and a threshold; and

in response to a result of the comparison;

calculate a second scene clarity score based on second data extracted from the data, the second data including the first data and additional data; and

initiate an action by a vehicle based on the second scene clarity score.

30. The apparatus of claim 29, wherein the action includes processing the data to produce clarified image data, activating a fog light of a vehicle, providing an audio warning, providing a visual warning, or a combination thereof.

31. A method for processing data at a computing device associated with a vehicle, the method comprising:

receiving, by the computing device, the data representative of at least a portion of an image captured by an image sensor, the image depicting a scene;

calculating, by the computing device, a first scene clarity score based on first data extracted from the data;

performing, by the computing device, a comparison based on the first scene clarity score and a threshold;

in response to a result of the comparison, calculating, by the computing device, a second scene clarity score based on second data extracted from the data, the second data including the first data and additional data; and

initiating, by the computing device and based on the second scene clarity score, processing the data to produce clarified image data.

32. The method of claim 31, further comprising processing, by the computing device, the data to produce the clarified image data.

33. The method of claim 31, wherein the computing device comprises a wearable computing device.

34. The method of claim 31, wherein the computing device is incorporated in the vehicle.

35. An apparatus comprising:

a memory; and

a processor coupled to the memory and configured to;

receive data representative of at least a portion of an image captured by an image sensor, the image depicting a scene associated with a vehicle;

calculate a first scene clarity score of the image based on first data extracted from the data;

perform a comparison based on the first scene clarity score and a threshold;

in response to a result of the comparison, calculate a second scene clarity score based on second data extracted from the data, the second data including the first data and additional data; and

initiate, based on the second scene clarity score, processing the data to produce clarified image data.

36. The apparatus of claim 35, wherein the processor is further configured to process the data to produce the clarified image data.

37. The apparatus of claim 35, wherein the memory and the processor are incorporated in a wearable computing device.

38. The apparatus of claim 35, wherein the memory and the processor are incorporated in the vehicle.