Monitoring ambient light for object detection

• US 10,803,743 B2
• Filed: 05/02/2018
• Issued: 10/13/2020
• Est. Priority Date: 05/02/2018
• Status: Active Grant

• Current Assignee: Magna Autonomous Systems LLC, Woven By Toyota US Incorporated
• Original Assignee: LYFT Incorporated
• Inventors: Abari, Farzad Cyrus Foroughi, Bangs, Corey Frederick
• Primary Examiner(s): Mistry, Oneal R
• Application Number:
  US15/969,693
• Publication Number:
  US 20190340924A1
• Time in Patent Office: 895 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/241 relating to the classification model, e.g. parametric or non-parametric approaches
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: G08G 1/09623 Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle G08G1/0967 takes precedence
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

In particular embodiments, a computing system may receive ambient light measurements of a scene. The ambient light measurements may be generated by one or more light sensors of a vehicle. The system may receive an image captured by an optical sensor of the vehicle. The image may include color information associated with an object in a scene. The system may process the image to estimate a color associated with the object based on the ambient light measurements and the color information in the image. The system may determine the object based on the estimated color and may provide instructions to activate a safety feature in response to the determined object.

Claims

1. A method comprising, by a computing system:

receiving ambient light measurements of ambient light of a scene from one or more light sensors of a vehicle, the ambient light measurements comprising one or more color wavelength ranges of the ambient light and an intensity of the one or more color wavelength ranges of the ambient light;

receiving an image captured by an optical sensor of the vehicle, the image comprising perceived color information associated with an object in the scene, wherein the perceived color information includes at least one color wavelength range associated with the image of the object;

comparing the one or more color wavelength ranges of the ambient light with the at least one color wavelength range associated with the image of the object;

based on the comparing, estimating a color associated with the object by using the intensity of the one or more color wavelength ranges of the ambient light and the at least one color wavelength range of the perceived color information associated with the image of the object;

determining the object based at least in part on the estimated color; and

providing instructions to perform a vehicle action in response to the determined object.

2. The method of claim 1, wherein:

the perceived color information further comprises a color intensity of each of red, green, and blue channels of the at least one color wavelength range associated with the image of the object; and

the ambient light measurements further comprise a color intensity of each of red, green, and blue channels of the one or more color wavelength ranges of the ambient light.

3. The method of claim 2, wherein comparing the one or more color wavelength ranges of the ambient light with the at least one color wavelength range associated with the image of the object comprises:

comparing the color intensity of the red, green, and blue channels of the at least one color wavelength range associated with the image of the object and the color intensity of the red, green, and blue channels of the one or more color wavelength ranges of the ambient light.

4. The method of claim 1, wherein estimating the color associated with the object and determining the object based at least in part on the estimated color are performed by a machine-learning model.

5. The method of claim 4, wherein the machine-learning model is trained based on a set of training samples, each of the training samples comprising:

a raw image of a sample visual scene including a sample object, wherein perceived color information of the sample object is a result of a true color of the sample object and ambient light reflected or bounced off from the sample object;

ambient light measurements for the sample visual scene; and

a corrected image of the sample visual scene, the corrected image representing the true color of the sample object.

6. The method of claim 1, wherein the one or more color wavelength ranges of the ambient light correspond to the at least one color wavelength range associated with the image of the object.

7. The method of claim 1, wherein the vehicle action comprises performing at least one of:

activating a safety feature of the vehicle;

adjusting a display setting of a display device within the vehicle based on the ambient light measurements;

or

generating an alert associated with the determined object.

8. The method of claim 1, wherein estimating the color associated with the object includes correcting the perceived color information.

9. The method of claim 1, wherein estimating the color associated with the object by using the intensity of the one or more color wavelength ranges of the ambient light and the at least one color wavelength range of the perceived color information associated with the image of the object comprises:

deconvolving the one or more color wavelength ranges of the ambient light from the at least one color wavelength range of the perceived color information associated with the image of the object.

10. The method of claim 9, wherein deconvolving the one or more color wavelength ranges of the ambient light from the at least one color wavelength range of the perceived color information associated with the image of the object comprises subtracting the intensity of the one or more color wavelength ranges of the ambient light from the at least one color wavelength range of the perceived color information associated with the image of the object.

11. The method of claim 1, wherein:

the ambient light measurements comprise a plurality of color wavelength ranges of the ambient light of the scene;

the perceived color information comprises a plurality of color wavelength ranges associated with the image of the object; and

comparing the one or more color wavelength ranges of the ambient light with the at least one color wavelength range associated with the image of the object comprises;

comparing an intensity of each wavelength of the plurality of color wavelength ranges associated with the image of the object with an intensity of a corresponding color wavelength range of the plurality of color wavelength ranges of the ambient light of the scene.

12. The method of claim 1, wherein comparing the one or more color wavelength ranges of the ambient light with the at least one color wavelength range associated with the image of the object comprises:

determining one or more color wavelength ranges of the ambient light that correspond with the at least one color wavelength range associated with the image of the object.

13. The method of claim 1, further comprising:

determining true red, green, and blue components for another color wavelength range associated with the image of the object after comparing the one or more color wavelength ranges of the ambient light with the at least one color wavelength range associated with the image of the object;

comparing the other color wavelength range associated with the image of the object with a corresponding wavelength range of the ambient light; and

further estimating the color associated with the object based on the comparison of the other color wavelength range associated with the image of the object with the corresponding wavelength range of the ambient light.

14. The method of claim 1, wherein the perceived color information is a result of at least the ambient light of the scene and reflectance properties of the object in the scene.

15. A system comprising:

one or more processors and one or more computer-readable non-transitory storage media coupled to the one or more processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by the one or more processors to cause the system to perform operations comprising;

receiving ambient light measurements of ambient light of a scene from one or more light sensors of a vehicle, the ambient light measurements comprising one or more color wavelength ranges of the ambient light and an intensity of the one or more color wavelength ranges of the ambient light;

receiving an image captured by an optical sensor of the vehicle, the image comprising perceived color information associated with an object in the scene, wherein the perceived color information includes at least one color wavelength range associated with the image of the object;

comparing the one or more color wavelength ranges of the ambient light with the at least one color wavelength range associated with the image of the object;

based on the comparing, estimating a color associated with the object by using the intensity of the one or more color wavelength ranges of the ambient light and the at least one color wavelength range of the perceived color information associated with the image of the object;

determining the object based at least in part on the estimated color; and

providing instructions to perform a vehicle action in response to the determined object.

16. The system of claim 15, wherein estimating the color associated with the object and determining the object based at least in part on the estimated color are performed by a machine-learning model.

17. The system of claim 16, wherein the machine-learning model is trained based on a set of training samples, wherein at least one training sample in the set of training samples comprises:

a corrected image whose perceived color information has been corrected based on detected ambient light; and

at least one identification of an object depicted in the corrected image.

18. The system of claim 15, wherein comparing the one or more color wavelength ranges of the ambient light with the at least one color wavelength range associated with the image of the object comprises:

determining one or more color wavelength ranges of the ambient light that correspond with the at least one color wavelength range associated with the image of the object.

19. One or more computer-readable non-transitory storage media embodying software that is operable when executed to cause one or more processors to perform operations comprising:

receiving ambient light measurements of ambient light of a scene from one or more light sensors of a vehicle, the ambient light measurements comprising one or more color wavelength ranges of the ambient light and an intensity of the one or more color wavelength ranges of the ambient light;

receiving an image captured by an optical sensor of the vehicle, the image comprising perceived color information associated with an object in the scene, wherein the perceived color information includes at least one color wavelength range associated with the image of the object;

comparing the one or more color wavelength ranges of the ambient light with the at least one color wavelength range associated with the image of the object;

based on the comparing, estimating a color associated with the object by using the intensity of the one or more color wavelength ranges of the ambient light and the at least one color wavelength range of the perceived color information associated with the image of the object;

determining the object based at least in part on the estimated color; and

providing instructions to perform a vehicle action in response to the determined object.

20. The one or more computer-readable non-transitory storage media of claim 19, wherein estimating the color associated with the object and determining the object based at least in part on the estimated color are performed by a machine-learning model.