MONITORING AMBIENT LIGHT FOR OBJECT DETECTION

• US 20190340924A1
• Filed: 05/02/2018
• Published: 11/07/2019
• Est. Priority Date: 05/02/2018
• Status: Active Grant

• Current Assignee: Magna Autonomous Systems LLC, Woven By Toyota US Incorporated
• Original Assignee: LYFT Incorporated
• Inventors: Abari, Farzad Cyrus Foroughi, Bangs, Corey Frederick
• Granted Patent: US 10,803,743 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/241 relating to the classification model, e.g. parametric or non-parametric approaches
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: G08G 1/09623 Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle G08G1/0967 takes precedence
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

In particular embodiments, a computing system may receive ambient light measurements of a scene. The ambient light measurements may be generated by one or more light sensors of a vehicle. The system may receive an image captured by an optical sensor of the vehicle. The image may include color information associated with an object in a scene. The system may process the image to estimate a color associated with the object based on the ambient light measurements and the color information in the image. The system may determine the object based on the estimated color and may provide instructions to activate a safety feature in response to the determined object.

Claims

1. A method comprising, by a computing system:

receiving ambient light measurements from one or more light sensors of a vehicle;

receiving an image captured by an optical sensor of the vehicle, the image comprising color information associated with an object in a scene;

processing the image to estimate a color associated with the object based on the ambient light measurements and the color information in the image;

determining the object based at least in part on the estimated color; and

providing instructions to activate a safety feature in response to the determined object.

2. The method of claim 1, wherein the color information comprises:

a color intensity of each of red, green, and blue channels over a wavelength range;

or

an intensity of ambient light in the wavelength range.

3. The method of claim 2, wherein processing the image to estimate the color associated with the object comprises:

deconvolving the ambient light present in one or more of the red, green, or blue channels in the wavelength range.

4. The method of claim 2, wherein the ambient light measurements comprise a plurality of light-intensity measurements at different wavelength subranges within the wavelength range.

5. The method of claim 1, wherein processing the image to estimate the color and detecting the object based on the estimated color are performed by a first machine-learning model.

6. The method of claim 5, wherein processing the image to estimate the color is performed by the first machine-learning model and detecting the object based on the estimated color is performed by a second machine-learning model.

7. The method of claim 6, wherein the estimated color by the first machine-learning model is fed into the second machine-learning model for object detection.

8. The method of claim 5, wherein the first machine-learning model is trained based on a set of training samples, each sample comprising:

a raw image of a sample visual scene including a sample object, wherein color information of the sample object is a result of a true color of the sample object and ambient light reflected or bounced off from the sample object;

ambient light measurements for the sample visual scene; and

a corrected image of the sample visual scene, the corrected image representing the true color of the sample object.

9. The method of claim 6, wherein the second machine-learning model is trained based on a set of training samples, wherein at least one training sample in the set of training samples comprises:

a corrected image whose color information has been corrected based on detected ambient light; and

at least one identification of an object depicted in the corrected image.

10. The method of claim 1, further comprising:

adjusting a display setting of a display device within the autonomous vehicle based on the ambient light measurements.

11. The method of claim 1, wherein the activated safety feature is configured to generate an alert associated with the determined object.

12. The method of claim 1, wherein the activated safety feature is configured to send one or more driving instructions to actuators of the vehicle.

13. A system comprising:

one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by one or more of the processors to cause the system to perform operations comprising;

receiving ambient light measurements from one or more light sensors of a vehicle;

receiving an image captured by an optical sensor of the vehicle, the image comprising color information associated with an object in a scene;

processing the image to estimate a color associated with the object based on the ambient light measurements and the color information in the image;

determining the object based at least in part on the estimated color; and

providing instructions to activate a safety feature in response to the determined object.

14. The system of claim 13, wherein processing the image to estimate the color and detecting the object based on the estimated color are performed by a first machine-learning model.

15. The system of claim 14, wherein processing the image to estimate the color is performed by the first machine-learning model and detecting the object based on the estimated color is performed by a second machine-learning model.

16. The system of claim 15, wherein the second machine-learning model is trained based on a set of training samples, wherein at least one training sample in the set of training samples comprises:

a corrected image whose color information has been corrected based on detected ambient light; and

at least one identification of an object depicted in the corrected image.

17. One or more computer-readable non-transitory storage media embodying software that is operable when executed to cause one or more processors to perform operations comprising:

receiving ambient light measurements from one or more light sensors of a vehicle;

receiving an image captured by an optical sensor of the vehicle, the image comprising color information associated with an object in a scene;

processing the image to estimate a color associated with the object based on the ambient light measurements and the color information in the image;

determining the object based at least in part on the estimated color; and

providing instructions to activate a safety feature in response to the determined object.

18. The media of claim 17, wherein processing the image to estimate the color and detecting the object based on the estimated color are performed by a first machine-learning model.

19. The media of claim 18, wherein processing the image to estimate the color is performed by the first machine-learning model and detecting the object based on the estimated color is performed by a second machine-learning model.

20. The media of claim 19, wherein the second machine-learning model is trained based on a set of training samples, wherein at least one training sample in the set of training samples comprises:

a corrected image whose color information has been corrected based on detected ambient light; and

at least one identification of an object depicted in the corrected image.