Intelligent lens masking system for an autonomous vehicle

• US 10,048,696 B2
• Filed: 12/22/2015
• Issued: 08/14/2018
• Est. Priority Date: 12/22/2015
• Status: Active Grant

• Current Assignee: Uatc, LLC (Uber Technologies, Inc.)
• Original Assignee: Uber Technologies, Inc.
• Inventors: Meyhofer, Eric, Rice, David
• Primary Examiner(s): Antonucci, Anne M
• Assistant Examiner(s): TAVERAS, KENNY ABEL
• Application Number:
  US14/979,351
• Publication Number:
  US 20170177000A1
• Time in Patent Office: 966 Days
• Field of Search: None
• CPC Class Codes: G02F 1/13306 Circuit arrangements or driving methods for the control of single liquid crystal cells G02F1/132, G02F1/133382 take precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices

Abstract

An intelligent lens masking system for a camera array of an autonomous vehicle (AV) can include an array interface to receive real-time data from the camera array as the AV travels along a given route, where each camera of the camera array may include a masking layer on its lens. The intelligent lens masking system can dynamically identify, in the real-time data, a number of light sources in the field of view for each respective camera in the camera array. Once the light source(s) for the respective camera is detected, the intelligent lens masking system can dynamically block the lights source(s) for the respective camera by activating a number of pixels of the masking layer.

Claims

1. An autonomous vehicle (AV) comprising:

a camera array continuously generating image data indicating a surrounding environment of the AV;

acceleration, braking, and steering systems; and

an AV control system comprising one or more processors executing instructions, causing the AV control system to;

process the image data by comparing the image data to stored sub-maps that provide surface data of a current route traveled by the AV;

based on comparing the image data with a current sub-map, autonomously operate the acceleration, braking, and steering systems along the current route;

dynamically identify, based on the image data, one or more light sources for each respective camera in the camera array, wherein a lens of the respective camera comprises a masking layer; and

dynamically block each of the one or more light sources for the respective camera by activating a number of pixels of the masking layer such that blocking each of the one or more light sources preemptively bolsters quality of the image data and reduces processing of the image data by the AV control system.

2. The AV of claim 1, wherein the executed instructions cause the AV control system to track the one or more light sources in a field of view of the respective camera while autonomously operating the AV along the current route.

3. The AV of claim 2, wherein the executed instructions cause the AV control system to dynamically activate and deactivate respective pixels of the masking layer, based on tracking the one or more light sources, to continuously block each of the one or more light sources as the AV travels along the current route and while the one or more light sources remain in the field of view of the respective camera.

4. The AV of claim 1, wherein the masking layer comprises a liquid crystal display (LCD) layer.

5. The AV of claim 4, wherein the LCD layer comprises one of a transparent passive-matrix LCD layer or a transparent active LCD layer.

6. The AV of claim 1, wherein the masking layer comprises a grey-scaled masking layer, and wherein the executed instructions further cause the AV control system to:

determine a brightness gradient for each light source of the one or more light sources;

wherein the executed instructions cause the AV control system to generate voltage signals that cause the masking layer to dynamically block each of the one or more light sources on a grey-scale based on the brightness gradient for the light source.

7. The AV of claim 1, wherein the masking layer comprises a binary masking layer, and wherein the executed instructions further cause the AV control system to:

determine, for each light source of the one or more light sources, whether a brightness of the light source exceeds a minimum threshold brightness for triggering binary masking of the light source on the masking layer, the minimum threshold brightness being correlated to a predicted impact of the light source on image data quality;

wherein the executed instructions cause the control system to dynamically block each light source of the one or more light sources based on the brightness of the light source exceeding the minimum threshold brightness.

8. A non-transitory computer readable medium storing instructions that, when executed by one or more processors of a control system of an autonomous vehicle (AV), cause the control system to:

process real-time image data from a camera system of the AV by comparing the real-time image data with stored sub-maps that provide surface data of a current route traveled by the AV;

based on comparing the real-time image data with a current sub-map, autonomously operate acceleration, braking, and steering systems of the AV along the current route;

dynamically identify, based on the real-time image data generated by the camera system, one or more light sources for each respective camera in the camera system, wherein a lens of the respective camera comprises a masking layer; and

dynamically block each of the one or more light sources for the respective camera by activating a number of pixels of the masking layer such that blocking each of the one or more light sources preemptively bolsters quality of the image data and reduces processing of the real-time data by the control system of the AV.

9. The non-transitory computer readable medium of claim 8, wherein the executed instructions further cause the control system to:

track the one or more light sources in a field of view of the respective camera in the real-time image data while autonomously operating the AV along the current route.

10. The non-transitory computer readable medium of claim 9, wherein the executed instructions further cause the control system to:

dynamically activate and deactivate respective pixels of the masking layer, based on tracking the one or more light sources, to continuously block each of the one or more light sources as the AV travels along the current route and while the one or more light sources remain in the field of view of the respective camera.

11. The non-transitory computer readable medium of claim 8, wherein the one or more light sources correspond to the Sun being in a field of view of the respective camera.

12. The non-transitory computer readable medium of claim 8, wherein the one or more light sources correspond to one or more of reflected sunlight, one or more streetlights, or one or more headlights from another vehicle.

13. The non-transitory computer readable medium of claim 8, wherein the masking layer comprises a grey-scaled masking layer, and wherein the executed instructions further cause the control system to:

determine a brightness gradient for each light source of the one or more light sources;

wherein the executed instructions cause the control system to generate voltage signals that cause the masking layer to dynamically block each of the one or more light sources on a grey-scale based on the brightness gradient for the light source.

14. The non-transitory computer readable medium of claim 8, wherein the masking layer comprises a binary masking layer, and wherein the executed instructions further cause the control system to:

determine, for each light source of the one or more light sources, whether a brightness of the light source exceeds a minimum threshold brightness for triggering binary masking of the light source on the masking layer, the minimum threshold brightness being correlated to a predicted impact of the light source on image data quality;

wherein the executed instructions cause the control system to dynamically block each light source of the one or more light sources based on the brightness of the light source exceeding the minimum threshold brightness.

15. A computer-implemented method of dynamically masking light sources for a camera system of an autonomous vehicle (AV), the method being performed by one or more processors of a control system of the AV and comprising:

processing real-time image data from the camera system of the AV by comparing the real-time image data with stored sub-maps providing surface data of a current route traveled by the AV;

based on comparing the real-time image data with a current sub-map, autonomously operating acceleration, braking, and steering systems of the AV along the current route;

dynamically identifying, based on the real-time image data generated by the camera system, one or more light sources for each respective camera in the camera system, wherein a lens of the respective camera comprises a masking layer; and

dynamically blocking each of the one or more light sources for the respective camera by activating a number of pixels of the masking layer such that blocking each of the one or more light sources preemptively bolsters quality of the image data and reduces processing of the real-time image data by the control system of the AV.

16. The computer-implemented method of claim 15, further comprising:

tracking the one or more light sources in a field of view of the respective camera in the real-time image data while autonomously operating the AV along the current route.

17. The computer-implemented method of claim 16, further comprising:

dynamically activating and deactivating respective pixels of the masking layer, based on tracking the one or more light sources, to continuously block each of the one or more light sources as the AV travels along the current route and while the one or more light sources remain in the field of view of the respective camera.

18. The computer-implemented method of claim 15, wherein the one or more light sources correspond to one or more of reflected sunlight, one or more streetlights, or one or more headlights from another vehicle.

19. The computer-implemented method of claim 15, wherein the masking layer comprises a grey-scaled masking layer, the method further comprising:

determining a brightness gradient for each light source of the one or more light sources;

wherein the control system of the AV generates voltage signals that cause the masking layer to dynamically block each of the one or more light sources on a grey-scale based on the brightness gradient for the light source.

20. The computer-implemented method of claim 15, wherein the masking layer comprises a binary masking layer, the method further comprising:

determining, for each light source of the one or more light sources, whether a brightness of the light source exceeds a minimum threshold brightness for triggering binary masking of the light source on the masking layer, the minimum threshold brightness being correlated to a predicted impact of the light source on image data quality;

wherein the control system of the AV dynamically blocks each light source of the one or more light sources based on the brightness of the light source exceeding the minimum threshold brightness.