Using multiple exposures to improve image processing for autonomous vehicles

• US 9,654,738 B1
• Filed: 08/07/2013
• Issued: 05/16/2017
• Est. Priority Date: 08/07/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Ogale, Abhijit S., Ferguson, David I.
• Primary Examiner(s): Luo, Kate
• Application Number:
  US13/960,969
• Time in Patent Office: 1,378 Days
• Field of Search: None
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

The invention relates to collecting different images using different camera parameters. As an example, a single camera may capture two different types of images: a dark exposure for light emitting objects and a normal exposure for passive objects. The camera may first capture an image. This first image may be processed to determine the ideal camera settings for capturing the average intensity of the environment. Fixed offset values may be added to these ideal camera settings and the dark exposure may be captured. The ideal camera settings are then used to capture a normal exposure, which in turn may be processed to determine new ideal camera settings. Again, the fixed offset values may be added to the new ideal camera settings and a dark exposure is captured. This process may repeat continuously and periodically, and the resulting images may be processed to identify emissive and passive objects.

Claims

1. A method comprising:

processing an image of a present scene in a vehicle'"'"'s environment to determine a first exposure value for capturing an average amount of light for the present scene to identify objects in the present scene with a camera, the first exposure value including a shutter time value and a gain value;

adding, by one or more processors, a fixed offset shutter time value to the shutter time value and a fixed offset gain value to the gain value to generate a second adjusted exposure value;

using the second exposure value to capture a first exposure image with the camera, wherein the first exposure image is a dark exposure image;

using the first exposure value to capture a second exposure image with the camera; and

using the first exposure image and the second exposure image to control the vehicle in an autonomous mode,

wherein the first exposure value is determined in order to maintain the average amount of light for the present scene in a predetermine range based on logarithmic control for the shutter time value and linear control for the gain value.

2. The method of claim 1, further comprising:

using the first exposure image to identify an emissive object in a scene of the first exposure image that emits light; and

using the identified emissive object to control the vehicle.

3. The method of claim 1, further comprising:

using the second exposure image to identify a passive object in a scene of the second exposure image that does not emit light; and

using the identified passive object to control the vehicle.

4. The method of claim 1, wherein the second exposure image includes a second scene and the method further comprises:

processing the second exposure image to determine a third exposure value for capturing an average amount of light for the second scene with the camera; and

using the third exposure value to capture a third exposure image with the camera.

5. The method of claim 4, further comprising:

adding the fixed offset value to the third exposure value to generate a fourth exposure value; and

using the fourth exposure value to capture a fourth exposure image with the camera, wherein the fourth exposure image is a second dark exposure image.

6. The method of claim 1, wherein the first exposure value is determined by maintaining the average amount of light for a previous scene of a previous exposure image captured prior to the image of the present scene using a previous exposure value by adjusting a shutter time value of the previous exposure value using logarithmic proportion control and adjusting the gain value of the previous exposure value using linear proportion control.

7. A method comprising:

processing a first image of a present scene in a vehicle'"'"'s environment to determine a first exposure value for capturing an average amount of light for the present scene to identify objects in the present scene with a camera, the exposure value including a shutter time value and a gain value;

accessing memory to identify a fixed predetermined exposure value for capturing dark exposure images, the fixed predetermined exposure value includes a fixed shutter time value and a fixed gain value;

capturing, with the camera, a first exposure image using the fixed predetermined exposure value, wherein the first exposure image is a dark exposure image;

using the first exposure value to capture a second exposure image with the camera; and

using the first exposure image and the second exposure image to control the vehicle in an autonomous mode,

wherein the first exposure value is determined in order to maintain the average amount of light for the present scene in a predetermine range based on logarithmic control for the shutter time value and linear control for the gain value.

8. The method of claim 7, wherein the second exposure image includes a second scene and the method further comprises:

processing the second exposure image to determine a second exposure value for capturing an average amount of light for the second scene with the camera; and

using the second exposure value to capture a third exposure image with the camera.

9. The method of claim 7, wherein the first exposure value is determined by maintaining the average amount of light for a previous scene of a previous exposure image captured prior to the image of the present scene using a previous exposure value by adjusting a shutter time value of the previous exposure value using logarithmic proportion control and adjusting a gain value of the previous exposure value using linear proportion control.

10. A system comprising one or more processors configured to:

process an image of a present scene in a vehicle'"'"'s environment to determine a first exposure value for capturing an average amount of light for the present scene to identify objects in the present scene with a camera, the first exposure value including a shutter time value and a gain value;

add a fixed offset shutter time value to the shutter time value and a fixed offset gain value to the gain value to generate a second exposure value;

send instructions to use the adjusted exposure value to capture a first exposure image with a camera, wherein the first exposure image is a dark exposure image;

send instructions for the camera to use the camera setting to capture a second exposure image with the camera; and

use the first exposure image and the second exposure image to control the vehicle in an autonomous mode,

wherein the first exposure value is determined in order to maintain the average amount of light for the present scene in a predetermine range based on logarithmic control for the shutter time value and linear control for the gain value.

11. The system of claim 10, wherein the one or more processors are further configured to:

use the first exposure image to identify an emissive object in a scene of the first exposure image that emits light; and

use the identified emissive object to control the vehicle.

12. The system of claim 10, wherein the one or more processors are further configured to:

use the second exposure image to identify a passive object in a scene of the second exposure image that does not emit light; and

use the identified passive object to control the vehicle.

13. The system of claim 10, wherein the second exposure image includes a second scene and the processor is further configured to:

process the second exposure image to determine a third exposure value for capturing an average amount of light for the second scene with the camera; and

send instructions to use the third exposure value to capture a third exposure image with the camera.

14. The system of claim 13, wherein the one or more processors further configured to:

add the fixed offset value to the third exposure value to generate a fourth exposure value; and

use the fourth exposure value to capture a fourth exposure image with the camera, wherein the fourth exposure image is a second dark exposure image.

15. The system of claim 10, further comprising the camera.

16. The system of claim 10, wherein the one or more processors are further configured to determine the first exposure value by maintaining the average amount of light for a previous scene of a previous exposure image captured prior to the image of the present scene using a previous exposure value by adjusting a shutter time value of the previous exposure value using logarithmic proportion control and adjusting a gain value of the previous exposure value using linear proportion control.

17. A system comprising:

memory storing a fixed predetermined camera setting for capturing dark exposure images;

one or more processors configured to;

process an image of a present scene in a vehicle'"'"'s environment to determine an first exposure value for capturing an average amount of light for the present scene to identify objects in the present scene with a camera, the first exposure value including a shutter time value and a gain value;

access the memory for the fixed predetermined camera setting which includes a fixed shutter time value and a fixed gain value;

send instructions to capture a first exposure image with the camera using the fixed predetermined camera setting, wherein the first exposure image is a dark exposure image;

send instructions to use the camera setting to capture a second exposure image with the camera using the first exposure value; and

use the first exposure image and the second exposure image to control the vehicle in an autonomous mode,

wherein the first exposure value is determined in order to maintain the average amount of light for the present scene in a predetermine range based on logarithmic control for the shutter time value and linear control for the gain value.

18. The system of claim 17, wherein the second exposure image includes a second scene and the one or more processors are further configured to:

process the second exposure image to determine a second exposure value for capturing an average amount of light for the second scene with the camera; and

send instructions to use the second exposure value to capture a third exposure image with the camera.

19. The system of claim 17, further comprising the camera.

20. The system of claim 17, wherein the one or more processors are further configured to determine the first exposure value by maintaining the average amount of light for a previous scene of a previous exposure image captured prior to the image of the present scene using a previous exposure value by adjusting a shutter time value of the previous exposure value using logarithmic proportion control and adjusting a gain value of the previous exposure value using linear proportion control.