Multiple operating modes to expand dynamic range

• US 10,757,320 B2
• Filed: 11/26/2018
• Issued: 08/25/2020
• Est. Priority Date: 12/28/2017
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Wendel, Andreas, Dittmer, Jeremy, Hermalyn, Brendan, Ingram, Benjamin
• Primary Examiner(s): Vieaux, Gary C
• Application Number:
  US16/199,998
• Publication Number:
  US 20190208111A1
• Time in Patent Office: 638 Days
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/745 Detection of flicker frequency or suppression of flicker wherein the flicker is caused by illumination, e.g. due to fluorescent tube illumination or pulsed LED illumination
• CPC Class Codes: H04N 23/75 by influencing optical camera components

Abstract

Example embodiments relate to multiple operating modes to expand dynamic range. An example embodiment includes a camera system. The camera system may include a first image sensor having a first dynamic range corresponding to a first range of luminance levels in a scene. The system may also include a second image sensor having a second dynamic range corresponding to a second range of luminance levels in the scene. The camera system may further include a processor coupled to the first image sensor and the second image sensor. The processor may be configured to execute instructions to identify objects of a first type in a first image of the scene captured by the first image sensor and identify objects of a second object type in a second image of the scene captured by the second image sensor.

Claims

1. A camera system, comprising:

a first image sensor having a first dynamic range corresponding to a first range of luminance levels in a scene;

a second image sensor having a second dynamic range corresponding to a second range of luminance levels in the scene, wherein the second image sensor receives light from the scene via a neutral-density filter, and wherein the second range of luminance levels includes luminance levels that are higher than the first range of luminance levels; and

a processor coupled to the first image sensor and the second image sensor, wherein the processor is configured to execute instructions to;

identify objects of a first object type in a first image of the scene captured by the first image sensor, wherein the first object type has an expected luminance within the first range of luminance levels; and

identify objects of a second object type in a second image of the scene captured by the second image sensor, wherein the second object type has an expected luminance within the second range of luminance levels.

2. The camera system of claim 1,

wherein the first image sensor has a variable exposure that is adjusted by a controller executing instructions based on illuminance, and

wherein the second image sensor has a fixed exposure.

3. The camera system of claim 2, wherein adjusting the variable exposure based on luminance comprises adjusting an exposure duration, an aperture size, a gain level, or an ISO sensitivity of the first image sensor.

4. The camera system of claim 2,

wherein the variable exposure is adjusted by modifying a shutter speed or gain associated with a shutter on the first image sensor,

wherein the first object type comprises:

objects in motion relative to the camera system and having high velocities relative to the shutter speed;

or

actively illuminated objects that are modulated at faster rates than the shutter speed, and

wherein the second object type comprises;

objects in motion relative to the camera system and having low velocities or comparable velocities relative to the shutter speed;

or

actively illuminated objects that are modulated at similar or slower rates than the shutter speed.

5. The camera system of claim 1, further comprising a third image sensor having a third dynamic range corresponding to a third range of luminance levels in the scene,

wherein the processor is coupled to the third image sensor,

wherein the third range of luminance levels includes at least one luminance level that is lower than the first range of luminance levels, and

wherein the processor is further configured to execute instructions to identify objects of a third object type in a third image of the scene captured by the third image sensor, wherein the third object type has an expected luminance within the third range of luminance levels.

6. The camera system of claim 5, wherein the third image sensor has a variable exposure that is adjusted by a controller executing instructions based on illuminance.

7. The camera system of claim 6, wherein adjusting the variable exposure based on luminance comprises adjusting an exposure duration, shutter speed, an aperture size, or an ISO sensitivity of the third image sensor.

8. The camera system of claim 1, wherein the first image sensor and the second image sensor are non-overlapping sensor regions of a single image sensor.

9. The camera system of claim 1, further comprising:

a third image sensor having the first dynamic range; and

a fourth image sensor having the second dynamic range,

wherein the fourth image sensor receives light from a separate perspective of the scene via an additional neutral-density filter,

wherein the processor is coupled to the third image sensor and the fourth image sensor, and

wherein the processor is further configured to execute instructions to;

identify objects of the first object type in a third image of the separate perspective of the scene captured by the third image sensor; and

identify objects of the second object type in a fourth image of the separate perspective of the scene captured by the fourth image sensor.

10. The camera system of claim 1, further comprising:

a third image sensor having the first dynamic range; and

a fourth image sensor having the second dynamic range,

wherein the fourth image sensor receives light from a separate perspective of the scene via an additional neutral-density filter,

wherein the processor is coupled to the third image sensor and the fourth image sensor, and

wherein the processor is further configured to execute instructions to;

identify objects of a third object type in a third image of the separate perspective of the scene captured by the third image sensor, wherein the third object type has an expected luminance within the first range of luminance levels; and

identify objects of a fourth object type in a fourth image of the separate perspective of the scene captured by the fourth image sensor, wherein the fourth object type has an expected luminance within the second range of luminance levels.

11. The camera system of claim 10,

wherein the third image sensor has a variable exposure that is adjusted by a controller executing instructions based on luminance, and

wherein the fourth image sensor has a fixed exposure.

12. The camera system of claim 11, wherein adjusting the variable exposure based on luminance comprises adjusting an exposure duration, an aperture size, or an ISO sensitivity of the third image sensor.

13. The camera system of claim 1, wherein the processor is further configured to execute instructions to:

identify objects of the first object type in the second image of the scene; and

check for errors in the identified objects of the first object type by comparing the objects of the first object type identified in the first image of the scene and the objects of the first object type identified in the second image of the scene.

14. The camera system of claim 1, further comprising:

a first lens optically coupled to the first image sensor and having a first focal length; and

a second lens optically coupled to the second image sensor and having a second focal length,

wherein the first focal length is different than the second focal length,

wherein, correspondingly, the first image of the scene and the second image of the scene are captured at different focal lengths,

wherein the processor is further configured to execute instructions to generate an image that simulates an image captured by a simulated image sensor having a simulated lens optically coupled thereto, and

wherein the simulated lens has a focal length between the first focal length and the second focal length.

15. The camera system of claim 1, further comprising:

a first lens optically coupled to the first image sensor and having a first focal length; and

a second lens optically coupled to the second image sensor and having a second focal length,

wherein the first focal length is different than the second focal length,

wherein, correspondingly, the first image of the scene and the second image of the scene are captured at different focal lengths, and

wherein the processor is further configured to execute instructions to;

identify objects of the first object type in the second image of the scene; and

determine a distance of one or more objects of the first object type in the scene from the camera system based on;

the first focal length;

the second focal length;

a focus of the one or more objects in the first image of the scene; and

a focus of the one or more objects in the second image of the scene.

16. The camera system of claim 1, wherein the first image and the second image are captured substantially simultaneously.

17. The camera system of claim 1, wherein the first image sensor and the second image sensor are vertically aligned.

18. The camera system of claim 1, wherein the camera system is mounted on an autonomous vehicle and used for object detection and avoidance.

19. A method, comprising:

capturing, by a first image sensor having a first dynamic range corresponding to a first range of luminance levels in a scene, a first image of the scene;

capturing, by a second image sensor having a second dynamic range corresponding to a second range of luminance levels in the scene, a second image of the scene,

wherein the second image sensor captures the second image of the scene via a neutral-density filter, and

wherein the second range of luminance levels includes luminance levels that are higher than the first range of luminance levels;

identifying, by a processor coupled to the first image sensor and the second image sensor, a first object of a first object type in the first image,

wherein the first object type has an expected luminance within the first range of luminance levels; and

identifying, by the processor, a second object of a second object type in the second image,

wherein the second object type has an expected luminance within the second range of luminance levels.

20. A non-transitory, computer-readable medium with instructions stored thereon, wherein the instructions, when executed by a processor, comprise:

receiving, from a first image sensor having a first dynamic range corresponding to a first range of luminance levels in a scene, a first image of the scene,

wherein the first image of the scene was captured by the first image sensor;

receiving, from a second image sensor having a second dynamic range corresponding to a second range of luminance levels in the scene, a second image of the scene,

wherein the second image of the scene was captured by the second image sensor via a neutral-density filter, and

wherein the second range of luminance levels includes luminance levels that are higher than the first range of luminance levels;

identifying a first object of a first object type in the first image,

wherein the first object type has an expected luminance within the first range of luminance levels; and

identifying a second object of a second object type in the second image,

wherein the second object type has an expected luminance within the second range of luminance levels.