Parallel high dynamic exposure range sensor

• US 10,609,295 B1
• Filed: 01/14/2019
• Issued: 03/31/2020
• Est. Priority Date: 05/31/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Lerner, Scott
• Primary Examiner(s): Daniels, Anthony J
• Application Number:
  US16/247,138
• Time in Patent Office: 442 Days
• Field of Search: 3482211, 348362
• CPC Class Codes: G02B 3/0037 characterized by the distribution or form of lenses
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 25/533 by using differing integration times for different sensor regions
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 9/00 Details of colour television systems

Abstract

Systems, apparatuses, and methods are presented for taking a combination of images taken synchronous in time with one another. According to one example, the present disclosure proposes one or more sensor arrays, each of which comprises multiple pixel sensors arranged to capture image data responsive to light exposure. Light is incident on the respective sensor arrays during substantially synchronous exposures. The one or more sensor arrays are configured such that the image data captured by the respective sensor arrays during the synchronous exposure differ in at least one of a luminance output or a color profile from one another.

Claims

1. A system comprising:

an image sensor including a first plurality of pixels and a second plurality of pixels, wherein a first pixel of the first plurality of pixels is disposed immediately adjacent a second pixel of the second plurality of pixels in a sensing area and the first plurality of pixels further includes at least a third pixel;

an optical device configured to control an amount of exposure of the first pixel and the second pixel such that the first pixel has a first exposure and the second pixel has a second exposure, wherein the first exposure and the second exposure are taken simultaneously synchronously the first exposure differs from the second exposure, the first pixel is disposed immediately adjacent the second pixel in the sensing area so that the first pixel and second pixel would be subject to substantially a same amount of light if the optical device were not present, and the first pixel and the at least the third pixel are both subject to the first exposure; and

a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising;

interpolate between the first pixel and the at least the third pixel at the first exposure to determine a luminance of the first pixel and the at least the third pixel;

determine both a first luminance from the first exposure and a second luminance from the second exposure; and

determine a composite luminance from the first luminance and the second luminance, wherein the composite luminance is determined by interpolating between the first luminance and the second luminance and according to a light condition and the interpolating between the first luminance and the second luminance comprises one or more of a bicubic, a simple average, a bicubic spline, a cubic nearest-neighbor, or a bilinear.

2. The system of claim 1, wherein the first exposure comprises substantially a full exposure and the second exposure comprises substantially fifty percent exposure.

3. The system of claim 1, wherein the optical device includes an optical filter that is configured to transmit a relatively different amount of light to the first pixel as compared to the second pixel.

4. The system of claim 1, wherein the optical device comprises a lenslet array that is configured to transmit a relatively different amount of light to the first pixel as compared to the second pixel.

5. The system of claim 1, wherein the image sensor includes a third plurality of pixels that includes a third pixel, the optical device is configured to control an amount of exposure of the third pixel of the plurality of third pixels such that the third pixel of the plurality of third pixels has third exposure taken simultaneously with and for substantially the same time duration as the first exposure and the second exposure, and the first exposure, the second exposure, and the third exposure all differ from one another.

6. The system of claim 1, wherein the first plurality of pixels and the second plurality of pixels are interspersed with one another in a non-random pattern in the sensing area.

7. A method comprising:

providing an image sensor with a first plurality of pixels and a second plurality of pixels, wherein the first plurality of pixels includes a first pixel and at least a third pixel and the second plurality of pixels includes a second pixel;

exposing the first pixels, the second pixel, and the at least the third pixel to light substantially synchronously;

controlling the exposing such that the first pixel and the at least the third pixel have a first exposure and the second pixel has a second exposure, wherein the first exposure differs from the second exposure, the controlling the exposing is performed with an optical device that is configured to transmit a relatively different amount of light to the first pixel as compared to the second pixel, and the first pixel is disposed immediately adjacent the second pixel in a sensing area so that the first pixel and second pixel would be subject to substantially a same amount of light if the optical device were not present;

interpolating between the first pixel and the at least the third pixel at the first exposure to determine a luminance of the first pixel and the at least the third pixel;

determining both a first luminance from the first exposure and a second luminance from the second exposure; and

determining a composite luminance from the first luminance and the second luminance, wherein the composite luminance is determined by interpolating between the first luminance and the second luminance and according to a light condition and the interpolating between the first luminance and the second luminance comprises one or more of a bicubic, a simple average, a bicubic spline, a cubic nearest-neighbor, or a bilinear.

8. The method of claim 7, further comprising capturing a series of images over a short time sequence.

9. The method of claim 7, wherein the first exposure comprises substantially a full exposure and the second exposure comprises substantially fifty percent exposure.

10. The method of claim 7, wherein the optical device comprises a lenslet array that is configured to transmit a relatively different amount of light to the first pixel as compared to the second pixel.

11. The method of claim 7, wherein the providing an image sensor with a first plurality of pixels and a second plurality of pixels further includes providing the image sensor with a third plurality of pixels that includes a third pixel, the controlling the exposing such that the first pixel has the first exposure and the second pixel has the second exposure and the first exposure differs from the second exposure further includes controlling an amount of exposure of the third pixel of the third plurality of pixels such that the third pixel of the third plurality of pixels has a third exposure taken simultaneously with and for substantially the same time duration as the first exposure and the second exposure, and the first exposure, the second exposure, and the third exposure all differ from one another.

12. The method of claim 7, wherein the first plurality of pixels and the second plurality of pixels are interspersed with one another in a non-random pattern in the sensing area.