Parallel high dynamic exposure range sensor

• US 10,212,356 B1
• Filed: 05/31/2017
• Issued: 02/19/2019
• Est. Priority Date: 05/31/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Lerner, Scott
• Primary Examiner(s): Daniels, Anthony J
• Application Number:
  US15/610,314
• Time in Patent Office: 629 Days
• Field of Search: 3482211, 348362
• CPC Class Codes: G02B 3/0037 characterized by the distribution or form of lenses
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 25/533 by using differing integration times for different sensor regions
• CPC Class Codes: H04N 25/535 by dynamic region selection
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 9/00 Details of colour television systems

Abstract

Systems, apparatuses, and methods are presented for taking a combination of images taken synchronous in time with one another. According to one example, the present disclosure proposes one or more sensor arrays, each of which comprises multiple pixel sensors arranged to capture image data responsive to light exposure. Light is incident on the respective sensor arrays during substantially synchronous exposures. The one or more sensor arrays are configured such that the image data captured by the respective sensor arrays during the synchronous exposure differ in at least one of a luminance output or a color profile from one another.

Claims

1. A system comprising:

an image sensor including a first plurality of pixels and a second plurality of pixels, wherein a first pixel from the first plurality of pixels is disposed immediately adjacent a second pixel of the second plurality of pixels in a sensing area;

an optical device configured to control an amount of exposure of the first pixel and the second pixel such that the first pixel has a first exposure and the second pixel has a second exposure, wherein the first exposure and the second exposure are taken simultaneously synchronously, and wherein the first exposure differs from the second exposure; and

a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising;

determine both a first luminance from the first exposure and a second luminance from the second exposure; and

determine a composite luminance from the first luminance and the second luminance wherein the composite luminance is determined by interpolating between the first luminance and the second luminance and according to a light condition, and wherein the interpolating comprises a bicubic interpolation where the value of the first pixel is based on sixteen pixels surrounding the first pixel.

2. The system of claim 1, wherein first exposure comprises substantially a full exposure and the second exposure comprises substantially fifty percent exposure.

3. The system of claim 1, wherein the optical device includes an optical filter that is configured transmit a relatively different amount of light to the first pixel as compared to the second pixel.

4. The system of claim 1, wherein the optical device comprises a lenslet array that is configured transmit a relatively different amounts of light to the first pixel as compared to the second pixel.

5. The system of claim 1, wherein the first pixel is disposed immediately adjacent the second pixel in the sensing area so that the first pixel and second pixel would be subject to substantially a same amount of light if the optical device were not present.

6. The system of claim 1, wherein the image sensor includes a third plurality of pixels that include a third pixel and the optical device is configured to control an amount of exposure of the third pixel such that the third pixel has third exposure taken simultaneously with one another and for substantially the same time duration as the first exposure and the second exposure, and wherein the first exposure, the second exposure, and the third exposure all differ from one another.

7. The system of claim 1, wherein the first plurality of pixels includes the first pixel and at least a third pixel, the first pixel and the third pixel both subject to the first exposure and further comprising:

a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including interpolating between first pixel and the third pixel at the first exposure of determine a luminance of the first pixel and the third pixel.

8. The system of claim 1, wherein the first plurality of pixels and the second plurality of pixels are interspersed with one another in a non-random pattern in the sensing area.

9. A method comprising:

providing an image sensor with a first plurality of pixels and a second plurality of pixels, wherein the first plurality of pixels includes a first pixel and the second plurality of includes a second pixel;

exposing the first pixel and the second pixel to light substantially synchronously; and

controlling the exposing such that the first pixel has a first exposure and the second pixel has a second exposure and the first exposure differs from the second exposure;

determining both a first luminance from the first exposure and a second luminance from the second exposure; and

determining a composite luminance from the first luminance and the second luminance, wherein determining the composite luminance includes interpolating between the first luminance and the second luminance and according to a light condition, and wherein the interpolating comprises a bicubic interpolation where the value of the first pixel is based on sixteen pixels surrounding the first pixel.

10. The method of claim 9, further comprising capturing a series of images over a short time sequence.