Image sensor combining high dynamic range techniques

• US 9,843,746 B2
• Filed: 05/03/2016
• Issued: 12/12/2017
• Est. Priority Date: 05/03/2016
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: AltaSens, Inc. (JVC Kenwood Corporation)
• Inventors: Koshkin, Ilya, Kozlowski, Lester Joseph, Petersen, Anders Kongstad, McKee, Jeffrey Alan
• Primary Examiner(s): SELBY, GEVELL V
• Application Number:
  US15/145,220
• Publication Number:
  US 20170324913A1
• Time in Patent Office: 588 Days
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 25/11 Arrangement of colour filter arrays [CFA]; Filter mosaics
• CPC Class Codes: H04N 25/57 Control of the dynamic range
• CPC Class Codes: H04N 25/585 with pixels having different sensitivities within the sensor, e.g. fast or slow pixels or pixels having different sizes
• CPC Class Codes: H04N 25/589 with different integration times, e.g. short and long exposures
• CPC Class Codes: H04N 25/59 by controlling the amount of charge storable in the pixel, e.g. modification of the charge conversion ratio of the floating node capacitance
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors

Abstract

Various technologies described herein pertain to combining high dynamic range techniques to enable rendering higher dynamic range scenes with an image sensor. The image sensor can implement a combination of spatial exposure multiplexing and temporal exposure multiplexing, for example. By way of another example, the image sensor can implement a combination of spatial exposure multiplexing and dual gain operation. Pursuant to another example, the image sensor can implement a combination of temporal exposure multiplexing and dual gain operation. In accordance with yet another example, the image sensor can implement a combination of spatial exposure multiplexing, temporal exposure multiplexing, and dual gain operation.

Claims

1. An image sensor, comprising:

a pixel array that comprises pixels;

a timing controller configured to control exposure times of the pixels in the pixel array, the timing controller configured to;

control a first subset of the pixels in the pixel array to have a first exposure time during a first time period, wherein a first frame of an input image stream is captured during the first time period;

control a second subset of the pixels in the pixel array to have a second exposure time during the first time period;

control the first subset of the pixels in the pixel array to have a third exposure time during a second time period, wherein a second frame of the input image stream is captured during the second time period; and

control the second subset of the pixels in the pixel array to have a fourth exposure time during the second time period;

wherein the first exposure time, the second exposure time, the third exposure time, and the fourth exposure time differ from each other;

a readout circuit configured to read out signals from the pixels in the pixel array for the first frame and the second frame of the input image stream; and

an analog-to-digital converter configured to convert the signals to pixel values for the first frame and the second frame of the input image stream;

wherein the first subset of the pixels in the pixel array comprise first non-rectangular units of pixels;

wherein the second subset of the pixels in the pixel array comprise second non-rectangular units of pixels;

wherein the first non-rectangular units of pixels oppose the second non-rectangular units of pixels to form a zigzag pattern in the pixel array; and

wherein each of the first non-rectangular units of pixels and each of the second non-rectangular units of pixels are L-shaped and comprise three adjacent pixels in a row of the pixel array and a fourth pixel in an adjacent row of the pixel array.

2. The image sensor of claim 1, further comprising:

an image signal processor configured to generate an output frame based on the pixel values for the first frame and the second frame of the input image stream.

3. The image sensor of claim 1, wherein:

the image sensor is configured to output the pixel values for the first frame and the second frame of the input image stream to an image signal processor; and

an output frame is generated by an image signal processor based on the pixel values for the first frame and the second frame of the input image stream.

4. The image sensor of claim 1, wherein:

the readout circuit is further configured to;

amplify the signals read out from the pixels in the pixel array by a first analog gain to output first amplified signals for the first frame and the second frame of the input image stream; and

amplify the signals read out from the pixels in the pixel array by a second analog gain to output second amplified signals for the first frame and the second frame of the input image stream, wherein the first analog gain differs from the second analog gain; and

the analog-to-digital converter is further configured to;

convert the first amplified signals to first amplified pixel values for the first frame and the second frame of the input image stream; and

convert the second amplified signals to second amplified pixel values for the first frame and the second frame of the input image stream.

5. The image sensor of claim 4, further comprising:

an image signal processor configured to generate an output frame based on;

the first amplified pixel values for the first frame and the second frame of the input image stream; and

the second amplified pixel values for the first frame and the second frame of the input image stream.

6. The image sensor of claim 4, wherein:

the image sensor is configured to output;

the first amplified pixel values for the first frame and the second frame of the input image stream; and

the second amplified pixel values for the first frame and the second frame of the input image stream; and

an output frame is generated by an image signal processor based on;

the first amplified pixel values for the first frame and the second frame of the input image stream; and

the second amplified pixel values for the first frame and the second frame of the input image stream.

7. An image sensor, comprising:

a pixel array that comprises pixels, wherein the pixel array comprises a first subset of the pixels and a second subset of the pixels, wherein the first subset of the pixels in the pixel array comprise first non-rectangular units of pixels, wherein the second subset of the pixels in the pixel array comprise second non-rectangular units of pixels, wherein the first non-rectangular units of pixels oppose the second non-rectangular units of pixels to form a zigzag pattern in the pixel array, and wherein each of the first non-rectangular units of pixels and each of the second non-rectangular units of pixels are L-shaped and comprise three adjacent pixels in a row of the pixel array and a fourth pixel in an adjacent row of the pixel array;

a timing controller configured to control exposure times of the pixels in the pixel array, the timing controller configured to control the exposure times of the pixels at least one of temporally or spatially such that an input image stream has two or more differing exposure times;

a readout circuit configured to;

read out signals from the pixels in the pixel array for the input image stream having the two or more differing exposure times;

amplify the signals read out from the pixels in the pixel array by a first analog gain to output first amplified signals for the input image stream having the two or more differing exposure times; and

amplify the signals read out from the pixels in the pixel array by a second analog gain to output second amplified signals for the input image stream having the two or more differing exposure times, wherein the first analog gain differs from the second analog gain; and

an analog-to-digital converter configured to;

convert the first amplified signals to first amplified pixel values for the input image stream having the two or more differing exposure times; and

convert the second amplified signals to second amplified pixel values for the input image stream having the two or more differing exposure times.

8. The image sensor of claim 7, the timing controller being configured to control the exposure times of the pixels temporally comprises the timing controller being configured to:

control the pixels in the pixel array to have a first exposure time during a first time period, wherein a first frame of the input image stream is captured during the first time period; and

control the pixels in the pixel array to have a second exposure time during a second time period, wherein a second frame of the input image stream is captured during the second time period;

wherein the first exposure time and the second exposure time differ from each other.

9. The image sensor of claim 7, the timing controller being configured to control the exposure times of the pixels spatially comprises the timing controller being configured to:

control the first subset of the pixels in the pixel array to have a first exposure time during a time period, wherein a frame of the input image stream is captured during the time period; and

control the second subset of the pixels in the pixel array to have a second exposure time during the time period; and

wherein the first exposure time and the second exposure time differ from each other.

10. The image sensor of claim 7, the timing controller being configured to control the exposure times of the pixels both temporally and spatially comprises the timing controller being configured to:

control the first subset of the pixels in the pixel array to have a first exposure time during a first time period, wherein a first frame of the input image stream is captured during the first time period;

control the second subset of the pixels in the pixel array to have a second exposure time during the first time period;

control the first subset of the pixels in the pixel array to have a third exposure time during a second time period, wherein a second frame of the input image stream is captured during the second time period; and

control the second subset of the pixels in the pixel array to have a fourth exposure time during the second time period;

wherein the first exposure time, the second exposure time, the third exposure time, and the fourth exposure time differ from each other.

11. The image sensor of claim 7, further comprising:

an image signal processor configured to generate an output frame based on;

the first amplified pixel values for the input image stream having the two or more differing exposure times; and

the second amplified pixel values for the input image stream having the two or more differing exposure times.

12. The image sensor of claim 7, wherein:

an output frame is generated by an image signal processor based on;

the first amplified pixel values for the input image stream having the two or more differing exposure times; and

the second amplified pixel values for the input image stream having the two or more differing exposure times.

13. A method of increasing dynamic range of an image sensor, the image sensor comprising a pixel array that comprises pixels, the method comprising:

controlling a first subset of the pixels in the pixel array to have a first exposure time during a first time period, wherein a first frame of an input image stream is captured during the first time period;

controlling a second subset of the pixels in the pixel array to have a second exposure time during the first time period;

reading out signals from the pixels in the pixel array for the first frame of the input image stream;

converting the signals for the first frame to digital pixel values for the first frame of the input image stream;

controlling the first subset of the pixels in the pixel array to have a third exposure time during a second time period, wherein a second frame of the input image stream is captured during the second time period;

controlling the second subset of the pixels in the pixel array to have a fourth exposure time during the second time period, wherein the first exposure time, the second exposure time, the third exposure time, and the fourth exposure time differ from each other;

reading out signals from the pixels in the pixel array for the second frame of the input image stream; and

converting the signals for the second frame to digital pixel values for the second frame of the input image stream;

wherein the first subset of the pixels in the pixel array comprise first non-rectangular units of pixels;

wherein the second subset of the pixels in the pixel array comprise second non-rectangular units of pixels;

wherein the first non-rectangular units of pixels oppose the second non-rectangular units of pixels to form a zigzag pattern in the pixel array; and

wherein each of the first non-rectangular units of pixels and each of the second non-rectangular units of pixels are L-shaped and comprise three adjacent pixels in a row of the pixel array and a fourth pixel in an adjacent row of the pixel array.

14. The method of claim 13, further comprising:

generating an output frame based on the digital pixel values for the first frame of the input image stream and the digital pixel values for the second frame of the input image stream.