Multiple sampling via a time-indexed method to achieve wide dynamic ranges

• US 6,975,355 B1
• Filed: 05/09/2000
• Issued: 12/13/2005
• Est. Priority Date: 02/22/2000
• Status: Expired due to Term

• Current Assignee: Pixim, Inc. (Sony Group Corp.)
• Original Assignee: Pixim, Inc. (Sony Group Corp.)
• Inventors: Deng, Zhonghan, Yang, David Xiao Dong
• Primary Examiner(s): Ometz, David L.
• Assistant Examiner(s): Ye, Lin
• Application Number:
  US09/567,786
• Time in Patent Office: 2,044 Days
• Field of Search: 348/302, 348/308, 348/362, 348/367, 348/294, 348/297, 348/241, 348/298, 250/208.1, 250/214
• US Class Current: 348/308
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 2201/0068 Converting from still picture data
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 25/533 by using differing integration times for different sensor regions
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors
• CPC Class Codes: H04N 25/772 comprising A/D, V/T, V/F, I/T or I/F converters

Abstract

An architecture for a digital pixel sensor is disclosed in which the dynamic range of the sensor is increased by taking samples of a subject to be recorded, where each sample is taken over an interval of a different duration than the other samples. In the preferred embodiment of the invention, an array of pixel elements is fabricated in an integrated circuit. Each of the pixel elements outputs a digital signal and comprises a photodetector and an analog to digital converter. The photodetector is integrated with the analog to digital converter. An array of threshold memory cells, each corresponding to one of the pixel elements, is also provided. An array of time memory cells, each corresponding to one of the pixel elements, establishes a different sampling time for each of the pixel elements for each of multiple samples. An array of memory elements, each coupled to one of the pixel elements, is also provided. The memory elements are also fabricated in the integrated circuit. The memory elements only receive a value from a corresponding one of the pixel elements when the content in a corresponding one of the threshold memory cells permits. In this way, multiple samples may be collected for a subject to be recorded to thereby extend the dynamic range of a photodetector. Integration of the photodetector and the memory that implements this mechanism into the same integrated circuit avoids the latency that would be experienced if an external memory was used.

Claims

1. An image sensor, comprising:

a sensor array comprising a two-dimensional array of pixel elements, each of said pixel elements of said sensor array outputting digital signals as pixel data representing an image of a scene, said digital signals being serial bit streams indicative of a light intensity impinging upon each pixel element, and said sensor array generating multiple sets of pixel data indicative of multiple representations of said image at a plurality of exposure times;

a data memory, in communication with said sensor array, for storing said pixel data for each of said pixel, elements where said pixel data for each pixel element is associated with a representation of said image taken at one of said plurality of exposure times; and

a threshold memory, in communication with said sensor array, for storing a threshold flag for each of said pixel elements, said threshold flag having a first value when said pixel data associated with a pixel element exceeds a predetermined threshold value and having a second value when said pixel data does not exceed the predetermined threshold value, wherein said threshold flag, once being set to said first value at one of said plurality of exposure times, retains said first value through the remaining exposure times of said plurality of exposure times,

wherein said data memory stores pixel data for each pixel element associated with a representation of said image at each of the plurality of exposure times when said threshold flag in said threshold memory associated with the respective pixel element has said second value; and

wherein said sensor array, said data memory and said threshold memory are fabricated in an integrated circuit.

2. The image sensor of claim 1, further comprising:

a time index memory, in communication with said sensor array, for storing a time index value for each of said pixel elements in said sensor array, said time index value indicating a respective one of said plurality of exposure times when said pixel data of a respective pixel element exceeds said predetermined threshold value, said time index memory being fabricated in said integrated circuit.

3. The image sensor of claim 1, wherein said sensor array is of N by M pixels and provides pixel data in k bits, said data memory is N by M by k bits and said threshold memory is N by M bits.

4. The image sensor of claim 2, wherein said sensor array is of N by M pixels and provides pixel data in k bits, said data memory is N by M by k bits, said threshold memory is N by M bits, and said time index memory is of N by M by m bits, where m is a time resolution.

5. The image sensor of claim 1, wherein said plurality of exposure times comprises any of a relatively short exposure time for capturing image information that is related to high illumination areas in said image, a relatively long exposure time for capturing image information that is related to low illumination areas in said image, and intermediate exposure times for capturing image information that is related to gradually increasing illumination areas in said image;

and wherein a combination of said multiple representations of said image at a plurality of exposure time establishes a wide dynamic range for said sensor array.

6. The image sensor of claim 1, wherein said data memory stores pixel data for each pixel element associated with a first representation of said image at a first exposure time and at each subsequent exposure time, said data memory updates pixel data for a pixel element associated with subsequent representations of said image only when said threshold flag in said threshold memory associated with the respective pixel element has said second value.

7. A method for generating electrical signals representing an image in a digital image sensor, comprising:

generating digital signals as pixel data at a plurality of exposure times, multiple sets of pixel data being generated indicative of multiple representations of said image at said plurality of exposure times, said digital signals being serial bit streams generated at each pixel element in a sensor array of pixel elements formed on an integrated circuit and corresponding to a level of an analog signal indicative of a light intensity impinging on said pixel element;

providing a threshold memory on the same integrated circuit including a plurality of threshold flag, each threshold flag associated with a pixel element in said sensor array;

setting each threshold flag in said threshold memory to a first value;

at each exposure time before the last exposure time of said plurality of exposure times, determining if said pixel data of a first pixel element in said sensor array exceeds a predetermined threshold value;

if said pixel data exceeds said predetermined threshold value and said threshold flag associated with said first pixel element has said first value, storing said pixel data in a location in a data memory provided on the same integrated circuit associated with said first pixel element and setting said threshold flag associated with said first pixel element to a second value, said threshold flag having said second value for preventing further update of pixel data for said first pixel element, wherein said threshold flag, once being set to said second value at one of said plurality of exposure times, retains said second value through the remaining exposure times of said plurality of exposure times; and

if said pixel data does not exceed said predetermined threshold value and said threshold flag associated with said first pixel element has said first value, storing said pixel data for said first pixel element in said location in said data memory.

8. The method of claim 7, further comprising:

if said pixel data exceeds said predetermined threshold value and said threshold flag associated with said first pixel element has said first value, storing a time index value in a location in a time index memory associated with said first pixel element, the time index value being indicative of said exposure time when pixel data exceeds said predetermined threshold value.

9. The method of claim 7, further comprising:

if said pixel data of said first pixel element does not exceed said predetermined threshold value before said last exposure time, storing said pixel data for said first pixel element recorded in the last exposure time in said data memory.

10. The method of claim 7, wherein said plurality of exposure times comprises any of a relatively short exposure time for capturing image information that is related to high illumination areas in said image, a relatively long exposure time for capturing image information that is related to low illumination areas in said image, and intermediate exposure times for capturing image information that is related to gradually increasing illumination areas in said image;

and wherein a combination of said multiple representations of said image at a plurality of exposure time establishes a wide dynamic range for said sensor array.

11. A method for generating electrical signals representing an image in a digital image sensor, comprising:

generating digital signals as pixel data at a plurality of exposure times, multiple sets of pixel data being generated indicative of multiple representations of said image at said plurality of exposure times, said digital signals being serial bit streams generated at each pixel element in a sensor array of pixel elements formed on an integrated circuit and corresponding to a level of an analog signal indicative of a light intensity impinging on said pixel element;

providing a threshold memory on the same integrated circuit including a plurality of threshold flag, each threshold flag associated with a pixel element in said sensor array;

setting each threshold flag in said threshold memory to a first value;

at a first exposure time, storing said pixel data for each of said pixel elements of said sensor array in a data memory on the same integrated circuit;

determining if pixel data of each of said pixel elements exceeds a predetermined threshold value;

when said pixel data of a pixel element exceeds said predetermined threshold value, setting said threshold flag associated with said pixel element to a second value, said threshold flag having said second value for preventing further update of pixel data for said pixel element, wherein said threshold flag, once being set to said second value at said first exposure time, retains said second value through the remaining exposure times of said plurality of exposures times;

at each exposure time after the first exposure time, determining if pixel data of each of said pixel elements exceeds said predetermined threshold value;

if said pixel data of a pixel element exceeds said predetermined threshold value and said threshold flag associated with said pixel element has said first value, storing said pixel data in a location in a data memory associated with first pixel element and setting said threshold flag associated with said pixel element to said second value; and

if said pixel data of a pixel element does not exceed said predetermined threshold value and said threshold flag associated with said pixel element has said first value, storing said pixel data for said pixel element in said location in said data memory.

12. The method of claim 11, further comprising:

when said pixel data of a pixel element exceeds said predetermined threshold value at said first exposure time, storing a time index value in a location in a time index memory associated with said pixel element, the time index value being indicative of said first exposure; and

if said pixel data of a pixel element exceeds said predetermined threshold value and said threshold flag associated with said pixel element has said first value, storing a time index value in a location in a time index memory associated with said pixel element, the time index value being indicative of said exposure time when pixel data exceeds said predetermined threshold value.

13. The method of claim 11, wherein said plurality of exposure times comprises any of a relatively short exposure time for capturing image information that is related to high illumination areas in said image, a relatively long exposure time for capturing image information that is related to low illumination areas in said image, and intermediate exposure times for capturing image information that is related to gradually increasing illumination areas in said image;

and wherein a combination of said multiple representations of said image at a plurality of exposure time establishes a wide dynamic range for said sensor array.