Method And Apparatus For Imaging Of Scenes Having Large Intensity Variance

• US 20080136953A1
• Filed: 12/06/2005
• Published: 06/12/2008
• Est. Priority Date: 12/07/2004
• Status: Active Grant

• Current Assignee: Noam Sorek
• Original Assignee: Bright Imaging Ltd.
• Inventors: Sorek, Noam, Barnea, Daniel I.
• Granted Patent: US 7,701,499 B2
• Time in Patent Office: Days
• US Class Current: 348/308
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 25/57 Control of the dynamic range
• CPC Class Codes: H04N 25/58 involving two or more exposures
• CPC Class Codes: H04N 25/772 comprising A/D, V/T, V/F, I/T or I/F converters

Abstract

The present invention is an electronic imaging device and method of expanding its dynamic range. The invention comprises modifying the imaging device by providing a comparator and reset trigger circuit located at or near the site of each of the pixels. The comparator triggers a reset mechanism when the voltage in the charged pixel reaches a reference voltage. During the global exposure time each individual pixel can be self-reset several times, depending on the intensity of the radiation incident upon it. At the end of the global exposure time, each pixel contains only the “residual value” of voltage that remains from the last individual self-reset to the end of the global integration time. An algorithm is implemented, which estimates and determines the number of resets undergone by each pixel during the global exposure time. From this number of resets the algorithm is able to reconstruct the true image. As opposed to prior art methods that make use of pixel reset the device of the invention does not contain electronic components for counting and remembering the number of times that each of the pixels was reset.

Claims

1. A method of expanding the dynamic range of an electronic imaging device, said method comprising:

a. providing a sensor chip comprised of a one or two-dimensional array of pixels and a centrally located sensor-chip logic area, wherein the global exposure time interval of a frame begins with reset signals and ends with sample signals;

b. providing a comparator and reset trigger circuit located at or near the site of each of said pixels, wherein said comparator triggers a reset mechanism when the voltage in the charged pixel reaches a reference voltage;

whereby, during said global exposure time, each individual pixel can be self-reset several times, depending on the intensity of the radiation incident upon it, and, at the end of said global exposure time, each pixel contains only the residual value of voltage that remains from the last individual self-reset to the end of said global integration time; and

c. providing an algorithm implemented on post acquisition data, wherein said algorithm comprises steps of;

i. using a time and/or space continuity assumption and an adjacent pixel inference rule to determine the number of individual resets for each of said pixels; and

ii. using said determined number of individual resets for each of said pixels, said residual value for each of said pixels, and a basic formula to determine the true image value for each of said pixels;

wherein, the true image is comprised of an array of said true image values for each of said pixels and the combination of said reset mechanism for each pixel in conjunction with the use of said algorithm allows the determination of said true image without the necessity of providing electronic components for counting and remembering the number of times that each of said pixels was reset.

2. A method according to claim 1, wherein the electronic imaging device comprises a CMOS sensor chip.

3. A method according to claim 1, wherein the threshold for the pixel saturation-reset signal is global.

4. A method according to claim 1, wherein different threshold mechanisms are used for each sub-array of pixels.

5. A method according to claim 1 wherein different starting reset signals for the global exposure time interval of a frame can be applied to various sub-arrays.

6. A method according to claim 1, wherein the threshold-reset mechanism is operated by a central reset-enable signal and resets can occur when a pixel value has reached the threshold value and said reset-enable signal is on.

7. A method according to claim 1, wherein the compare and threshold-reset electronics are time shared by a group of pixels;

wherein during a predetermined sequence, each pixel is connected to said electronics and a reset is generated to said pixel if it has passed the threshold.

8. A method according to claim 1, wherein the threshold-reset mechanism samples one pixel, or an average of several pixels, and performs the reset on a group of neighboring pixels, including said pixels that were sampled.

9. A method according to claim 1, wherein the source of the reference voltage that determines the reset value is provided from the central location in the sensor-chip logic area

10. A method according to claim 1, wherein it is assumed that the continuity assumption holds for the whole image and the algorithm comprises the following steps:

a. choose a trajectory of pixels, which are ordered for sequential treatment of the algorithmic process;

b. visit each pixel along the trajectory following a visit to an adjacent neighbor;

c. determine sequentially the n-values for each pixel by using said continuity assumption and the adjacent pixel inference rule relative to one or more of the already visited pixel values; and

d. calculate the true image using the basic formula.

11. A method according to claim 10, wherein a finite, yet small, probability that the continuity assumption may be violated for some inferences of adjacent n-values is assumed resulting in a few wrong n-inferences along specific trajectories used to visit a pixel and the resulting errors are corrected by introducing correction procedures into the algorithm;

wherein such procedures comprise;

a. determining the n-values of the pixels by employing, for the whole pixel array, a succession of iterations utilizing trajectories with different pixel orderings, which will identify violations automatically and gradually converge to a reduced (or zero) number of n-errors.

b. grouping the pixels into relatively ordered n-sets, after error corrections if necessary;

c. assigning the value n=0 to the lowest set in the order;

d. assigning higher n values in order, for the other n-sets, in a consistent manner, up to the highest n value; and

e. incorporating a determined confidence level associated with each estimated n-value in the estimate of the following n-values,

12. A method according to claim 11, wherein step a. of the algorithm comprises the following steps:

a. determine a trajectory of pixel visits.

b. make the n-inference for each new pixel from all of the adjacent pixels that have already been estimated or reference neighbors;

c. produce an estimate of an integer n-step, on the basis of the relevant inference details including incorporation of previously determined confidence levels associated with earlier estimates, if all the n inferences for said new pixel are not identical;

d. assign a “

non-final”

estimate for the n-value of said pixel;

e. interrupt and change said original trajectory, when necessary, so as to approach said “

non-finalized”

pixel from a different direction;

f. estimate the n-value again incorporating said new trajectory;

g. generate a new maximum likelihood estimate by including all the additional reference pixels;

h. adopt said new estimate if it is consistent with all the previously generated estimates;

i. repeat steps e to g if said new estimate is not consistent with all the previously generated estimates;

j. repeat steps a to i for all the regions of the image, until all pixel inferences are completed; and

k. apply combinatorial trials of modified estimates if there are left pockets of inconsistent estimates until consistencies are achieved.

13. A method according to claim 12, wherein the estimate of an integer n-step is made using a maximum likelihood type of estimate.

14. A method according to claim 1, wherein the array of pixels is divided into a main group of pixels that are reset individually during the global exposure time interval whenever the voltage of the individual pixel of said main group reaches the reference voltage and a sparse sub-array of pixels whose integration time or sensitivity are altered such that the voltage of essentially all of said pixels of said sub-array reaches said reference voltage only if the light intensity on such pixel is higher then a predetermined maximal light intensity.

15. A method according to claim 14, wherein the sub-array is created by making a small hardware modification such that the pixels of said sub-array will have an integration time which is shorter than the global exposure time interval.

16. A method according to claim 14, wherein a reduced sensitivity for the sub-array, could also be achieved by use of an attenuating cover for these pixels, which would reduce the intensity of the radiation falling on said covered pixels.

17. A method according to claim 14, wherein the algorithm comprises the steps of:

a. estimating, by direct calculation, the n-value for each of the pixels of the sub-array;

b. apply the n estimates for said sub-array and the continuity rules to estimate the n-values for the main array by making for each sub-array estimate inferences concerning the group of nearest neighboring pixels in said main array;

c. use the estimated values of n for each pixel in said main-array to calculate the “

true values”

of the pixels in said main array, and

d. calculate the compatibly scaled “

true values”

of said sub-array by direct compensation for the shortness of their integration time interval.

18. A method according to claim 17, wherein additional steps of data smoothing using the final results for the main array are carried out to smooth the values of the potentially noisier sub-array pixels.

19. A method according to claim 1, wherein two adjacent exposures having different exposure times are used, wherein:

a. the first exposure time is the global exposure time interval and the pixels are reset individually whenever their voltage reaches the reference voltage; and

b. the second exposure time is shortened such that the voltage of essentially all of said pixels reach said reference voltage only if the light falling on these pixels is stronger then a predetermined maximal light intensity;

said method comprising the steps of;

i. estimating the n-values of the pixels of the short exposed image and,

ii. using said n-values to reconstruct the true n-values of the pixels of the global exposure time image.

20. A method according to claim 19, wherein motion artifacts are corrected by the following steps:

a. applying a registration method comprising repeatedly trying incremental shifts between the residual image of the full exposure time image and the reconstructed estimate of residual values derived from the short exposure time image until maximal correlation is achieved;

b. deriving the n-estimates for the pixels of the short exposure and associating said n-estimates to the registration fitted pixels of the long exposed image, and

c. estimating the true n-values for the pixels of the global exposure time image from said n-estimates.

21. A method according to claim 19, wherein motion artifacts are corrected by applying a registration method that adaptively and successively treats different image regions.

22. A method according to claim 1, wherein the electronic imaging device comprises pixels sensitive to light having one color or subsets of pixels each of which is sensitive to the same or different colors.

23. A method according to claim 1, wherein the comparator is connected to each pixel site in one of the following ways:

a. a comparator is located at each of said sites;

b. each of said sites is connected to a comparator shared by a group of pixels; and

c. a group of sites are connected to a single comparator and additional circuitry;

wherein the resultant reset affects a group of neighboring pixels that can or can not be members of said group that is connected to said comparator.

24. An electronic imaging device having expanded dynamic range, said device comprising:

a. a sensor chip comprised of a two-dimensional array of pixels;

b. a centrally located sensor-chip logic area, comprising circuitry for controlling the global exposure time interval of a frame beginning with reset signals and ending with sample signals;

c. a comparator and reset trigger circuit located at or near the site of each of said pixels, wherein said comparator triggers a reset mechanism when the voltage in the charged pixel exceeds a reference voltage;

whereby, during said global exposure time, each individual pixel can be self-reset several times, depending on the intensity of the radiation incident upon it, and, at the end of said global exposure time, each pixel contains only the residual value of voltage that remains from the last individual self-reset to the end of said global integration time; and

d. processing circuitry for carrying out at least part of an algorithm which uses said residual values of voltage to determine the true image value for each of said pixels;

wherein, the true image is comprised of an array of said true image values for each of said pixels and the combination of said reset mechanism for each pixel in conjunction with the use of said algorithm allows the determination of said true image without the necessity of providing electronic components for counting and remembering the number of times that each of said pixels was reset.

25. An electronic imaging device according to claim 24, wherein said electronics imaging device comprises pixels sensitive to light having one color or subsets of pixels each of which is sensitive to the same or different colors.

26. An electronic imaging device according to claim 24, wherein the comparator is connected to each pixel site in one of the following ways:

a. a comparator is located at each of said sites;

b. each of said sites is connected to a comparator shared by a group of pixels; and

c. a group of sites are connected to a single comparator and additional circuitry, wherein the resultant reset affects a group of neighboring pixels that can or cannot be members of said group that is connected to said comparator.