Low-light-level imaging and image processing

• US 5,880,777 A
• Filed: 04/15/1996
• Issued: 03/09/1999
• Est. Priority Date: 04/15/1996
• Status: Expired due to Fees

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Reich, Robert K., Carrick, James E., Loomis, Andrew H., Gove, Alan N., Fay, David A., Waxman, Allen M., Mountain, Robert W., Gregory, James A., McGonagle, William H., Burke, Barry E., Kosicki, Bernard B., Savoye, Eugene D.
• Primary Examiner(s): Faile, Andrew
• Assistant Examiner(s): Harrington, Alicia M.
• Application Number:
  US08/632,746
• Time in Patent Office: 1,058 Days
• Field of Search: 348/317, 348/229, 348/216, 348/217, 257/223, 257/227, 257/228, 257/229, 257/230
• US Class Current: 348/217.1
• CPC Class Codes: H01L 27/14818 Optical shielding
• CPC Class Codes: H01L 27/14887 Blooming suppression
• CPC Class Codes: H01L 27/14893 comprising a photoconductive layer deposited on the CCD structure
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 25/633 by using optical black pixels
• CPC Class Codes: H04N 25/702 SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout
• CPC Class Codes: H04N 5/20 Circuitry for controlling amplitude response

Abstract

An imaging system is provided for imaging a scene to produce a sequence of image frames of the scene at a frame rate, R, of at least about 25 image frames per second. The system includes an optical input port, a charge-coupled imaging device, an analog signal processor, and an analog-to-digital processor (A/D). The A/D digitizes the amplified pixel signal to produce a digital image signal formatted as a sequence of image frames each of a plurality of digital pixel values and having a dynamic range of digital pixel values represented by a number of digital bits, B, where B is greater than 8. A digital image processor is provided for processing digital pixel values in the sequence of image frames to produce an output image frame sequence at the frame rate, R, representative of the imaged scene, with a latency of no more than about 1/R and a dynamic range of image frame pixel values represented by a number of digital bits, D, where D is less than B. The output image frame sequence is characterized by noise-limited resolution of at least a minimum number, N_M, of line pairs per millimeter, referred to the charge-coupled imaging device pixel array, in an imaged scene as a function of illuminance of the input light impinging the charge-coupled imaging device pixels.

Claims

1. An imaging system for imaging a scene to produce a sequence of image frames of the scene at a frame rate, R, of at least about 25 image frames per second, the imaging system comprising:

an optical input port for accepting input light from the scene;

a charge-coupled imaging device comprising an array of pixels configured in a charge storage medium, the charge-coupled imaging device located in relation to the input port such that input light from the scene impinges device pixels, the charge-coupled imaging device producing an electrical pixel signal of analog pixel values based on the input light;

an analog signal processor connected to the charge-coupled imaging device for amplifying the pixel signal;

an analog-to-digital processor connected to the analog signal processor for digitizing the amplified pixel signal to produce a digital image signal formatted as a sequence of image frames each of a plurality of digital pixel values and having a dynamic range of digital pixel values represented by a number of digital bits, B, where B is greater than 8; and

a digital image processor connected to the analog-to-digital processor for processing digital pixel values in the sequence of image frames to produce an output image frame sequence at the frame rate, R, representative of the imaged scene, with a latency of no more than about 1/R;

the output image frame sequence being characterized by noise-limited resolution of at least a minimum number, N_M, of line pairs per millimeter, referred to the charge-coupled imaging device pixel array, in an imaged scene as a function of illuminance of the input light impinging the charge-coupled imaging device pixels, wherein for a scene characterized by a contrast of about 0.3, for a human observation time of about 0.05 seconds, and for an image scene frame rate of about 30 frames per second, N_M is given as N_M =1900 L⁰.51, where L is the value of illuminance of the input light impinging the charge-coupled imaging device pixels, for at least one illuminance value between a range of illuminance values of about 1×

10^-2 LUX and 5×

10^-7 LUX.

2. The imaging system of claim 1, wherein L, the illuminance of the input light impinging the charge-coupled imaging device pixels, ranges between about 1×

10^-3 LUX and 1×

10^-6 LUX.

3. The imaging system of claim 1, wherein the frame rate, R, is at least about 30 frames per second.

4. The imaging system of claim 1, wherein the digital image processor processes digital pixel values in the sequence of image frames from the analog-to-digital processor to remap the dynamic range of the frames to a compressed dynamic range of remapped pixel values represented by a number of digital bits, D, where D is less than the number, B, a sequence of output image frames of remapped pixel values representative of the imaged scene being produced at the frame rate, R, with a latency time of no more than about 1/R.

5. The imaging system of claim 1, wherein the optical input port comprises a lens.

6. The imaging system of claim 5 wherein the pixels of the charge-coupled imaging device are configured in a charge storage medium comprising a substrate that is curved in a selected nonplanar focal surface profile and located a selected distance from the lens with the focal surface facing the lens, the focal surface profile and lens-to-substrate distance selected such that the input light is in focus at the location of the substrate.

7. The imaging system of claim 1, further comprising a display connected to receive the output image frame sequence and to display the sequence at the frame rate, R.

8. The imaging system of claim 7, wherein the digital image processor and the display are in communication with but located remote from the charge-coupled imaging device and the analog-to-digital processor.

9. The imaging system of claim 7, further comprising a power supply, and wherein the charge-coupled imaging device, the analog-to-digital processor, and the display are in communication with but located remote from the digital image processor and the power supply.

10. The imaging system of claim 1 further comprising a communication link connected to the digital image processor for transmitting to a remotely located receiver the sequence of output image frames.

11. The imaging system of claim 1 further comprising a user controller for controlling the frame rate, R, and resolution of the imaging system, within operational limits of the imaging system.

12. The imaging system of claim 1 wherein the charge-coupled imaging device pixels are configured in a charge storage medium comprising a charge storage substrate having a front side supporting pixel interconnections and a back side having no substantial topology, buried channels in the substrate defining charge packet storage wells for the pixels, the charge-coupled imaging device located in relation to the optical input port such that input light from the scene impinges the back side of the substrate.

13. The imaging system of claim 12 wherein the charge storage substrate comprises a silicon substrate characterized by a resistivity of at least about 1000 Ω

-cm.

14. The imaging system of claim 12 further comprising a cooling device in contact with the charge storage substrate to suppress dark current charge packet generation in pixels of the charge-coupled imaging device.

15. The imaging system of claim 12 wherein the configuration of pixels in the charge storage substrate comprises an imaging pixel array on which input light impinges to produce charge packets in the buried channels of the imaging array pixels and a frame storage pixel array shielded from impinging input light, charge packets in the imaging pixel array being transferred to the frame storage pixel array for producing an electrical pixel signal of analog pixel values based on the input light.

16. The imaging system of claim 15 wherein the imaging pixel array comprises an array of electronically-shuttered pixels, each pixel in the array being selectively electronically controllable by the pixel interconnections to inhibit storage of charge packets in the buried channel of that pixel while input light impinges the charge-coupled imaging device.

17. The imaging system of claim 15 wherein the pixel interconnections supported on the front side of the charge storage medium define a three-phase clocking configuration for transferring charge packets in the buried channels.

18. The imaging system of claim 15 wherein the pixel interconnections supported on the front side of the charge storage substrate provide interconnections for selective electronic transfer of a charge packet in a given pixel of the imaging pixel array to an adjacent pixel located in a different row of the imaging pixel array, and provide interconnections for selective electronic transfer of a charge packet in a given pixel in the imaging array to an adjacent pixel located in a different column of the imaging pixel array.

20. A charge-coupled imaging device for imaging a scene to produce a sequence of image frames of the scene at an image frame rate, R, of at least about 25 image frames per second, the charge-coupled imaging device comprising:

a charge storage medium comprising a substrate having a front side and a backside;

an array of pixels defined in the charge storage substrate by pixel interconnections supported on the front side of the substrate, exposure of the substrate to light from the scene producing charge packets in the pixels, the pixel interconnections providing selective electronic temporal control of transfer of charge packets from one pixel to another in the substrate;

means for suppressing generation of dark current charge packet generation in the substrate pixels; and

an output circuit for converting the charge packets in the pixels to an electrical pixel signal of output pixel values based on the light from the scene, a plurality of pixel values together forming an image frame, the output circuit characterized by a capacitance of less than about 10 femtoFarads;

the output pixel values being characterized by noise-limited resolution of at least a minimum number, N_M, of line pairs per millimeter, referred to the charge-coupled imaging device pixel array, in an image frame configured of the output pixel values as a function of illuminance of the light from the scene impinging the charge-coupled imaging device pixels, wherein for a scene characterized by a contrast of about 0.3, for a human observation time of about 0.05 seconds, and for an image scene frame rate of about 30 frames per second, N_M is given as N_M =1900 L⁰.51, where L is the value of illuminance of the input light impinging the charge-coupled imaging device pixels, for at least one illuminance value between a range of illuminance values of about 1×

10^-2 LUX and 1×

10^-7 LUX.

19. The imaging system of claim 20 wherein charge packets resident in a portion of the pixels in the frame storage pixel array are summed prior to production of an electrical pixel signal of analog pixel values.

21. The charge-coupled imaging device of claim 20 further comprising an output pixel processor for amplifying the electrical pixel signal of output pixel values and for filtering electrical noise introduced into the signal by the output circuit.

22. The charge-coupled imaging device of claim 20 wherein the electrical pixel signal of output pixel values has a dynamic range of at least about 1000 distinct pixel value levels.

23. The charge-coupled imaging device of claim 22 wherein the electrical pixel signal of output pixel values has a dynamic range of at least about 3000 distinct pixel value levels.

24. The charge-coupled imaging device of claim 20 wherein the image scene frame rate, R, is at least about 30 frames per second.

25. The charge-coupled imaging device of claim 20 wherein the back side of the charge storage substrate has no substantial topology and the charge storage substrate is oriented such that light from the scene to be imaged impinges the back side of the charge storage substrate, and further comprising buried channels in the substrate to define charge packet storage wells for the pixels.

26. The charge-coupled imaging device of claim 25 wherein the charge storage substrate comprises a silicon substrate characterized by a resistivity of at least about 1000 Ω

-cm.

27. The charge-coupled imaging device of claim 25 wherein the pixel interconnections supported by the charge storage substrate define an imaging pixel array on which light from the scene to be imaged impinges to produce charge packets in the imaging array pixel buried channels, and a frame storage pixel array shielded from impinging light from the scene, charge packets in the imaging pixel array being transferred to the frame storage pixel array for producing the electrical pixel signal.

28. The imaging system of claim 27 wherein the imaging pixel array comprises an array of electronically-shuttered pixels, each pixel in the array being selectively electronically controllable by the pixel interconnections to inhibit storage of charge packets in the buried channel of that pixel while light from the scene impinges the charge-coupled imaging device.

29. The imaging system of claim 27 wherein the pixel interconnections supported on the front side of the charge storage substrate define a three-phase clocking configuration for transferring charge packets in the buried channels.

30. The imaging system of claim 27 wherein the pixel interconnections supported on the front side of the charge storage substrate provide interconnections for selective electronic transfer of a charge packet in a given pixel of the imaging pixel array to an adjacent pixel located in a different row of the imaging pixel array, and provide interconnections for selective electronic transfer of a charge packet in a given pixel in the imaging array to an adjacent pixel located in a different column of the imaging pixel array.

31. The imaging system of claim 27 wherein charge packets in a portion of the pixels in the frame storage pixel array are summed prior to production of the electrical pixel signal.

32. The charge-coupled imaging device of claim 20 wherein the means for suppressing dark current charge packet generation comprises a cooling device in contact with the charge storage substrate.

33. The charge-coupled imaging device of claim 32 wherein the cooling device comprises a thermoelectric cooling device.

34. An imaging system for imaging a scene to produce a sequence of color image frames of the scene at a frame rate, R, of at least about 25 image frames per second, the imaging system comprising:

an optical input port for accepting input light from the scene;

a spectral range separator located in re lation t o the input port to receive the input light, the spectral range separator separating the input light into distinct spectral ranges;

an imager system comprising at least one charge-coupled imager having an array of a plurality of pixels configured in a charge storage medium, the at least one charge-coupled imager being located in relation to the spectral range separator such that separated spectral ranges of input light impinge charge storage medium pixels, the at least one charge-coupled imager being characterized by monochromatic noise-limited resolution of at least a minimum number, N_M, of line pairs per millimeter, referred to the imager pixel array, in an imaged scene as a function of illuminance of input light impinging the pixel array, wherein for a scene characterized by a contrast of about 0.3, for a human observation time of about 0.05 seconds, and for a monochromatic image scene frame rate of about 30 frames per second, N_M is given as N_M =1900 L⁰.51, where L is the value of illuminance of input light impinging the pixel array, for at least one illuminance value between a range of illuminance values of about 1×

10^-2 LUX and 5×

10^-7 LUX, the imager system producing a plurality of electrical pixel signals of analog pixel values based on the spectral ranges of input light that impinge the pixels, each electrical pixel signal corresponding to a distinct spectral range;

an analog signal processor connected to receive each pixel signal for amplifying the pixel signals;

an analog-to-digital processor connected to the analog signal processor for digitizing the amplified pixel signals to produce corresponding digital image signals having a dynamic range of digital pixel values represented by a number of digital bits, B, where B is greater than 8; and

an image processor connected to receive the digital image signals and to combine together the digital image signals to produce a sequence of color image frames, each color image frame having pixel values based on the separated spectral ranges of input light, the sequence of color image frames being produced at the frame rate, R, with a latency of no more than about 1/R.

35. The imaging system of claim 34 wherein the imager system comprises a plurality of charge-coupled imagers, each imager designated for a distinct spectral range of the input light and located to receive at that imager'"'"'s pixels the corresponding distinct spectral range of input light separated by the spectral range separator to produce an electrical pixel signal of analog pixel values based on the corresponding spectral range of input light.

36. The imaging system of claim 35 wherein the spectral range separator comprises a dichroic beam splitter configured between the optical input port and the charge-coupled imagers.

37. The imaging system of claim 35 wherein the spectral range separator is adapted to separate the input light into a spectral range of visible light and a spectral range of near-infrared light;

wherein the imager system comprises a first charge-coupled imager designated for a spectral range of visible input light and located to receive at that imager'"'"'s pixels the separated visible input light from the spectral range separator to produce a visible image pixel signal and a second charge-coupled imager designated for a spectral range of near-infrared input light and located to receive at that imager'"'"'s pixels the separated near-infrared light from the spectral range separator to produce a near-infrared pixel signal;

and wherein the image processor is connected to receive digitized visible and near-infrared pixel signals and to fuse together the visible and near-infrared pixel signals to produce the sequence of color image frames.

39. The imaging system of claim 35 further comprising a color display connected to receive and display at the frame rate, R, the sequence of color image frames produced by the image processor;

and

wherein the charge-coupled imager designated for a red spectral range is connected as a red display color channel, the charge-coupled imager designated for a green spectral range is connected as a green display color channel, and the charge-coupled imager designated for a blue spectral range is connected as a blue display color channel.

40. The imaging system of claim 39 wherein the image processor is connected to independently control each of the red display color channel, the green display color channel, and the blue display color channel for independent scaling of each channel.

41. The imaging system of claim 34 wherein the imager system comprises a charge-coupled imager having clusters of pixels configured in a charge storage medium, each pixel cluster comprising a plurality of pixels that are each designated for a corresponding spectral range of input light;

and

wherein the spectral range separator is located in relation to the optical input port and the plurality of pixel clusters to direct to each pixel in the clusters the spectral range of the input light for which that pixel is designated.

42. The imaging system of claim 41 wherein the spectral range separator comprises a plurality of spectral filters each characterized by an ability to substantially transmit one spectral range that is substantially distinct from transmittable spectral ranges of the other filters, each spectral filter layered over at least one cluster pixel designated for a spectral range that substantially coincides with the transmittable spectral range of that spectral filter.

43. The imaging system of claim 42 wherein each pixel cluster comprises at least one pixel designated for a red spectral range, at least one pixel designated for a green spectral range, and at least one pixel designated for a blue spectral range;

and

wherein the plurality of spectral filters comprises a red spectral filter layered over pixels designated for the red spectral range and characterized by an ability to substantially transmit only the red spectral range, a green spectral filter layered over pixels designated for the green spectral range and characterized by an ability to substantially transmit only the green spectral range, and a blue spectral filter layered over pixels designated for the blue spectral range and characterized by an ability to substantially transmit only the blue spectral range.

44. The imaging system of claim 42 wherein each pixel cluster comprises at least one pixel designated for a visible spectral range and at least one pixel designated for a near-infrared spectral range;

and

wherein the plurality of spectral filters comprises a visible spectral filter layered over pixels designated for a visible spectral range and characterized by an ability to substantially transmit only a visible spectral range, and a near-infrared spectral filter layered over pixels designated for a near-infrared spectral range and characterized by an ability to substantially transmit only a near-infrared spectral range.

45. The imaging system of claim 41 wherein the spectral range separator comprises a plurality of diffractive element assemblies, each diffractive element assembly designated for a corresponding pixel cluster and configured between the input port and the corresponding pixel cluster to direct to each cluster pixel the input spectral range for which that pixel is designated.

46. The imaging system of claim 45 wherein each pixel cluster comprises pixels designated for a red spectral range, pixels designated for a green spectral range, and pixels designated for a blue spectral range;

and

wherein each diffractive element assembly is configured to direct a red spectral range of input light to cluster pixels designated for a red spectral range, to direct a green spectral range of input light to cluster pixels designated for a green spectral range, and to direct a blue spectral range of input light to cluster pixels designated for a blue spectral range.

47. The imaging system of claim 45 wherein each pixel cluster comprises pixels designated for a visible spectral range and pixels designated for a near-infrared spectral range;

and

wherein each diffractive element assembly is configured to direct a visible spectral range of input light to cluster pixels designated for a visible spectral range and to direct a near-infrared spectral range of input light to cluster pixels designated for a near-infrared spectral range.

38. The imaging system of claim 58 wherein the spectral range separator is adapted to separate the input light into a red spectral range of input light, a green spectral range of input light, and a blue spectral range of input light;

and

wherein the imager system comprises a first charge-coupled imager designated for a red spectral range and located to receive at that imager'"'"'s pixels the separated red input light from the spectral range separator, a second charge-coupled imager designated for a green spectral range and located to receive at that imager'"'"'s pixels the separated green input light from the spectral range separator, and a third charge-coupled imager designated for a blue spectral range and located to receive at that imager'"'"'s pixels the separated blue input light from the spectral range separator.