Apparatus and method for capturing still images and video using coded aperture techniques

• US 7,767,950 B2
• Filed: 09/06/2007
• Issued: 08/03/2010
• Est. Priority Date: 01/18/2005
• Status: Active Grant

• Current Assignee: Rearden, LLC
• Original Assignee: Rearden, LLC
• Inventors: Busboom, Axel, Perlman, Stephen G., Cotter, Tim S., van der Laan, Roger, Speck, John
• Primary Examiner(s): Luu; Thanh X
• Application Number:
  US11/899,814
• Publication Number:
  US 20080001069A1
• Time in Patent Office: 1,062 Days
• Field of Search: 250/208.1, 250/216, 348/14.01
• US Class Current: 250/208.1
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G02B 2207/129 Coded aperture imaging
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise

Abstract

A system is described for capturing images comprising: a display for displaying graphical images and text; a plurality of apertures formed in the display; an image detector array configured behind the display and configured to sense light transmitted through the apertures in the display, the light reflected from a subject positioned in front of the display; and image processing logic to generate image data using the light transmitted through the apertures, the image data representing an image of a subject.

Claims

1. A data processing system for capturing images comprising:

a plurality of apertures arranged according to a coded aperture mask pattern, the coded aperture mask pattern comprising an arrangement of apertures suitable for image reconstruction by means of Coded Aperture Imaging (“

CAI”

) decoding, the apertures arranged in a first dimension and a second dimension and coupled together to form a coded pattern cyclically repeated at least once in either dimension or both dimensions, wherein the coded pattern is arranged to cause an overlapping of images projected through the apertures onto a light-sensitive image detector array;

the light sensitive image detector array positioned behind the apertures and configured to sense light within the visible spectrum transmitted through the apertures, the light received from a substantially unconstrained environment in front of the apertures;

the apertures having a specified width, height and thickness to establish maximum angles at which the visible light from the unconstrained environment can pass through the coded aperture mask pattern in the first dimension and the second dimension and reach the light-sensitive image detector array;

a readout subsystem comprised of an analog-to-digital (“

A/D”

) converter having a specified dynamic range, the A/D converter configured to receive an analog signal from the image detector array and to responsively convert the analog signal to a digital signal, the analog signal comprising an analog representation of light transmitted through the apertures and the digital signal comprising a digital representation of light transmitted through the apertures

wherein the readout subsystem further comprises logic and/or circuitry electrically coupled to the light sensitive image detector array and the A/D converter, the logic and/or circuitry to apply zero offset and gain values to analog signals output from the light sensitive image detector array, the values of zero offset and gain selected based on the specified dynamic range of the A/D converter; and

digital image processing logic to process the digital signal and generate a reconstructed image of the unconstrained environment by reduction of crosstalk from objects in the scene at multiple ranges.

2. The data processing system as in claim 1 wherein the image detector array is a Complementary Metal Oxide Semiconductor (“

CMOS”

) array comprised of a plurality of CMOS image sensors.

3. The data processing system as in claim 1 wherein the image detector array is comprised of a plurality of Charge Coupled Devices (“

CCDs”

).

4. The data processing system as in claim 1 wherein the image processing logic comprises CAI decoder logic.

5. The data processing system as in claim 4 wherein the apertures are arranged in a Modified Uniformly Redundant Array (“

MURA”

) pattern and wherein the CAI decoder logic comprises MURA decoder logic.

6. The data processing system as in claim 4 wherein the apertures are arranged in a Uniformly Redundant Array (“

URA”

) pattern and wherein the CAI decoder logic comprises URA decoder logic.

7. The data processing system as in claim 4 wherein the apertures are arranged in a Perfect Binary Array (“

PBA”

) pattern and wherein the CAI decoder logic comprises PBA decoder logic.

8. The data processing system as in claim 1 further comprising:

a graphics processing unit to reproduce the image on a display device using the image data generated by the image processing logic.

9. The data processing system as in claim 1 further comprising:

a storage device to store the image data for later retrieval.

10. The data processing system as in claim 1 wherein the digital image processing logic further, comprises:

an image reconstruction processor to reconstruct the image into a grayscale or a color image.

11. The data processing system as in claim 1 implemented within a portable data processing device, the portable data processing device having both wireless telephony capabilities and data processing capabilities.

12. The data processing system as in claim 1 wherein the specified width, height and thickness are sized to limit the field of view (FOV) to be greater than or equal to a fully-coded FOV projected onto the light-sensitive image detector array.

13. The data processing system as in claim 1 wherein intensity is adjusted at different regions of the reconstructed image by multiplying each pixel of the reconstructed image by an inverse of a baffle attenuation value the pixel was subjected to.

14. The data processing system as in claim 1 further comprising:

a noise-reducing smoothing filter having a filter characteristic adjusted for each pixel or each region of the reconstructed image according to signal-to-noise ratio variations within the reconstructed image.

15. The data processing system as in claim 1 wherein the light-sensitive image detector array comprises a color filter array (“

CFA”

) for capturing color images.

16. The data processing system as in claim 1 wherein to generate the reconstructed image, the digital image processing logic performs the additional operation of calculating a periodic cross-correlation function of the digital signal received from the A/D converter with a periodic inverse filter.

17. A data processing system for capturing images comprising:

a plurality of apertures arranged according to a coded aperture mask pattern, the coded aperture mask pattern comprising an arrangement of apertures suitable for image reconstruction by means of Coded Aperture Imaging (“

CAI”

) decoding, the apertures arranged in a first dimension and a second dimension and coupled together to form a coded pattern cyclically repeated at least once in either dimension or both dimensions, wherein the coded pattern is arranged to cause an overlapping of images projected through the apertures onto a light-sensitive image detector array;

the light sensitive image detector array positioned behind the apertures and configured to sense light within the visible spectrum transmitted through the apertures, the light received from a substantially unconstrained environment in front of the apertures;

the apertures having a specified width, height and thickness to limit a field of view (FOV) of the unconstrained environment to be equal to or greater than a fully-coded FOV projected onto the light-sensitive image detector array;

a readout subsystem comprised of an analog-to-digital (“

A/D”

) converter having a specified dynamic range, the A/D converter configured to receive an analog signal from the image detector array and to responsively convert the analog signal to a digital signal, the analog signal comprising an analog representation of light transmitted through the apertures and the digital signal comprising a digital representation of light transmitted through the apertures

wherein the readout subsystem further comprises logic and/or circuitry electrically coupled to the light sensitive image detector array and the A/D converter, the logic and/or circuitry to apply zero offset and gain values to analog signals output from the light sensitive image detector array, the values of zero offset and gain selected based on the specified dynamic range of the A/D converter; and

digital image processing logic to process the digital signal and generate a reconstructed image of the unconstrained environment by reduction of crosstalk from objects in the scene at multiple ranges.

18. The data processing system as in claim 17 wherein intensity is adjusted at different regions of the reconstructed image by multiplying each pixel of the reconstructed image by an inverse of a baffle attenuation value the pixel was subjected to.

19. The data processing system as in claim 17 further comprising:

a noise-reducing smoothing filter having a filter characteristic adjusted for each pixel or each region of the reconstructed image according to signal-to-noise ratio variations within the reconstructed image.

20. The data processing system as in claim 17 wherein the light-sensitive image detector array comprises a color filter array (“

CFA”

) for capturing color images.

21. The data processing system as in claim 17 wherein the readout subsystem further comprises logic and/or circuitry electrically coupled to the light sensitive image detector array and the A/D converter, the logic and/or circuitry to apply zero offset and gain values to analog signals output from the light sensitive image detector array, the values of zero offset and gain selected based on the specified dynamic range of the A/D converter.

22. The data processing system as in claim 17 wherein to generate the reconstructed image, the digital image processing logic performs the additional operation of calculating a periodic cross-correlation function of the digital signal received from the A/D converter with a periodic inverse filter.

23. An apparatus for coding images comprising:

a light-sensitive sensor configured to detect visible light representing a scene; and

a plurality of apertures arranged according to a coded aperture mask pattern, the coded aperture mask pattern comprising an arrangement of apertures suitable for image reconstruction by means of Coded Aperture Imaging (“

CAI”

) decoding, the apertures arranged in a first dimension and a second dimension and coupled together to form a coded pattern cyclically repeated at least once in either dimension or both dimensions, wherein the coded pattern is arranged to cause an overlapping of images projected through the apertures onto a light-sensitive image detector array;

an analog to digital converter electrically coupled to the light sensitive sensor, the A/D converter to convert an analog signal representing visible light detected by the light-sensitive sensor to a digital signal;

an image sensor readout subsystem electrically coupled to the A/D converter and the light-sensitive sensor, the image sensor readout subsystem to dynamically adjust zero offset and gain applied to pixel values read from the light-sensitive sensor by performing the operations of;

selecting an initial zero offset value and an initial gain value;

determining whether any pixel values output from the A/D converter have a zero value and, if so, then decreasing a current zero offset value by a zero offset step value; and

determining whether any pixel values output from the A/D converter have a maximum output value and, if so, then decreasing a current gain value by a gain step value.

24. The apparatus as in claim 23 wherein the image sensor readout subsystem performs the additional operations of:

if there are no pixel values output from the A/D converter which have a zero value, then determining if the current zero offset step value is a minimum possible zero offset step value and, if not, then adding the current zero offset step value to the current zero offset value and decreasing the current zero offset step value to produce a Previously Presented zero offset step value; and

determining again whether any pixel values output from the A/D converter have a zero value and, if so, then decreasing a current zero offset value by the Previously Presented zero offset step value.

25. The apparatus as in claim 24 wherein the image sensor readout subsystem performs the additional operations of:

if there are no pixel values output from the A/D converter which have a maximum output value, then determining if the current gain step value is a minimum possible gain step value and, if not, then adding the current gain step value to the current gain value and decreasing the current gain step value to produce a Previously Presented gain step value; and

determining again whether any pixel values output from the A/D converter have a maximum output value and, if so, then decreasing a current gain value by the Previously Presented gain step value.

26. The apparatus as in claim 23 wherein the coded aperture mask pattern is a Uniformly Redundant Array (URA) pattern.

27. The apparatus as in claim 23 wherein the coded aperture mask pattern is a Modified Uniformly Redundant Array (MURA) pattern.

28. The apparatus as in claim 23 wherein the coded aperture mask pattern is a random pattern.

29. The apparatus as in claim 23 further comprising:

collimating light passing between the coded aperture mask pattern and the light-sensitive semiconductor sensor.

30. The apparatus as in claim 29 wherein the collimating is performed using a prismatic film or a honeycomb collimator.