Method and apparatus to extend the effective dynamic range of an image sensing device and use residual images

• US 20020130957A1
• Filed: 01/24/2001
• Published: 09/19/2002
• Est. Priority Date: 01/24/2001
• Status: Active Grant

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Gallagher, Andrew C., Spaulding, Kevin E.
• Granted Patent: US 6,943,831 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: H01L 27/14601 Structural or functional details thereof
• CPC Class Codes: H01L 27/14621 Colour filter arrangements
• CPC Class Codes: H04N 1/64 Systems for the transmission or the storage of the colour picture signal; Details therefor, e.g. coding or decoding means therefor H04N19/00 takes precedence
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/585 with pixels having different sensitivities within the sensor, e.g. fast or slow pixels or pixels having different sizes
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 9/804 involving pulse code modulation of the colour picture signal components

Abstract

An image capture system for generating and storing an extended dynamic range digital image, includes a sparsely sampled extended dynamic range image sensing device having fast photosites with a predetermined response to light exposure interspersed with slow photosites with a slower response to the same light exposure for producing a sparsely sampled high resolution digital image having fast pixel values produced by the fast photosites and slow pixel values produced by the slow photosites; a digital image processor that employs the slow pixel values to expand the dynamic range of the fast pixel values in the sparsely sampled high resolution digital image to form a full resolution digital image having an extended dynamic range; a color encoder for reducing the dynamic range of the full resolution digital image to fit within the dynamic range of a storage color space having a dynamic range less than the dynamic range of the full resolution digital image to form a limited dynamic range digital image represented in the storage color space and for producing a residual image representing a difference between the full resolution digital image and the limited dynamic range digital image that can be used with the limited dynamic range digital image to reconstruct the full resolution digital image; and a digital image store for storing the limited dynamic range digital image in association with the residual image.

Claims

1. An image capture system for generating and storing an extended dynamic range digital image, comprising:

a) a image sensing device having fast photosites with a predetermined response to light exposure interspersed with slow photosites with a slower response to the same light exposure for producing a sparsely sampled high resolution digital image having fast pixel values produced by the fast photosites and slow pixel values produced by the slow photosites;

b) a digital image processor that employs the slow pixel values to expand the dynamic range of the fast pixel values in the sparsely sampled high resolution digital image to form a full resolution digital image having an extended dynamic range;

c) an encoder for reducing the dynamic range of the full resolution digital image to fit within the dynamic range of a storage space having a dynamic range less than the dynamic range of the full resolution digital image to form a limited dynamic range digital image represented in the storage space and for producing a residual image representing a difference between the full resolution digital image and the limited dynamic range digital image that can be used with the limited dynamic range digital image to reconstruct the full resolution digital image; and

d) a digital image store for storing the limited dynamic range digital image in association with the residual image.

2. The image capture system claimed in claim 1, wherein the image sensor is a color image sensor having an array of photosites and a color filter array arranged over the array of photosites;

wherein the digital image processor includes a color filter array interpolator for interpolating color values at all of the photosites; and

wherein the storage space is a storage color space.

3. The image capture system claimed in claim 1, wherein the digital image processor also employs the fast pixel values to expand the dynamic range of the slow pixel values in the sparsely sampled high resolution digital image.

4. The image capture system claimed in claim 1, wherein the image sensing device is located in a digital camera and the digital image processor is located in a host computer separate from the digital camera.

5. The image capture system claimed in claim 1, wherein the image sensing device and the digital image processor are included in a digital camera.

6. The image capture system claimed in claim 1, wherein the slow photosites have a response that is slower by at least one stop compared to the fast photosites.

7. The system claimed in claim 2, wherein the color filter array is a Bayer array.

8. The image capture system claimed in claim 1, further comprising an image reconstructor that employs the residual image and the limited dynamic range digital image to form a reconstructed full resolution digital image.

9. The image capture system claimed in claim 8, wherein the image reconstructor applies an image modification to the reconstructed full resolution digital image.

10. The image capture system claimed in claim 1, wherein the limited dynamic range digital image is stored by the image store in a digital image file which includes the residual image as additional data in the digital image file.

11. The image capture system as claimed in claim 1, wherein the limited dynamic range digital image and the residual image are stored by the image store in separate associated digital image files.

13. The method as claimed in claim 12, further comprising the step of expanding the dynamic range of the slow pixel values by utilizing neighboring fast pixel values.

14. The method as claimed in claim 13, wherein the step of expanding the respective response of the fast and slow pixel values, comprises the steps of:

a) comparing fast pixel values against a high exposure response threshold and replacing the fast pixel values exceeding the high exposure response threshold with pixel values that are a function of neighboring slow pixel values; and

b) comparing slow pixel values against a low exposure response threshold and replacing the slow pixel values less than the low exposure response threshold with pixel values that are a function of neighboring fast pixel values.

15. The method claimed in claim 12, wherein a residual image representing a luminance difference and a residual image representing a chrominance difference are produced and stored.

16. The method claimed in claim 12, wherein the residual image is stored as tiles representing subsets of pixels in separate tags in an image file.

12. A method for generating and storing an extended dynamic range digital image, comprising:

a) employing an image sensing device having fast photosites with a predetermined response to light exposure interspersed with slow photosites with a slower response to the same light exposure to produce a sparsely sampled high resolution digital image having fast pixel values produced by the fast photosites and slow pixel values produced by the slow photosites;

b) expanding the dynamic range of the fast pixel values in the sparsely sampled high resolution digital image to form a full resolution digital image having an extended dynamic range;

c) reducing the dynamic range of the full resolution digital image to fit within the dynamic range of a storage space having a dynamic range less than the dynamic range of the full resolution digital image to form a limited dynamic range digital image represented in the storage space and for producing a residual image representing a difference between the full resolution digital image and the limited dynamic range digital image that can be used with the limited dynamic range digital image to reconstruct the full resolution digital image; and

d) storing the limited dynamic range digital image in association with the residual image.

17. An image capture system for generating and storing an extended dynamic range digital image, comprising:

a) a image sensing device having fast photosites with a predetermined response to light exposure interspersed with slow photosites with a slower response to the same light exposure for producing a sparsely sampled high resolution digital image having fast pixel values produced by the fast photosites and slow pixel values produced by the slow photosites;

b) a digital image processor that employs the slow pixel values to expand the dynamic range of the fast pixel values in the sparsely sampled high resolution digital image to form a full resolution digital image having an extended dynamic range;

c) an encoder for reducing the dynamic range of the full resolution digital image to fit within the dynamic range of a storage space having a dynamic range less than the dynamic range of the full resolution digital image to form a limited dynamic range digital image represented in the storage space and for producing a residual image representing a difference between the sparsely sampled high resolution digital image and the limited dynamic range digital image that can be used with the limited dynamic range digital image to reconstruct the sparsely sampled high resolution digital image; and

d) a digital image store for storing the limited dynamic range digital image in association with the residual image.