Compensating for sensor saturation and microlens modulation during light-field image processing

• US 9,386,288 B2
• Filed: 12/15/2014
• Issued: 07/05/2016
• Est. Priority Date: 02/28/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Akeley, Kurt Barton, Cabral, Brian, Pitts, Colvin, Liang, Chia-Kai, Wilburn, Bennett, Knight, Timothy James, Ng, Yi-Ren
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US14/571,103
• Publication Number:
  US 20150097985A1
• Time in Patent Office: 568 Days
• US Class Current: 1/1
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

According to various embodiments, the system and method of the present invention process light-field image data so as to reduce color artifacts, reduce projection artifacts, and/or increase dynamic range. These techniques operate, for example, on image data affected by sensor saturation and/or microlens modulation. Flat-field images are captured and converted to modulation images, and then applied on a per-pixel basis, according to techniques described herein.

Claims

1. A method for adjusting pixel sensitivities in an image capture device, comprising:

capturing a frame of light-field image data representing a scene, the light-field image data comprising a plurality of pixels, each pixel having a plurality of values associated with different colors;

for each color, determining a flat-field response contour for each of at least one region of an image sensor;

for each color, generating a modulation image based on the at least one flat-field response contour;

for each color, generating a demodulation image from the modulation image;

applying the generated demodulation images to the captured light-field image data to generate a demodulated light-field image;

applying a demosaicing operation to the demodulated light-field image;

applying an automatic white-balance correction algorithm to the demodulated light-field image;

estimating a chrominance of the scene illumination;

adjusting sensor gain individually for each color, based on the estimated chrominance;

capturing a subsequent frame of light-field image data, using the adjusted sensor gain for each of the colors; and

storing the captured subsequent frame.

2. The method of claim 1, further comprising iteratively repeating the steps of capturing a frame, estimating a chrominance, and adjusting sensor gain individually for each of the different colors based on the estimated chrominance.

3. A method for adjusting pixel sensitivities in an image capture device, comprising:

in a processor, receiving light-field image data representing a scene, the light-field image data comprising a plurality of pixels, each pixel having a plurality of values associated with different colors;

for each color, determining a flat-field response contour for each of at least one region of an image sensor;

for each color, generating a modulation image based on the at least one flat-field response contour; and

for each color, generating a demodulation image from the modulation image;

applying the generated demodulation images to the received light-field image data to generate a demodulated light-field image;

applying a demosaicing operation to the demodulated light-field image;

applying an automatic white-balance correction algorithm to the demodulated light-field image;

in a processor, estimating a chrominance of the scene illumination;

in the processor, based on the estimated chrominance, determining maximum sensor values for each of the different colors;

in the processor, clamping pixel values to corresponding maximum sensor values, to generate a processed image; and

outputting the processed image on a display device.

4. The method of claim 3, further comprising, subsequent to clamping the pixel values:

reapplying the generated demodulation images to the processed image generate a demodulated processed image; and

reapplying a demosaicing operation to the demodulated processed image.

5. A method for adjusting pixel sensitivities in an image capture device, comprising:

in a processor, receiving image data representing a scene, the image data comprising a plurality of pixels, each pixel having a plurality of values associated with different colors;

in the processor, estimating a chrominance of the scene illumination;

in the processor, based on the estimated chrominance, determining maximum sensor values for each of the different colors;

in the processor, clamping pixel values to corresponding maximum sensor values, to generate a processed image;

performing white-point adjustment on the processed image; and

outputting the processed image on a display device.

6. A method for adjusting pixel sensitivities in an image capture device, comprising:

in a processor, receiving image data representing a scene, the image data comprising a plurality of pixels, each pixel having a plurality of values associated with different colors;

in the processor, estimating a chrominance of the scene illumination;

in the processor, based on the estimated chrominance, determining maximum sensor values for each of the different colors;

in the processor, clamping pixel values to corresponding maximum sensor values, to generate a processed image;

converting the processed image to a chrominance component and a luminance component; and

outputting the converted image on a display device.

7. The method of claim 6, further comprising, subsequent to converting the processed image to a chrominance component and a luminance component, performing chrominance compensation to the chrominance component.

8. The method of claim 7, wherein performing chrominance compensation to the chrominance component comprises:

for each pixel in the image data, estimating the severity of the pixel'"'"'s saturation and the likelihood that the chrominance of the saturation matches the estimated chrominance of the scene illumination; and

responsive to the estimated severity of the pixel'"'"'s saturation exceeding a predetermined threshold severity level, and further responsive to the likelihood that the chrominance of the saturation matches the estimated chrominance of the scene illumination exceeding a predetermined threshold likelihood, replacing the chrominance of the pixel with the chrominance of the scene illumination.

9. The method of claim 8, wherein performing chrominance compensation to the chrominance component further comprises:

responsive to the estimated severity of the pixel'"'"'s saturation not exceeding the predetermined threshold severity level or the likelihood that the chrominance of the saturation matches the estimated chrominance of the scene illumination not exceeding a predetermined threshold likelihood, replacing the chrominance of the pixel with a linear interpolation between the original chrominance of the pixel and the chrominance of the scene illumination.

10. The method of claim 6, further comprising, subsequent to clamping the pixel values, performing spatial filtering on the processed image.

11. The method of claim 10, wherein performing spatial filtering on the processed image comprises:

applying a variable blur/sharpen filter to the chrominance component, based on a filter-control computation; and

applying a variable blur/sharpen filter to the luminance component, based on a filter-control computation.

12. The method of claim 6, further comprising, subsequent to clamping the pixel values, performing tone mapping on the processed image.

13. The method of claim 12, wherein performing tone mapping on the processed image comprises:

determining a luminance gain function;

determining a chrominance gain function;

for each pixel in the luminance component of the processed image, multiplying the pixel value by the luminance gain function; and

for each pixel in the chrominance component of the processed image, multiplying the pixel value by the chrominance gain function.

14. The method of claim 13, wherein determining each gain function comprises:

applying a blurring filter to the luminance component of the processed image, to generate a blurred image;

determining a histogram of luminance values from the blurred image;

computing a gain function from the determined histogram.

15. A non-transitory computer-readable medium for adjusting pixel sensitivities in an image capture device, comprising instructions stored thereon, that when executed by a processor, perform the steps of:

receiving image data representing a scene, the image data comprising a plurality of pixels, each pixel having a plurality of values associated with different colors;

for each color, determining a flat-field response contour for each of at least one region of an image sensor;

for each color, generating a modulation image based on the at least one flat-field response contour;

for each color, generating a demodulation image from the modulation image;

applying the generated demodulation images to the received light-field image data to generate a demodulated light-field image;

applying a demosaicing operation to the demodulated light-field image;

applying an automatic white-balance correction algorithm to the demodulated light-field image;

estimating a chrominance of the scene illumination;

based on the estimated chrominance, determining maximum sensor values for each of the different colors;

clamping pixel values to corresponding maximum sensor values, to generate a processed image;

causing a display device to output the processed image.

16. A system for adjusting pixel sensitivities in an image capture device, comprising:

circuitry configured to perform the steps of;

receiving image data representing a scene, the image data comprising a plurality of pixels, each pixel having a plurality of values associated with different colors;

for each color, determining a flat-field response contour for each of at least one region of an image sensor;

for each color, generating a modulation image based on the at least one flat-field response contour;

for each color, generating a demodulation image from the modulation image;

applying the generated demodulation images to the received light-field image data to generate a demodulated light-field image;

applying a demosaicing operation to the demodulated light-field image;

applying an automatic white-balance correction algorithm to the demodulated light-field image;

estimating a chrominance of the scene illumination;

based on the estimated chrominance, determining maximum sensor values for each of the different colors; and

clamping pixel values to corresponding maximum sensor values, to generate a processed image; and

a display device, communicatively coupled to the circuitry, configured to output the processed image.