COMPENSATING FOR SENSOR SATURATION AND MICROLENS MODULATION DURING LIGHT-FIELD IMAGE PROCESSING

• US 20130222652A1
• Filed: 02/22/2013
• Published: 08/29/2013
• Est. Priority Date: 02/28/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Akeley, Kurt Barton, Cabral, Brian, Pitts, Colvin, Liang, Chia-Kai, Knight, Timothy James, Ng, Yi-Ren, Wilburn, Bennett
• Granted Patent: US 8,948,545 B2
• Time in Patent Office: Days
• US Class Current: 348/241
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

According to various embodiments, the system and method of the present invention process light-field image data so as to reduce color artifacts, reduce projection artifacts, and/or increase dynamic range. These techniques operate, for example, on image data affected by sensor saturation and/or microlens modulation. Flat-field images are captured and converted to modulation images, and then applied on a per-pixel basis, according to techniques described herein.

Claims

1. In a light-field image capture device having a plurality of microlenses, a method for compensating for sensor saturation and microlens modulation, comprising:

in a processor, determining a flat-field response contour for each of at least one region of an image sensor;

in the processor, generating a modulation image based on the at least one flat-field response contour;

in the processor, generating a demodulation image from the modulation image;

in the processor, receiving light-field image data;

in the processor, applying the generated demodulation image to the received light-field image data to generate a demodulated light-field image; and

outputting the light-field demodulated image on a display device.

2. The method of claim 1, wherein determining each flat-field response contour comprises determining a flat-field response contour for a region corresponding to a microlens in the plurality of microlenses.

3. The method of claim 1, wherein determining each flat-field response contour comprises determining a flat-field response contour for a region projected by one of the microlenses.

4. The method of claim 1, wherein the at least one region of the image sensor comprises a plurality of pixels, wherein some of the pixels have different spectral sensitivities than others of the pixels, and wherein:

determining a flat-field response contour comprises determining a flat-field response contour for each of the different spectral sensitivities; and

generating a demodulation image comprises generating a demodulation image indicating demodulation values for each of the different spectral sensitivities.

5. The method of claim 4, wherein the pixels in the at least one region are arranged in a Bayer pattern.

6. The method of claim 1, wherein the at least one region of the image sensor comprises a plurality of pixels, wherein some of the pixels have different spectral sensitivities than others of the pixels, and wherein:

determining a flat-field response contour comprises determining a flat-field response contour for each of the different spectral sensitivities; and

generating a demodulation image comprises generating demodulation images for each of the different spectral sensitivities.

7. The method of claim 6, wherein the pixels in the at least one region are arranged in a Bayer pattern.

8. The method of claim 1, wherein generating a modulation image comprises generating a plurality of modulation images, each modulation image being associated with a set of parameters of the light-field image capture device.

9. The method of claim 8, wherein applying the generated demodulation image to the received light-field image data comprises:

determining a set of parameters for the received light-field image data;

selecting a generated modulation image corresponding to the determined parameters;

generating a demodulation image from the selected modulation image; and

applying the generated demodulation image to the received light-field image data.

10. The method of claim 8, wherein applying the generated demodulation image to the received light-field image data comprises:

determining a set of parameters for the received light-field image data;

generating an interpolated modulation image by interpolating between at least two generated modulation images based on the determined parameters;

generating a demodulation image from the generated modulation image; and

applying the generated demodulation image to the received light-field image data.

11. The method of claim 1, further comprising storing the generated modulation image in a storage device.

12. The method of claim 1, further comprising storing the generated demodulation image in a storage device.

13. The method of claim 1, wherein the modulation image comprises a plurality of pixels having values, and wherein generating a demodulation image from the modulation image comprises determining a reciprocal of each pixel value.

14. The method of claim 1, wherein the demodulated light-field image comprises a plurality of pixels, each pixel being associated with a plurality of values, each value corresponding to a different spectral range, the method further comprising applying a demosaicing operation to the demodulated light-field image.

15. The method of claim 14, wherein applying a demosaicing operation to the demodulated light-field image comprises, for each pixel in the demodulated light-field image, estimating at least one value based on at least one value of a nearby pixel.

16. The method of claim 1, further comprising:

estimating a scene illumination color; and

adjusting pixel values based on the estimated scene illumination color.

17. The method of claim 1, further comprising:

estimating a scene illumination color; and

for each pixel;

determining a probability that the pixel is saturated; and

steering a chrominance value for the pixel toward the estimated scene illumination color, wherein the steering is performed to an extent proportional to the determined probability.

18. The method of claim 1, wherein the demodulated light-field image comprises a plurality of pixels, each pixel being associated with a plurality of values, each value corresponding to a different spectral range, and wherein the method further comprises:

converting the demodulated light-field image to chrominance and luminance data.

19. The method of claim 1, further comprising, prior to outputting the light-field demodulated image, applying an automatic white-balance correction algorithm to the demodulated light-field image.

20. The method of claim 1, wherein receiving light-field image data for processing comprises receiving a light-field image comprising a plurality of pixels, each pixel having a chrominance value, the method further comprising, prior to outputting the light-field demodulated image,

calibrating the light-field image capture device with a plurality of illuminants;

storing, in a storage device, a chrominance associated with each illuminant;

for each pixel in the received light-field image;

determining a demosaiced chrominance value;

plotting the determined demosaiced chrominance value in Cartesian space;

calculating a vector between the plotted demosaiced chrominance value and each illuminant chrominance;

selecting an illuminant having a lowest-magnitude vector among the calculated vectors;

determining which illuminant was selected for the most pixels; and

applying an automatic white-balance correction algorithm, using the determined illuminant, to the light-field demodulated image.

21. A method for adjusting pixel sensitivities in an image capture device, comprising:

capturing a frame of image data representing a scene, the image data comprising a plurality of pixels, each pixel having a plurality of values associated with different colors;

estimating a chrominance of the scene illumination;

adjusting sensor gain individually for each of the different colors, based on the estimated chrominance;

capturing a subsequent frame of image data, using the adjusted sensor gain for each of the different colors; and

storing the captured frame.

22. The method of claim 21, further comprising iteratively repeating the steps of capturing a frame, estimating a chrominance, and adjusting sensor gain individually for each of the different colors based on the estimated chrominance.

23. The method of claim 21, wherein the image data comprises light-field image data, the method further comprising:

prior to capturing the frame of light-field image data;

for each color, determining a flat-field response contour for each of at least one region of an image sensor;

for each color, generating a modulation image based on the at least one flat-field response contour; and

for each color, generating a demodulation image from the modulation image; and

prior to estimating a chrominance of the scene illumination;

applying the generated demodulation images to the received light-field image data to generate a demodulated light-field image;

applying a demosaicing operation to the demodulated light-field image; and

applying an automatic white-balance correction algorithm to the demodulated light-field image.

24. A method for adjusting pixel sensitivities in an image capture device, comprising:

in a processor, receiving image data representing a scene, the image data comprising a plurality of pixels, each pixel having a plurality of values associated with different colors;

in the processor, estimating a chrominance of the scene illumination;

in the processor, based on the estimated chrominance, determining maximum sensor values for each of the different colors;

in the processor, clamping pixel values to corresponding maximum sensor values, to generate a processed image; and

outputting the processed image on a display device.

25. The method of claim 24, wherein the image data comprises light-field image data, the method further comprising:

for each color, determining a flat-field response contour for each of at least one region of an image sensor;

for each color, generating a modulation image based on the at least one flat-field response contour; and

for each color, generating a demodulation image from the modulation image; and

prior to estimating a chrominance of the scene illumination;

applying the generated demodulation images to the received light-field image data to generate a demodulated light-field image;

applying a demosaicing operation to the demodulated light-field image; and

applying an automatic white-balance correction algorithm to the demodulated light-field image.

26. The method of claim 25, further comprising, subsequent to clamping the pixel values:

reapplying the generated demodulation images to the processed image generate a demodulated processed image; and

reapplying a demosaicing operation to the demodulated processed image.

27. The method of claim 24, further comprising, subsequent to clamping the pixel values, performing white-point adjustment on the processed image.

28. The method of claim 24, further comprising, subsequent to clamping the pixel values, converting the processed image to a chrominance component and a luminance component.

29. The method of claim 28, further comprising, subsequent to converting the processed image to a chrominance component and a luminance component, performing chrominance compensation to the chrominance component.

30. The method of claim 29, wherein performing chrominance compensation to the chrominance component comprises:

for each pixel in the image data, estimating the severity of the pixel'"'"'s saturation and the likelihood that the chrominance of the saturation matches the estimated chrominance of the scene illumination; and

responsive to the estimated severity of the pixel'"'"'s saturation exceeding a predetermined threshold severity level, and further responsive to the likelihood that the chrominance of the saturation matches the estimated chrominance of the scene illumination exceeding a predetermined threshold likelihood, replacing the chrominance of the pixel with the chrominance of the scene illumination.

31. The method of claim 30, wherein performing chrominance compensation to the chrominance component further comprises:

responsive to the estimated severity of the pixel'"'"'s saturation not exceeding the predetermined threshold severity level or the likelihood that the chrominance of the saturation matches the estimated chrominance of the scene illumination not exceeding a predetermined threshold likelihood, replacing the chrominance of the pixel with a linear interpolation between the original chrominance of the pixel and the chrominance of the scene illumination.

32. The method of claim 28, further comprising, subsequent to clamping the pixel values, performing spatial filtering on the processed image.

33. The method of claim 32, wherein performing spatial filtering on the processed image comprises:

applying a variable blur/sharpen filter to the chrominance component, based on a filter-control computation; and

applying a variable blur/sharpen filter to the luminance component, based on a filter-control computation.

34. The method of claim 28, further comprising, subsequent to clamping the pixel values, performing tone mapping on the processed image.

35. The method of claim 34, wherein performing tone mapping on the processed image comprises:

determining a luminance gain function;

determining a chrominance gain function;

for each pixel in the luminance component of the processed image, multiplying the pixel value by the luminance gain function; and

for each pixel in the chrominance component of the processed image, multiplying the pixel value by the chrominance gain function.

36. The method of claim 35, wherein determining each gain function comprises:

applying a blurring filter to the luminance component of the processed image, to generate a blurred image;

determining a histogram of luminance values from the blurred image;

computing a gain function from the determined histogram.

37. A computer program product for compensating for sensor saturation and microlens modulation in a light-field image capture device having a plurality of microlenses, comprising:

a non-transitory computer-readable storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

determining a flat-field response contour for each of at least one region of an image sensor;

generating a modulation image based on the at least one flat-field response contour;

generating a demodulation image from the modulation image;

receiving light-field image data;

applying the generated demodulation image to the received light-field image data to generate a demodulated light-field image; and

causing a display device to output the light-field demodulated image.

38. The computer program product of claim 37, wherein the computer program code configured to cause at least one processor to generate a modulation image comprises computer program code configured to cause at least one processor to generate a plurality of modulation images, each modulation image being associated with a set of parameters of the light-field image capture device.

39. The computer program product of claim 37, wherein the modulation image comprises a plurality of pixels having values, and wherein the computer program code configured to cause at least one processor to generate a demodulation image from the modulation image comprises computer program code configured to cause at least one processor to determine a reciprocal of each pixel value.

40. The computer program product of claim 37, wherein the demodulated light-field image comprises a plurality of pixels, each pixel being associated with a plurality of values, each value corresponding to a different spectral range, the computer program product further comprising computer program code configured to cause at least one processor to apply a demosaicing operation to the demodulated light-field image.

41. The computer program product of claim 37, further comprising computer program code configured to cause at least one processor to perform the steps of:

estimating a scene illumination color; and

adjusting pixel values based on the estimated scene illumination color.

42. The computer program product of claim 37, further comprising computer program code configured to cause at least one processor to perform the steps of:

estimating a scene illumination color; and

for each pixel;

determining a probability that the pixel is saturated; and

steering a chrominance value for the pixel toward the estimated scene illumination color, wherein the steering is performed to an extent proportional to the determined probability.

43. The computer program product of claim 37, wherein the demodulated light-field image comprises a plurality of pixels, each pixel being associated with a plurality of values, each value corresponding to a different spectral range, and wherein the computer program product further comprises computer program code configured to cause at least one processor to convert the demodulated light-field image to chrominance and luminance data.

44. The computer program product of claim 37, further comprising computer program code configured to cause at least one processor to, prior to causing a display device to output the light-field demodulated image, apply an automatic white-balance correction algorithm to the demodulated light-field image.

45. A computer program product for adjusting pixel sensitivities in an image capture device, comprising:

a non-transitory computer-readable storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

capturing a frame of image data representing a scene, the image data comprising a plurality of pixels, each pixel having a plurality of values associated with different colors;

estimating a chrominance of the scene illumination;

adjusting sensor gain individually for each of the different colors, based on the estimated chrominance;

capturing a subsequent frame of image data, using the adjusted sensor gain for each of the different colors; and

storing the captured frame.

46. A computer program product for adjusting pixel sensitivities in an image capture device, comprising:

a non-transitory computer-readable storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

receiving image data representing a scene, the image data comprising a plurality of pixels, each pixel having a plurality of values associated with different colors;

estimating a chrominance of the scene illumination;

based on the estimated chrominance, determining maximum sensor values for each of the different colors;

clamping pixel values to corresponding maximum sensor values, to generate a processed image; and

causing a display device to output the processed image.

47. The computer program product of claim 46, wherein the image data comprises light-field image data, the computer program product further comprising computer program code configured to cause at least one processor to perform the steps of:

for each color, determining a flat-field response contour for each of at least one region of an image sensor;

for each color, generating a modulation image based on the at least one flat-field response contour; and

for each color, generating a demodulation image from the modulation image; and

prior to estimating a chrominance of the scene illumination;

applying the generated demodulation images to the received light-field image data to generate a demodulated light-field image;

applying a demosaicing operation to the demodulated light-field image; and

applying an automatic white-balance correction algorithm to the demodulated light-field image.

48. A system for compensating for sensor saturation and microlens modulation in a light-field image capture device having a plurality of microlenses, comprising:

circuitry configured to perform the steps of;

determining a flat-field response contour for each of at least one region of an image sensor;

generating a modulation image based on the at least one flat-field response contour;

generating a demodulation image from the modulation image;

receiving light-field image data; and

applying the generated demodulation image to the received light-field image data to generate a demodulated light-field image; and

a display device, communicatively coupled to the circuitry, configured to output the light-field demodulated image.

49. The system of claim 48, wherein the circuitry configured to generate a modulation image comprises circuitry configured to generate a plurality of modulation images, each modulation image being associated with a set of parameters of the light-field image capture device.

50. The system of claim 48, wherein the modulation image comprises a plurality of pixels having values, and wherein the circuitry configured to generate a demodulation image from the modulation image comprises circuitry configured to determine a reciprocal of each pixel value.

51. The system of claim 48, wherein the demodulated light-field image comprises a plurality of pixels, each pixel being associated with a plurality of values, each value corresponding to a different spectral range, the system further comprising circuitry configured to apply a demosaicing operation to the demodulated light-field image.

52. The system of claim 48, further comprising circuitry configured to perform the steps of:

estimating a scene illumination color; and

adjusting pixel values based on the estimated scene illumination color.

53. The system of claim 48, further comprising circuitry configured to perform the steps of:

estimating a scene illumination color; and

for each pixel;

determining a probability that the pixel is saturated; and

steering a chrominance value for the pixel toward the estimated scene illumination color, wherein the steering is performed to an extent proportional to the determined probability.

54. The system of claim 48, wherein the demodulated light-field image comprises a plurality of pixels, each pixel being associated with a plurality of values, each value corresponding to a different spectral range, and wherein the system further comprises circuitry configured to convert the demodulated light-field image to chrominance and luminance data.

55. The system of claim 48, further comprising circuitry configured to, prior to causing a display device to output the light-field demodulated image, apply an automatic white-balance correction algorithm to the demodulated light-field image.

56. A system for adjusting pixel sensitivities in an image capture device, comprising:

circuitry configured to perform the steps of;

capturing a frame of image data representing a scene, the image data comprising a plurality of pixels, each pixel having a plurality of values associated with different colors;

estimating a chrominance of the scene illumination;

adjusting sensor gain individually for each of the different colors, based on the estimated chrominance; and

capturing a subsequent frame of image data, using the adjusted sensor gain for each of the different colors; and

a storage device, communicatively coupled to the circuitry, configured to store the captured frame.

57. A system for adjusting pixel sensitivities in an image capture device, comprising:

circuitry configured to perform the steps of;

receiving image data representing a scene, the image data comprising a plurality of pixels, each pixel having a plurality of values associated with different colors;

estimating a chrominance of the scene illumination;

based on the estimated chrominance, determining maximum sensor values for each of the different colors; and

clamping pixel values to corresponding maximum sensor values, to generate a processed image; and

a display device, communicatively coupled to the circuitry, configured to output the processed image.

58. The system of claim 57, wherein the image data comprises light-field image data, the system further comprising circuitry configured to to perform the steps of:

for each color, determining a flat-field response contour for each of at least one region of an image sensor;

for each color, generating a modulation image based on the at least one flat-field response contour; and

for each color, generating a demodulation image from the modulation image; and

prior to estimating a chrominance of the scene illumination;

applying the generated demodulation images to the received light-field image data to generate a demodulated light-field image;

applying a demosaicing operation to the demodulated light-field image; and

applying an automatic white-balance correction algorithm to the demodulated light-field image.