Automatic lens flare detection and correction for light-field images

• US 10,205,896 B2
• Filed: 04/30/2018
• Issued: 02/12/2019
• Est. Priority Date: 07/24/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Kuang, Jiangtao, Liang, Chia-Kai
• Primary Examiner(s): Chen, Chia Wei A
• Application Number:
  US15/967,076
• Publication Number:
  US 20180255254A1
• Time in Patent Office: 288 Days
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

According to various embodiments, the system and method disclosed herein process light-field image data so as to mitigate lens flare effects. A light-field image may be captured with a light-field image capture device with a microlens array and received in a data store. A plurality of flare-affected pixels may be identified in the light-field image. The flare-affected pixels may have flare-affected pixel values. Flare-corrected pixel values may be generated for the flare-affected pixels. Relative to the flare-affected pixel values, the flare-corrected pixel values may at least partially remove the lens flare effects. The flare-corrected pixel values may be used to generate a corrected light-field image in which the lens flare effects are at least partially corrected. The corrected light-field image may be displayed on a display screen.

Claims

1. A method for correcting a light-field image to remove effects of a first lens flare, the method comprising:

in a data store, receiving the light-field image after the light-field image has been captured by a light-field image capture device having a sensor and a plurality of microlenses;

in a processor, identifying a plurality of flare-affected pixels of the light-field image that have been affected by the first lens flare, wherein the flare-affected pixels have flare-affected pixel values, wherein identifying the plurality of flare-affected pixels of the light-field image comprises;

identifying a plurality of maximum value pixels, each of which has a pixel value higher than neighboring pixel values of neighboring pixels;

generating a global parametric flare model of effects of the first lens flare based on locations of the maximum value pixels; and

using the global parametric flare model to compute a probability that the pixel value of each of the maximum value pixels has been affected by the first lens flare;

in the processor, generating flare-corrected pixel values for the flare-affected pixels, wherein, relative to the flare-affected pixel values, the flare-corrected pixel values at least partially remove effects of the first lens flare; and

in the processor, using the flare-corrected pixel values to generate a corrected light-field image in which effects of the first lens flare are at least partially corrected.

2. The method of claim 1, wherein generating the global parametric flare model of effects of the first lens flare based on locations of the maximum value pixels comprises generating a simple affine model that is predictive of the locations of the maximum value pixels.

3. The method of claim 1, wherein generating flare-corrected pixel values for the flare-affected pixels comprises, for each of the flare-affected pixels, using neighboring pixel values to interpolate the flare-corrected pixel value for the flare-affected pixel.

4. The method of claim 3, wherein generating flare-corrected pixel values for the flare-affected pixels further comprises:

identifying at least one edge-located pixel of the flare-affected pixels, wherein each edge-located pixel is located proximate an edge depicted in the light-field image; and

identifying a plurality of edge-located neighboring pixels that are located proximate the edge;

wherein, for each of the edge-located pixels, using the neighboring pixel values to interpolate the flare-corrected pixel value comprises using exclusively the neighboring pixel values of the edge-located neighboring pixels to interpolate the flare-corrected pixel value.

5. The method of claim 1, wherein using the flare-corrected pixel values to generate a corrected light-field image comprises, for each of the flare-affected pixels, blending the flare-affected pixel value with the flare-corrected pixel value, with the flare-corrected pixel value weighted in proportion to the probability that the pixel value of the maximum value pixel has been affected by the first lens flare.

6. The method of claim 1, wherein generating flare-corrected pixel values for the flare-affected pixels comprises, for each of the maximum value pixels, assigning a weight, inversely proportional to the probability that the pixel value of the maximum value pixel has been affected by the first lens flare;

wherein generating the flare-corrected pixel values comprises processing the light-field image with at least one process in which each of the flare-affected pixels is excluded from processing in proportion to the weight assigned to it to obtain the flare-corrected pixel values.

7. The method of claim 1, wherein using the flare-corrected pixel values to generate the corrected light-field image comprises:

using the flare-corrected pixel values to generate an intermediate light-field image in which effects of the first lens flare are at least partially corrected;

identifying a plurality of intermediate flare-affected pixels of the intermediate light-field image that have been affected by a selection from the group consisting of the first lens flare and a second lens flare, wherein the intermediate flare-affected pixels have intermediate flare-affected pixel values;

generating intermediate flare-corrected pixel values for the intermediate flare-corrected pixels, wherein, relative to the intermediate flare-affected pixel values, the intermediate flare-corrected pixel values at least partially remove effects of the selection; and

using the intermediate flare-corrected pixel values to generate the corrected light-field image.

8. The method of claim 7, wherein the selection comprises the second lens flare.

9. The method of claim 1, further comprising, in the processor, prior to identifying the plurality of flare-affected pixels, at an input device, receiving user input comprising a parameter;

wherein at least one of identifying the plurality of flare-affected pixels, generating the flare-corrected pixel values for the flare-affected pixels, and using the flare-corrected pixel values to generate the corrected light-field image comprises using the parameter.

10. The method of claim 9, wherein the parameter comprises a designation of a subset of the light-field image;

wherein using the parameter comprises identifying the plurality of flare-affected pixels exclusively within the subset.

11. A non-transitory computer-readable medium for correcting a light-field image to remove effects of a first lens flare, comprising instructions stored thereon, that when executed by a processor, perform the steps of:

causing a data store to receive the light-field image after the light-field image has been captured by a light-field image capture device having a sensor and a plurality of microlenses;

identifying a plurality of flare-affected pixels of the light-field image that have been affected by the first lens flare, wherein the flare-affected pixels have flare-affected pixel values, wherein identifying the plurality of flare-affected pixels of the light-field image comprises;

identifying a plurality of maximum value pixels, each of which has a pixel value higher than neighboring pixel values of neighboring pixels;

generating a global parametric flare model of effects of the first lens flare based on locations of the maximum value pixels; and

using the global parametric flare model to compute a probability that the pixel value of each of the maximum value pixels has been affected by the first lens flare;

wherein generating the global parametric flare model of effects of the first lens flare based on locations of the maximum value pixels comprises generating a simple affine model that is predictive of the locations of the maximum value pixels;

generating flare-corrected pixel values for the flare-affected pixels, wherein, relative to the flare-affected pixel values, the flare-corrected pixel values at least partially remove effects of the first lens flare; and

using the flare-corrected pixel values to generate a corrected light-field image in which effects of the first lens flare are at least partially corrected.

12. The non-transitory computer-readable medium of claim 11, wherein generating flare-corrected pixel values for the flare-affected pixels comprises, for each of the flare-affected pixels, using neighboring pixel values to interpolate the flare-corrected pixel value for the flare-affected pixel.

13. The non-transitory computer-readable medium of claim 12, wherein generating flare-corrected pixel values for the flare-affected pixels further comprises:

identifying at least one edge-located pixel of the flare-affected pixels, wherein each edge-located pixel is located proximate an edge depicted in the light-field image; and

identifying a plurality of edge-located neighboring pixels that are located proximate the edge;

wherein, for each of the edge-located pixels, using the neighboring pixel values to interpolate the flare-corrected pixel value comprises using exclusively the neighboring pixel values of the edge-located neighboring pixels to interpolate the flare-corrected pixel value.

14. The non-transitory computer-readable medium of claim 11, wherein generating flare-corrected pixel values for the flare-affected pixels comprises, for each of the maximum value pixels, assigning a weight, inversely proportional to the probability that the pixel value of the maximum value pixel has been affected by the first lens flare;

wherein generating the flare-corrected pixel values comprises processing the light-field image with at least one process in which each of the flare-affected pixels is excluded from processing in proportion to the weight assigned to it to obtain the flare-corrected pixel values.

15. The non-transitory computer-readable medium of claim 11, wherein using the flare-corrected pixel values to generate the corrected light-field image comprises:

using the flare-corrected pixel values to generate an intermediate light-field image in which effects of the first lens flare are at least partially corrected;

identifying a plurality of intermediate flare-affected pixels of the intermediate light-field image that have been affected by a second lens flare, wherein the intermediate flare-affected pixels have intermediate flare-affected pixel values;

generating intermediate flare-corrected pixel values for the intermediate flare-corrected pixels, wherein, relative to the intermediate flare-affected pixel values, the intermediate flare-corrected pixel values at least partially remove effects of the second lens flare; and

using the intermediate flare-corrected pixel values to generate the corrected light-field image.

16. The non-transitory computer-readable medium of claim 11, further comprising instructions stored thereon, that when executed by a processor, perform the steps of, prior to identifying the plurality of flare-affected pixels causing an input device to receive user input comprising a parameter;

wherein at least one of identifying the plurality of flare-affected pixels, generating the flare-corrected pixel values for the flare-affected pixels, and using the flare-corrected pixel values to generate the corrected light-field image comprises using the parameter.

17. A system for correcting a light-field image to remove effects of a first lens flare, the system comprising:

a data store configured to receive the light-field image after the light-field image has been captured by a light-field image capture device having a sensor and a plurality of microlenses; and

a processor, communicatively coupled to the data store, configured to;

identify a plurality of flare-affected pixels of the light-field image that have been affected by the first lens flare, wherein the flare-affected pixels have flare-affected pixel values, wherein the processor is configured to identify the plurality of flare-affected pixels of the light-field image by;

identifying a plurality of maximum value pixels, each of which has a pixel value higher than neighboring pixel values of neighboring pixels;

generating a global parametric flare model of effects of the first lens flare based on locations of the maximum value pixels; and

using the global parametric flare model to compute a probability that the pixel value of each of the maximum value pixels has been affected by the first lens flare; and

wherein the processor is further configured to generate the global parametric flare model of effects of the first lens flare based on locations of the maximum value pixels by generating a simple affine model that is predictive of the locations of the maximum value pixels;

generate flare-corrected pixel values for the flare-affected pixels, wherein, relative to the flare-affected pixel values, the flare-corrected pixel values at least partially remove effects of the first lens flare; and

use the flare-corrected pixel values to generate a corrected light-field image in which effects of the first lens flare are at least partially corrected.

18. The system of claim 17, wherein the processor is further configured to generate the flare-corrected pixel values for the flare-affected pixels by, for each of the flare-affected pixels, using neighboring pixel values to interpolate the flare-corrected pixel value for the flare-affected pixel.

19. The system of claim 18, wherein the processor is further configured to generate the flare-corrected pixel values for the flare-affected pixels by:

identifying at least one edge-located pixel of the flare-affected pixels, wherein each edge-located pixel is located proximate an edge depicted in the light-field image; and

identifying a plurality of edge-located neighboring pixels that are located proximate the edge;

wherein, for each of the edge-located pixels, the processor is further configured to use using the neighboring pixel values to interpolate the flare-corrected pixel value by using exclusively the neighboring pixel values of the edge-located neighboring pixels to interpolate the flare-corrected pixel value.

20. The system of claim 17, wherein the processor is further configured to generate flare-corrected pixel values for the flare-affected pixels by, for each of the maximum value pixels, assigning a weight, inversely proportional to the probability that the pixel value of the maximum value pixel has been affected by the first lens flare;

wherein the processor is further configured to generate the flare-corrected pixel values by processing the light-field image with at least one process in which each of the flare-affected pixels is excluded from processing in proportion to the weight assigned to it to obtain the flare-corrected pixel values.

21. The system of claim 17, wherein the processor is further configured to use the flare-corrected pixel values to generate the corrected light-field image by:

using the flare-corrected pixel values to generate an intermediate light-field image in which effects of the first lens flare are at least partially corrected;

identifying a plurality of intermediate flare-affected pixels of the intermediate light-field image that have been affected by a second lens flare, wherein the intermediate flare-affected pixels have intermediate flare-affected pixel values;

generating intermediate flare-corrected pixel values for the intermediate flare-corrected pixels, wherein, relative to the intermediate flare-affected pixel values, the intermediate flare-corrected pixel values at least partially remove effects of the second lens flare; and

using the intermediate flare-corrected pixel values to generate the corrected light-field image.

22. The system of claim 17, further comprising an input device configured to, prior to identification of the plurality of flare-affected pixels, receive user input comprising a parameter;

wherein the processor is further configured to carry out at least one of identifying the plurality of flare-affected pixels, generating the flare-corrected pixel values for the flare-affected pixels, and using the flare-corrected pixel values to generate the corrected light-field image by using the parameter.