Robust selection and weighting for gray patch automatic white balancing

• US 9,307,213 B2
• Filed: 11/05/2012
• Issued: 04/05/2016
• Est. Priority Date: 11/05/2012
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Pieper, Sean Midthun
• Primary Examiner(s): Tran, Sinh
• Assistant Examiner(s): GEBRIEL, SELAM T
• Application Number:
  US13/668,708
• Publication Number:
  US 20140125836A1
• Time in Patent Office: 1,247 Days
• Field of Search: 348/221.1, 348/222.1, 348/223.1
• US Class Current: 1/1
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control

Abstract

Embodiments of the present invention are directed to methods and systems for robust weighting of gray patches in automatic white balancing in an image-capture device by utilizing kernel density estimation techniques with dynamically variable bandwidth to determine the probability density of samples to create an initial estimate, then verifying the initial gray point estimate to account for outliers. In one embodiment, given a set of image data, an initial gray point estimate in a color space is determined for the set of image data. The initial estimate is then refined by weighting the sub-population with the greatest probability of being gray. A final evaluation that includes a further comparison to pre-programmed constraints determines a final estimate, which can still be further tuned according to user preferences by adjusting color biases. The resulting final gray point estimate provides greater stability, and greatly improved accuracy over traditional techniques and solutions.

Claims

1. A method for calculating a white balance for an image capture device, the method comprising:

receiving a set of image data comprising a plurality of pixels;

calculating a position in a color space for each of the plurality of pixels, the color space comprising a plot of common lights;

determining a plurality of distances between the positions of the pixels in the color space and the plot of common lights;

sorting the plurality of pixels into a plurality of sample populations based on the distances;

selecting a first sample population from the plurality of sample populations by calculating an estimate of a probability density as a function of distance from the plot of common lights for the plurality of sample populations;

averaging the distances in the first sample population to determine an initial gray point estimate; and

refining the initial gray point estimate into a final gray point estimate.

2. The method according to claim 1, further comprising converting the final gray point estimate into a set of white balance gains.

3. The method according to claim 2, further comprising storing the set of white balance gains to a memory.

4. The method according to claim 2, further comprising applying the set of white balance gains to the image data.

5. The method according to claim 4, wherein the set of image data comprises a down-sampled image.

6. The method according to claim 5, wherein the image capture device comprises an electronic viewfinder.

7. The method according to claim 6, wherein the applying the set of white balance gains to the image data comprises:

rendering the image by applying the set of white balance gains to the set of image data; and

displaying the image in the electronic viewfinder.

8. The method according to claim 7, wherein the rendering the image and the displaying the image in the electronic viewfinder are performed live.

9. The method according to claim 5, wherein the down-sampled image comprises a set of values in a sensor color space.

10. The method according to claim 1, further comprising converting the final gray point estimate into a correlated color temperature (CCT).

11. The method according to claim 10, further comprising applying the CCT to control multi-lens shading in the image capture device.

12. The method according to claim 1, wherein the color space comprises a harmonic mean color space.

13. The method according to claim 1, wherein the common lights in the plot of common lights comprise Planckian lights.

14. The method according to claim 1, wherein the plot of common lights comprises a Planckian locus.

15. The method according to claim 13, wherein the common lights in the plot of common lights comprise a plurality of fluorescent lights.

16. The method according to claim 1, wherein the calculating an estimate of the probability density as a function of distance from the plot of common lights for the plurality of sample populations comprises:

applying a probability density function to the plurality of pixels comprised in the plurality of sample populations;

plotting the probability density function for the plurality of sample populations;

identifying a first peak in a plot of the probability density; and

determining a first sample population from the plurality of sample populations based on the identified first peak in the plot of probability density.

17. The method according to claim 16, wherein the calculating a probability density comprises applying a kernel density estimator.

18. The method according to claim 17, wherein the applying a kernel density estimator comprises estimating a bandwidth for the kernel density estimator.

19. The method according to claim 18, wherein the bandwidth is dynamically variable.

20. The method according to claim 18, wherein the bandwidth is fixed.

21. The method according to claim 17, wherein the applying the kernel density estimator comprises:

sorting the plurality of sample populations using a radix sort; and

applying a kernel with a finite extent using a rolling window.

22. The method according to claim 16, wherein the refining the initial gray point comprises:

deriving a refined gray point estimate;

evaluating whether the refined gray point estimate is a reasonable final gray point estimate; and

selecting the refined gray point estimate as the final gray point estimate if the refined gray point estimate is a reasonable final gray point estimate.

23. The method according to claim 22, wherein the deriving the refined gray point estimate comprises:

sorting the first sample population based on the distance between the samples to the initial gray point;

applying a kernel density estimator to the first sample population to calculate a distribution of the samples in the first sample population;

identifying a plurality of sub-populations in the first sample population;

calculating an average for each of the plurality of sub-populations;

for each sub-population of the plurality of sub-populations, weighting the average of a sub-population based on the distance of the sub-population from the initial gray point and a number of samples in the sub-population; and

combining the weighted averages of the plurality of sub-populations into the refined gray point estimate.

24. The method according to claim 23, wherein the evaluating whether the refined gray point estimate is a reasonable final gray point estimate comprises determining a first distance comprising a distance between the refined gray point estimate and a set of feasible lights.

25. The method according to claim 24, wherein the set of feasible lights comprises a plurality of Planckian lights, a plurality of fluorescent lights, and a plurality of mixtures of lighting.

26. The method according to claim 24, further comprising:

comparing the first distance to a pre-determined threshold distance; and

replacing the refined gray point estimate with a final gray estimate based on a last known gray point estimate when the first distance is greater than the pre-determined threshold distance.

27. The method according to claim 26, wherein the replacing the refined gray point estimate with a final estimate based on a last known gray point estimate comprises:

calculating an average of the refined gray point estimate and a last known gray point estimate based on a exponential falloff,

calculating the final gray point estimate by weighting the average of the refined gray point estimate and the last known gray point estimate.

28. The method according to claim 24, further comprising:

comparing the number of samples in the first sample population to a pre-determined threshold population; and

replacing the refined gray point estimate with a last known gray point estimate when the number of samples in the first sample population is less than the pre-determined threshold population.

29. The method according to claim 24, further comprising:

calculating a confidence value for the refined gray point estimate based on the first distance; and

generating a final gray point estimate by applying the confidence value to the refined gray point estimate.

30. The method according to claim 29, wherein the calculating a confidence value for the refined gray point estimate based on the first distance comprises calculating a product of a linear function based on the number of samples in the first sample population, and an exponential decay as a function of distance of the estimate from the set of feasible lights.

31. The method according to claim 30, wherein the confidence value is incorporated into damping of a white balance control loop in a continuous capture system.

32. The method according to claim 30, wherein the per-frame confidence value and optionally a weight indicating general reliability are used to combine the refined gray estimate with a plurality of estimates from other methods for white balance.

33. The method according to claim 22, further comprising applying a correlated color temperature bias to the final gray point estimate.

34. The method according to claim 22, further comprising applying an isothermic bias to the final gray point estimate.

35. The method according to claim 22, further comprising applying a color preference bias to the final gray point estimate, the color preference bias comprising an offset based on a pre-specified user preference.

36. A non transitory computer readable medium containing program instructions embodied therein for causing an image capture device to calculate a gray point estimate for a captured image, the program instructions comprising:

instructions to receive a set of image data comprising a plurality of pixels;

instructions load a set of characterization data corresponding to a color space;

instructions to calculate a plot of common lights in the color space and a position in the color space for each of the plurality of pixels;

instructions to determine a plurality of distances between the positions of the pixels in the color space and the plot of feasible lights;

instructions to sort the plurality of pixels into a plurality of sample populations based on the distances;

instructions to select a first sample population from the plurality of sample populations by calculating an estimate of a probability density as a function of distance from the plot of common lights for the plurality of sample populations;

instructions to average the first sample population of the plurality of sample populations to determine an initial gray point estimate; and

instructions to refine the initial gray point to determine a refined gray point estimate.

37. A system for calculating a white balance estimate in an image capture device, said system comprising:

an image capture device comprising a lens, an image sensor and an image signal processor, the image capture device being configured to generate an image comprising a set of image data;

a memory, communicatively coupled to the image capture device, and configured to store the set of image data and a plurality of programmed instructions; and

a processor, communicatively coupled to the memory, and configured to execute the plurality of programmed instructions and to calculate a white balance estimate from the set of image data by refining an initial gray point estimate generated from averaging a selection of a plurality of sample pixels comprised in the set of image data and stored in the memory, the selection of the plurality of sample pixels comprising the pixels with corresponding positions in a color space closest to a pre-plotted line based on an application of a probability density function,

wherein the initial gray point estimate is refined by applying a kernel density estimator to selection of the plurality of sample pixels to identify a plurality of sub-populations and by taking a weighted average of the plurality of sub-populations,

further wherein the white balance estimate is applied to the set of image data.