FLEXIBLE COLOR SPACE SELECTION FOR AUTO-WHITE BALANCE PROCESSING

• US 20120050563A1
• Filed: 09/01/2010
• Published: 03/01/2012
• Est. Priority Date: 09/01/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Côté, Guy, Frederiksen, Jeffrey E., Hubel, Paul Matthew, Chawla, Sumit
• Granted Patent: US 8,605,167 B2
• Time in Patent Office: Days
• US Class Current: 348/223.1
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/673 based on contrast or high frequency components of image signals, e.g. hill climbing method
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

Various techniques are disclosed for processing statistics data in an image signal processor (ISP). In one embodiment, a statistics collection engine may be configured to acquire statistics relating to auto white-balance. The statistics collection engine may receive raw Bayer RGB data acquired by an image sensor and may be configured to perform one or more color space conversions to obtain pixel data in other color spaces. A set of pixel filters may be configured to accumulate sums of the pixel data conditionally based upon YC1C2 characteristics, as defined by a pixel condition per pixel filter. Depending on a selected color space, the pixel filters may generate color sums, which may be used to match a current illuminant against a set of reference illuminants with which the image sensor has been previously calibrated.

Claims

1. An image signal processing system comprising:

a front-end pixel processing unit configured to receive a frame of multi-color raw image data comprising pixels representing an image scene acquired using a digital image sensor, wherein the front-end pixel processing unit comprises statistics collection logic configured to collect statistics based upon the raw frame pixel data, wherein the collected statistics comprises at least one of auto-white balance statistics and auto-exposure statistics, and wherein the statistics collection logic comprises;

an input configured to receive the raw frame pixel data;

color space conversion logic configured to convert the raw frame pixel data into multiple sets of converted pixel data in, each of the converted pixel data sets being in a different color space; and

a set of pixel filters, each being configured to receive the raw frame pixel data and the multiple sets of converted pixel data, to select one set of either the raw frame pixel data or the converted pixel data, and to analyze the selected set of pixel data and determine one or more accumulated color sum values;

wherein the image signal processing system comprises control logic configured to determine a set of white point balance gains to apply to the multi-color raw frame pixels based at least partially upon the accumulated color sum values.

2. The image signal processing system of claim 1, wherein each pixel filter is configured to evaluate each pixel of the selected set of pixel data against a set of pixel conditions in a luma and chroma color space and to include a current pixel in a corresponding color sum value if the current pixel satisfies the set of pixel conditions.

3. The image signal processing system of claim 2, wherein the set of pixel conditions defines a region in a color space of the luma and chroma color space, and wherein evaluating each pixel of the selected set of pixel data comprises determining whether luma and chroma values corresponding to the current pixel satisfies at least one line condition.

4. The image signal processing system of claim 3, wherein the luma and chroma color space is selected from one of a linear luma and chroma color space or a non-linear camera luma and chroma color space.

5. The image signal processing system of claim 1, wherein the raw frame pixel data comprises Bayer RGB pixel data, and wherein the color space conversion logic is configured to:

use a first color correction matrix to convert the Bayer RGB pixel data into a corresponding set of pixel data in a sRGB_linear color space;

use a non-linear look-up table to convert the sRGB_linear set of pixel data into a corresponding set of pixel data in a sRGB color space;

use a second color correction matrix to convert the sRGB set of pixel data to a corresponding set of pixel data in a luma (Y) and chroma (C1, C2) color space; and

use a third color correction matrix to convert the Bayer RGB pixel data into a corresponding set of pixel data in a camera luma (camY) and chroma (camC1, camC2) color space.

6. The image signal processing system of claim 5, wherein the Bayer RGB data is decimated before being provided to the color space conversion logic.

7. The image signal processing system of claim 5, wherein each of the Bayer RGB pixel data, the sRGB_linear pixel data, the sRGB pixel data, the YC1C2 pixel data, and the camYC1C2 are provided to each of the pixel filters, and wherein each pixel filter selects one of these pixel data sets and determine one or more accumulated color sum values for the selected pixel data set.

8. The image signal processing system of claim 7, wherein, if a pixel filter is configured to acquire white point balance statistics, the one or more accumulated color sum values are based upon the Bayer RGB pixel data or the sRGB_linear pixel data;

and

wherein if a pixel filter is configured to perform scene analysis, the one or more accumulated color sum values are based upon the YC1C2 pixel data.

9. The image signal processing system of claim 1, wherein determining the set of white point balance gains to apply to the multi-color raw pixel data comprises:

determining red-to-green and blue-to-green ratios based on the accumulated color sum values; and

based at least partially on the red-to-green and blue-to-green ratios, selecting a reference illuminant from a set of reference illuminants that most closely matches a current illuminant of image scene;

wherein white point balance gains corresponding to the selected reference illuminant are applied to the multi-color raw frame pixels.

10. A method for collecting auto-white balance (AWB) statistics using an image signal processor comprising:

receiving an image frame having Bayer RGB raw image pixels acquired using an image sensor;

converting the Bayer RGB raw image pixels into corresponding sets of pixels including at least a first set of pixels in a sRGB_linear color space, a second set of pixels in a sRGB color space, and a third set of pixels in a luma and chroma color space (YC1C2), and a fourth set of pixels in a camera luma and chroma color space (camYC1C2); and

providing the Bayer RGB raw image pixels, the sRGB_linear pixels, the sRGB pixels, the YC pixels, and the camYC1C2 pixels to each of a set of selected pixel filters, wherein each of the selected pixel filters is configured to;

use a first selection logic to select a first selected pixel set as either the YC1C2 pixels or the camYC1C2 pixels;

use a second selection logic to select a second selected pixel set as one of the Bayer RGB pixels, the sRGB_linear pixels, the sRGB pixels, or the first selected pixel set; and

evaluating the first selected pixel set against a set of pixel conditions to determine whether which pixels from the second selected pixel set are to be accumulated in a color sum.

11. The method of claim 10, wherein determining whether to include each pixel from the second selected pixel set in an accumulated color sum comprises evaluating whether a current corresponding pixel from the first selected pixel set satisfies a set of pixel conditions associated with a current pixel filter.

12. The method of claim 11, wherein evaluating whether a current corresponding pixel from the first selected pixel set satisfies a set of pixel conditions comprises:

determining whether a value of first chroma component associated with the current corresponding pixel is within a minimum and maximum range for the first chroma component, the minimum and maximum range being determined by a first point and a second point in a color space (C1-C2) corresponding to the first selected pixel set;

determining whether a value of a second chroma component associated with the current corresponding pixel is within a minimum and maximum range for the second chroma component, the minimum and maximum range being determined by the first point and the second point in the color space (C1-C2);

determining whether a value of a luma component associated with the current corresponding pixel is within a minimum and maximum luma range; and

determining whether the coordinates of the current corresponding pixel at least one line condition defined based at least partially upon the first point and the second point.

13. The method of claim 12, wherein determining whether the coordinates of the current corresponding pixel satisfies the set of pixel conditions comprises:

determining a first delta between a first chroma component associated with the first point and a first chroma component associated with the second point;

determining a second delta between a second chroma component associated with the first point and a second chroma component associated with the second point;

determining an offset as the multiplying the first delta by the value at which a line between the first and second points intercepts an axes corresponding to the second chroma component;

determining a first perpendicular distance from the current corresponding pixel to the line;

determining a maximum distance by multiplying the first perpendicular distance by a normalization factor;

determining a first value as the difference between the product of the second delta and the value of the first chroma component associated with the current corresponding pixel and the product of the first delta and the value of the second chroma component associated with the current corresponding pixel;

adding the offset to the first value to obtain a second value; and

determining the absolute value of the second value and determining whether the absolute value of the second value is less than the maximum distance.

14. The method of claim 13, wherein the normalization factor is determined by summing the squares of the first delta value and the second delta value, and obtaining a square root of the result.

15. The method of claim 12, wherein determining whether the coordinates of the current corresponding pixel satisfies the set of pixel conditions comprises:

identifying a region with the color space (C1-C2) as at least one polygon formed using the first point, the second point, and at least one additional point as its vertices; and

for each line forming a side of the polygon, determining with the coordinates of the current corresponding pixel are located on a side of the line such that the current corresponding pixel would be enclosed by the polygon.

16. The method of claim 15, wherein the at least one polygon comprises multiple overlapping polygons.

17. A method for determining white balance gains using an image signal processor comprising:

receiving an image frame having Bayer RGB raw image pixels and being representative of an image scene acquired using an image sensor;

providing a set of reference illuminants for calibrating the image sensor, wherein each reference illuminant is associated with a set of white balance gains for red, green, and blue color channels;

using color space conversion logic to convert the Bayer RGB raw image pixels into multiple sets of converted pixel data in, each of the converted pixel data sets being in a different color space;

using a set of pixel filters, each being configured to receive the Bayer RGB raw image pixels and the multiple sets of converted pixel data, to select one set of either the Bayer RGB raw image pixels or the converted pixel data, and to analyze the selected set of pixel data and determine one or more accumulated color sum values;

using color ratios based at least partially upon the accumulated color sum values to select a reference illuminant from the set of reference illuminants that most closely matches lighting conditions corresponding to the image scene and applying the set of white balance gains associated with the selected reference illuminant.

18. The method of claim 17, wherein if the lighting conditions corresponding to the current image scene falls between two reference illuminants, the white balance gains for each of the two reference illuminants are interpolated to determine the set of white balance gains for the current image scene.

19. The method of claim 17, wherein the set of reference illuminants comprises a low color temperature reference illuminant, a middle-low color temperature reference illuminant, a middle color temperature reference illuminant, a middle-high color temperature reference illuminant, and a high color temperature reference illuminant.

20. The method of claim 18, wherein the low color temperature reference illuminant comprises a Horizon reference illuminant having a color temperature of approximately 2300 degrees, wherein the middle-low color temperature reference illuminant comprises an incandescent reference illuminant having a color temperature of between approximately 2800 to 2900 degrees, wherein the middle color temperature reference illuminant comprises a D50 reference illuminant having a color temperature of approximately 5000 degrees, wherein the middle-high color temperature reference illuminant comprises a D65 reference illuminant having a color temperature of approximately 6500 degrees, and wherein the high color temperature reference illuminant comprises a D75 reference illuminant having a color temperature of approximately 7500 degrees.

21. An electronic device comprising:

an imaging device comprising a digital image sensor;

an interface configured to communicate with the digital image sensor;

a memory device;

a display device configured to display a visual representation of an image scene corresponding to raw image data acquired by the digital image sensor; and

an image signal processing sub-system comprising a front-end pixel processing unit configured to receive a frame of raw image data comprising pixels acquired using an imaging device having a digital image sensor, wherein the front-end pixel processing unit comprises a statistics collection unit comprising;

color space conversion logic configured to convert the raw frame pixel data into multiple sets of converted pixel data in, each of the converted pixel data sets being in a different color space, wherein at least one of the converted pixel data sets is in a linear color space and at least one of the converted pixel data sets is in a non-linear color space; and

a set of pixel filters, each being configured to receive the raw frame pixel data and the multiple sets of converted pixel data, to select one set of either the raw frame pixel data or the converted pixel data, and to analyze the selected set of pixel data and determine one or more accumulated color sum values, wherein each pixel filter is configured to evaluate each pixel of the selected set of pixel data against a set of pixel conditions in a luma and chroma color space and to include a current pixel in a corresponding color sum value if the current pixel satisfies the set of pixel conditions.

22. The electronic device of claim 21, wherein the image signal processing sub-system is configured to determine a set of white point balance gains to apply to the raw frame pixels based at least partially upon the accumulated color sum values.

23. The electronic device of claim 21, wherein digital image sensor comprises at least one of a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via the interface, or some combination thereof.

24. The electronic device of claim 21, wherein the raw frame pixel data comprises Bayer RGB pixel data.

25. The electronic device of claim 24, wherein the Bayer RGB pixel data is down-scaled by averaging a 4×

4 block of pixels comprising 2×

2 Bayer quads prior to being provided to the color space conversion logic.

26. The electronic device of claim 21, wherein the sensor interface comprises a Standard Mobile Imaging Architecture (SMIA) interface.

27. The electronic device of claim 21, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof.