TECHNIQUES FOR ACQUIRING AND PROCESSING STATISTICS DATA IN AN IMAGE SIGNAL PROCESSOR

• US 20120050567A1
• Filed: 09/01/2010
• Published: 03/01/2012
• Est. Priority Date: 09/01/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Côté, Guy, Frederiksen, Jeffrey E.
• Granted Patent: US 8,531,542 B2
• Time in Patent Office: Days
• US Class Current: 348/224.1
• CPC Class Codes: H04N 23/673 based on contrast or high frequency components of image signals, e.g. hill climbing method
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/745 Detection of flicker frequency or suppression of flicker wherein the flicker is caused by illumination, e.g. due to fluorescent tube illumination or pulsed LED illumination
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/86 for controlling the colour saturation of colour signals, e.g. automatic chroma control circuits
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/46 by combining or binning pixels
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 25/68 applied to defects
• CPC Class Codes: H04N 9/68 for controlling the amplitude of colour signals, e.g. automatic chroma control circuits H04N9/71, H04N9/73 take precedence; camera processing pipelines for controlling the colour saturation of colour signals H04N23/86

Abstract

Various techniques are disclosed for processing statistics data in an image signal processor (ISP). In one embodiment, a statistics collection engine may be implemented in a front-end processing unit of the ISP, such that statistics are collected prior to processing by an ISP pipeline downstream from the front-end processing unit. In one embodiment, the statistics collection engine may be configured to acquire statistics relating to auto white-balance, auto-exposure, and auto-focus, as well as flicker detection. Collected statistics may be output to a memory and used by the ISP to process acquired image data.

Claims

1. An image signal processing system comprising:

a front-end pixel processing unit configured to receive a frame of raw image data comprising pixels acquired using a digital image sensor, wherein the front-end pixel processing unit comprises statistics collection logic configured to collect statistics based upon the raw frame pixel data, wherein the collected statistics comprises at least one of auto-white balance statistics, auto-exposure statistics, auto-focus statistics, and flicker detection statistics, and wherein the collection of the statistics occurs prior to the raw frame being processed by an image signal processing pipeline coupled downstream of the front-end pixel processing unit.

2. The image signal processing system of claim 1, wherein the statistics collection logic comprises:

an input configured to receive the raw frame pixels;

color space conversion logic configured to convert the raw frame pixel data into multiple sets of converted pixel data, each of the converted pixel data sets being in a different color space; and

a set of pixel filters, each being configured to receive the raw frame pixel data and the multiple sets of converted pixel data, to select one set of either the raw frame pixel data or the converted pixel data, and to determine one or more accumulated color sum values by analyzing the selected set of pixel data.

3. The image signal processing system of claim 2, wherein the multiple sets of converted pixel data include at least one set of pixel data in a non-linear color space and one set of pixel data in a linear color space.

4. The image signal processing system of claim 3, wherein the non-linear color space comprises at least one of a sRGB color space, a sRGB_linear color space, or a YC1C2 color space.

5. The image signal processing system of claim 3, wherein the linear color space comprises a camera color space conversion.

6. The image signal processing system of claim 2, comprising down-scaling logic configured to down-scale the raw frame pixel data before the raw frame pixel data is provided to the color space conversion logic.

7. The image signal processing system of claim 6, wherein the raw frame pixel data comprises Bayer RGB data, and wherein the down-scaling logic down-scales the Bayer RGB data by averaging a 4×

4 block of pixels comprising 2×

2 Bayer quads.

8. A method comprising:

receiving a frame of raw image data acquired using a digital image sensor;

using a statistics collection engine of an image signal processor to apply a linear camera color space conversion to the raw image data to generated a converted set of pixel data in the camera color space;

accumulating a luma value over each row of the raw frame to determine a luma row sum; and

applying frequency analysis to the luma row sum to detect whether flicker is present in the raw image data.

9. The method of claim 8, wherein the flicker is the result of a periodic variation in an illuminant source powered by AC power.

10. The method of claim 9, wherein applying frequency analysis comprises analyzing the row average of the luma row sum to determine a frequency of the AC power.

11. The method of claim 9, wherein applying frequency analysis comprises analyzing row average differences of consecutive image frames to determine a frequency of the AC power.

12. The method of claim 9, comprising determining a flicker period corresponding to a frequency of the AC power.

13. The method of claim 12, comprising reducing the appearance of flicker by adjusting the integration time of the digital image sensor to be an integer multiple of the determined flicker period.

14. A method comprising

using an image signal processor (ISP):

receiving a frame of raw image data acquired using a digital image sensor; and

converting the raw image data into a non-linear luma and chroma color space and a camera luma and chroma color space;

selecting one of a non-linear luma and chroma color space and a camera luma and chroma color space; and

analyzing the raw frame using a statistics collection engine of a front-end pixel processing unit of the ISP to generate a two-dimensional color histogram for a window within a raw frame, wherein the two-dimensional color histogram is in the selected luma and chroma color space and includes a plurality of bins arranged at fixed intervals.

15. The method of claim 14, wherein generating the two-dimensional color histogram in the selected luma and chroma color space comprises:

for a current luma and chroma pixel data set, determining a first bin index for a first chroma component and determining a second bin index for a second chroma component; and

incrementing a bin corresponding to the first and second bin indices by a count value if the first and second bin indices are within a range corresponding to the number of bins defining a height and width, respectively, of the window.

16. The method of claim 15, wherein, for each pixel in the current luma and chroma pixel data set, the count value for each of the chroma components for a current pixel is weighted based on a luma component associated with the current pixel.

17. The method of claim 15, wherein the count value for each of the chroma components is weighted by evaluating the value of the luma component of the current pixel relative to a plurality of luma intervals defined by one or more luma thresholds, and wherein the count value for each of the chroma components of the current pixel is weighted more heavily in its corresponding bin depending on the brightness of the current pixel increases, as indicated by the luma interval corresponding to the luma component associated with the current pixel.

18. The method of claim 15, wherein the first chroma component and the second chroma component are represented by 8-bit values, and wherein the first and second bin indices are determined based at least partially upon the upper five bits of the first and second chroma components.

19. An electronic device comprising:

a digital image sensor;

an interface configured to communicate with the digital image sensor;

a memory device;

a display device configured to display a visual representation of an image scene corresponding to raw image data acquired by the digital image sensor; and

an image signal processing sub-system comprising a front-end pixel processing unit having defective pixel detection and correction logic, black level compensation logic, and statistics collection logic, wherein the statistics collection logic is configured to receive the raw image data after the raw image data is processed by the defective pixel detection and correction logic and the black level compensation logic and to collect one or more sets of statistics from the raw image data relating to one or more of auto-white balance, flicker detection, auto-focus, and auto-exposure;

wherein the one or more sets of statistics comprises tile statistics collected for one window, floating window statistics for auto-white balance and auto-exposure, luma row sum values, two-dimensional color histogram, one or more component histograms, and one or more sets of edge sum values for auto-focus statistics.

20. The electronic device of claim 19, wherein the one or more sets of statistics are written to the memory device using a DMA interface.

21. The electronic device of claim 20, wherein the memory device comprises double-buffered memory address registers configured to allow for a new location in the memory device to be specified for each image frame.

22. The electronic device of claim 19, wherein the raw image data comprises Bayer RGB pixel data, and wherein the electronic device comprises color space conversion logic configured to convert the Bayer RGB pixel data into a corresponding set of pixels in a sRGB_linear color space using a first linear transform, to convert the sRGB_linear pixels into a corresponding set of pixels in a sRGB color space using a non-linear look-up table, and to convert the sRGB pixels into a corresponding set of pixels in a luma and chroma color space using a second linear transform.

23. The electronic device of claim 22, wherein the one or more component histograms comprises a first histogram and a second histogram, wherein the first histogram comprises components selected from one of the Bayer RGB pixel data received by the statistics collection logic, the sRGB_linear pixels, the sRGB pixels, or the luma and chroma pixels, and wherein the second histogram comprises components selected from Bayer RGB data received by the front-end pixel processing unit that has not been processed by the defective pixel detection and correction logic and the black level compensation logic.

24. The electronic device of claim 23, wherein the second histogram is used by the black level compensation logic to determine an offset for applying black level compensation to the Bayer RGB data received by the front-end pixel processing unit.

25. The electronic device of claim 19, wherein digital image sensor comprises at least one of a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via the interface, or some combination thereof.

26. The electronic device of claim 19, wherein the sensor interface comprises a Standard Mobile Imaging Architecture (SMIA) interface.

27. The electronic device of claim 19, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof.