Systems and methods for determining noise statistics of image data

• US 8,953,882 B2
• Filed: 05/31/2012
• Issued: 02/10/2015
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Lim, Suk Hwan, Silverstein, D. Amnon, Zimmer, Mark A., Cote, Guy
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US13/485,299
• Publication Number:
  US 20130321673A1
• Time in Patent Office: 985 Days
• Field of Search: 382/168, 382/169, 382/190, 382/191, 382/195, 348/241
• US Class Current: 382/168
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

The present disclosure generally relates to systems and methods for image data processing. In certain embodiments, an image processing pipeline may compute noise statistics associated with image data by receiving a frame of the image data having a plurality of pixels. The image processing pipeline may then identify a plurality of portions of the frame of the image data such that each portion of the plurality of portions has a flat surface. The image processing pipeline may then calculate a plurality of gradients for each portion of the plurality of portions, determine one or more dominant gradient orientations for each portion of the plurality of portions, and generate a histogram that represents a plurality of dominant gradient orientations that corresponds to the plurality of portions. After generating the histogram, the image processing pipeline may store the histogram, which may represent the noise statistics, in a memory.

Claims

1. Image signal processing circuitry, comprising:

an image processing pipeline configured to compute noise statistics associated with image data by;

receiving a frame of the image data comprising a plurality of pixels acquired using a digital image sensor;

identifying a plurality of portions of the frame of the image data, wherein each portion of the plurality of portions comprises a flat surface;

calculating a plurality of gradients for each portion of the plurality of portions;

determining one or more dominant gradient orientations for each portion of the plurality of portions;

generating a histogram that represents a plurality of dominant gradient orientations that corresponds to the plurality of portions; and

storing the histogram in a memory, wherein the histogram represents the noise statistics.

2. The image signal processing circuitry of claim 1, wherein the flat surface corresponds to an area of the frame of the image data that has an isotropic distribution of gradient orientations.

3. The image signal processing circuitry of claim 1, wherein the plurality of gradients comprise a plurality of spatial gradients for at least one color component of Bayer quads.

4. The image signal processing circuitry of claim 1, wherein the image processing pipeline is further configured to:

determine a sum of pixel intensities for each portion of the plurality of portions; and

store the sum of pixel intensities for each portion in the memory.

5. The image signal processing circuitry of claim 1, wherein the image processing pipeline is further configured to:

determine a peak gradient magnitude for each portion of the plurality of portions; and

store the peak gradient magnitude for each portion in the memory.

6. The image circuitry processing system of claim 1, wherein the image processing pipeline is further configured to:

determine a sum of gradient magnitudes for each portion of the plurality of portions; and

store the sum of gradient magnitudes for each portion in the memory.

7. The image signal processing circuitry of claim 1, wherein each portion of the plurality of portions are equal in size.

8. The image signal processing circuitry of claim 1, wherein the image processing pipeline is configured to store the histogram in the memory according to a scan order.

9. The image signal processing circuitry of claim of claim 1, wherein the plurality of portions of the frame of the image data comprise a plurality of non-overlapping portions of the frame of the image data.

10. The image signal processing circuitry of claim of claim 1, wherein the image processing pipeline is further configured to perform a demosaicing operation, a noise filtering operation, an image sharpening operation, or any combination thereof on the image data based at least in part on the noise statistics.

11. A method for processing image data comprising:

receiving into circuitry an image frame comprising a plurality of pixels;

calculating a plurality of gradients for a plurality of portions of the frame of image;

determining a dominant gradient orientation for each portion of the plurality of portions;

generating noise statistics for the plurality of portions based on the dominant gradient orientation for each portion of the plurality of portions; and

storing the noise statistics in a memory.

12. The method of claim 11, wherein calculating the plurality of gradients comprises:

generating a first set of gradients by determining a difference between a plurality of pixel values in at least four orientations for each portion of the plurality of portions, wherein the at least four orientations comprise one horizontal orientation, one vertical orientations, and two diagonal orientations; and

generating a second set of gradients by multiplying the difference between the plurality of pixel values in the at least four orientations for each portion of the plurality of portions by negative one.

13. The method of claim 11, wherein calculating the plurality of gradients comprises determining a difference between a plurality of pixel values in at least eight orientations for each portion of the plurality of portions, wherein the at least eight orientations comprise two horizontal orientations, two vertical orientations, and four diagonal orientations.

14. The method of claim 13, wherein determining the difference between the plurality of pixel values in the two horizontal orientations comprises:

determining a first difference between two pixel values along a horizontal axis in a first direction in each portion of the plurality of portions; and

determining a second difference between two pixel values along the horizontal axis in a second direction in each portion of the plurality of portions, wherein the first direction is opposite to the second direction.

15. The method of claim 14, wherein determining the difference between the plurality of pixel values in the two vertical orientations comprises:

determining a first difference between two pixel values along a vertical axis in a first direction in each portion of the plurality of portions; and

determining a second difference between two pixel values along the vertical axis in a second direction in each portion of the plurality of portions, wherein the first direction is opposite to the second direction.

16. The method of claim 14, wherein determining the difference between the plurality of pixel values in the four diagonal orientations comprises:

determining a first difference between two pixel values along a first diagonal axis in a first direction in each portion of the plurality of portions;

determining a second difference between two pixel values along the first diagonal axis in a second direction in each portion of the plurality of portions, wherein the first direction is opposite to the second direction;

determining a third difference between two pixel values along a second diagonal axis in a third direction in each portion of the plurality of portions;

determining a fourth difference between two pixel values along the second diagonal axis in a fourth direction in each portion of the plurality of portions, wherein the third direction is opposite to the fourth direction.

17. The method of claim 11, wherein calculating the plurality of gradients comprises applying a filter convolution to the plurality of pixel values in the plurality of portions of the frame of image.

18. The method of claim 11, wherein calculating the plurality of gradients comprises:

determining a difference between a plurality of pixel values in at least a vertical orientation and a horizontal orientation for each portion of the plurality of portions; and

determining an orientation angle for each portion of the plurality of portions based at least in part on an arctangent function and the plurality of pixel values in at least the vertical orientation and the horizontal orientation.

19. The method of claim 11, wherein determining the dominant gradient orientation comprises identifying one gradient magnitude in each portion of the plurality of portions that corresponds to the maximum gradient magnitude.

20. The method of claim 11, wherein generating the noise statistics comprises:

determining a count of a plurality of dominant gradient orientations in the plurality of portions of the frame of image;

incrementing a respective bin in a histogram by each count of the plurality of dominant gradient orientations; and

storing the histogram in the memory.

21. Image signal processing circuitry, comprising:

a statistics processing unit configured to receive a frame of image data, wherein the statistics processing unit comprises;

noise statistics logic configured to;

calculate a plurality of gradients for a plurality of portions of the frame of image data;

determine one or more dominant gradient orientations for a first portion of the plurality of portions;

increment a respective bin in a histogram by one if the first portion of the plurality of portions has one dominant gradient orientation, wherein the histogram comprises a bin for each gradient orientation;

increment respective bins in the histogram by 1/2 if the first portion of the plurality of portions has two or three dominant gradient orientations;

increment respective bins in the histogram by 1/4 if the first portion of the plurality of portions has more than three dominant gradient orientations; and

store the histogram in a memory.

22. The image signal processing circuitry of claim 21, wherein the histogram is represented by a 16-bit value having at least two fractional bits.

23. The image signal processing circuitry of claim 21, wherein the noise statistics logic is further configured to detect and correct one or more defective pixels in the frame of image data.

24. The image signal processing circuitry of claim 21, wherein the noise statistics logic is configured to determine the one or more dominant gradient orientations by:

determining a plurality of pixel gradient values along a horizontal axis, a vertical axis, a first diagonal axis, and a second diagonal axis for the first portion, wherein the first diagonal axis has a positive slope and the second diagonal axis has a negative slope; and

identifying one or more maximum pixel gradient values from the plurality of pixel gradient values.

25. The image signal processing circuitry of claim 24, wherein the plurality of pixel gradient values along the horizontal axis, the vertical axis, the first diagonal axis, or the second diagonal axis is determined based on two pixel values that correspond to two pixels having the same color component.

26. An electronic device comprising:

a digital image sensor;

a sensor interface configured to communicate with the digital image sensor;

a memory device;

a display device configured to display a visual representation of an image scene corresponding to raw image data acquired by the digital image sensor; and

an image signal processing sub-system comprising;

an image processing pipeline configured to;

receive the raw image data comprising a plurality of pixels;

calculate a plurality of gradients for a plurality of portions of the raw image data;

determine a dominant gradient orientation for each portion of the plurality of portions;

generate noise statistics for the plurality of portions based on the dominant gradient orientation for each portion of the plurality of portions; and

store the noise statistics in the memory device.

27. The electronic device of claim 26, wherein the digital image sensor comprises at least one of a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via an input/output port, or some combination thereof.

28. The electronic device of claim 26, wherein the sensor interface comprises a Standard Mobile Imaging Architecture (SMIA) interface, a serial imaging interface, a parallel imaging interface, or some combination thereof.

29. The electronic device of claim 26, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof.

30. Image signal processing circuitry, comprising:

an image processing pipeline configured to compute noise statistics associated with image data by;

receiving a frame of the image data comprising a plurality of pixels acquired using a digital image sensor;

identifying a plurality of portions of the frame of the image data, wherein each portion of the plurality of portions comprises a flat surface;

calculating a plurality of gradients for each portion of the plurality of portions;

determine a peak gradient magnitude for each portion of the plurality of portions; and

store the peak gradient magnitude for each portion in the memory.

31. An image signal processing circuitry, comprising:

an image processing pipeline configured to compute noise statistics associated with image data by;

receiving a frame of the image data comprising a plurality of pixels acquired using a digital image sensor;

identifying a plurality of portions of the frame of the image data, wherein each portion of the plurality of portions comprises a flat surface;

calculating a plurality of gradients for each portion of the plurality of portions;

determine a mean value of gradient magnitudes for each portion of the plurality of portions; and

store the mean value of gradient magnitudes for each portion in the memory.

32. An image signal processing circuitry, comprising:

an image processing pipeline configured to compute noise statistics associated with image data by;

receiving a frame of the image data comprising a plurality of pixels acquired using a digital image sensor;

identifying a plurality of portions of the frame of the image data, wherein each portion of the plurality of portions comprises a flat surface;

calculating a plurality of gradients for each portion of the plurality of portions;

determining a standard deviation value for the plurality of gradients for each portion of the plurality of portions;

generating a standard deviation versus pixel intensity curve based at least in part on the standard deviation value for the plurality of gradients for each portion of the plurality of portions; and

storing the standard deviation versus pixel intensity curve in a memory, wherein the standard deviation versus pixel intensity curve represents the noise statistics.

33. The image signal processing circuitry of claim 32, wherein determining the standard deviation value comprises:

removing one or more outlier gradients from the plurality of gradients, thereby generating a updated plurality of gradients; and

determining a standard deviation value based at least in part on the updated plurality of gradients.

34. The image signal processing circuitry of claim 32, generating the standard deviation versus pixel intensity curve comprises generating a curve that fits the standard deviation value for the plurality of gradients for each portion of the plurality of portions, wherein the curve is a linear curve, a quadratic curve, or a polynomial curve.