System and method for processing image data using an image processing pipeline of an image signal processor

• US 8,358,319 B2
• Filed: 05/28/2010
• Issued: 01/22/2013
• Est. Priority Date: 10/20/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cote, Guy, Frederiksen, Jeffrey E.
• Primary Examiner(s): McDowell, Jr., Maurice L
• Application Number:
  US12/789,815
• Publication Number:
  US 20110090381A1
• Time in Patent Office: 970 Days
• Field of Search: 345/611, 382/312
• US Class Current: 345/611
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 25/611 Correction of chromatic aberration
• CPC Class Codes: H04N 25/683 by defect estimation performed on the scene signal, e.g. real time or on the fly detection

Abstract

Various techniques are provided herein for processing raw image data acquired using a digital image sensor in an image processing pipeline of an image signal processing system. In one embodiment, the image processing pipeline may first process the raw image data (e.g., Bayer image data) for the detection and correction of defective pixels. Next, the image processing pipeline may process the raw image data to reduce noise. Thereafter, the image processing pipeline may correct lens shading distortion in the raw image data and, subsequently, apply a demosaicing algorithm to convert the raw image data into full color image data (e.g., RGB image data). The color image data may be further processed by the image processing pipeline to correct color and gamma properties prior to being converted into a luma and chroma color space (e.g., YCbCr color space).

Claims

1. A method for processing image data using an image signal processor, comprising:

using an image signal processor;

receiving raw image data corresponding to an image scene captured using a digital image sensor;

converting the raw image data into color image data, wherein converting the raw image data into the color image data comprises the steps of;

prior to detecting any defective pixels in the raw image data, for each pixel of the raw image data, applying a first offset to a current pixel, applying a first digital gain to the current pixel after applying the first offset, and, after applying the first digital gain, clipping the value of the current pixel to a first minimum value if the value of the current pixel is below the first minimum value or to a first maximum value if the value of the current pixel is above the first maximum value;

detecting any defective pixels in the raw image data;

correcting any detected defective pixels in the raw image data;

processing the raw image data to reduce noise after the detected defective pixels are corrected;

processing the raw image data to correct lens shading distortion after processing the raw image data to reduce noise;

after processing the raw image data to correct lens shading distortion, for each pixel of the raw image data, applying a second offset to the current pixel, applying a second digital gain to the current pixel after applying the second offset, and, after applying the second digital gain, clipping the value of the current pixel to a second minimum value if the value of the current pixel is below the second minimum value or to a second maximum value if the value of the current pixel is above the second maximum value; and

applying a demosaicing process to convert the raw image data into the color image data after the correction of the lens shading distortion;

converting the color image data to a luma and chroma color space to produce a corresponding set of luma and chroma image data; and

displaying a visual representation of the image scene using the luma and chroma image data.

2. A method for processing image data using an image signal processor, comprising:

using an image signal processor;

receiving raw image data corresponding to an image scene captured using a digital image sensor;

converting the raw image data into color image data, wherein converting the raw image data into the color image data comprises the steps of;

detecting any defective pixels in the raw image data;

correcting any detected defective pixels in the raw image data;

processing the raw image data to reduce noise after the detected defective pixels are corrected, wherein processing the raw image data to reduce noise comprises applying horizontal filtering and vertical filtering to each color component of the raw image data;

processing the raw image data to correct lens shading distortion after processing the raw image data to reduce noise; and

applying a demosaicing process to convert the raw image data into the color image data after the correction of the lens shading distortion;

converting the color image data to a luma and chroma color space to produce a corresponding set of luma and chroma image data; and

displaying a visual representation of the image scene using the luma and chroma image data.

3. The method of claim 2, wherein the raw image data comprises Bayer image data, wherein applying the horizontal filtering to each of red, blue, and green components of the Bayer image data using a horizontal filter comprises, for a current pixel of the raw image data, identifying a set of horizontally adjacent pixels of the same color as the current pixel, wherein a center tap of the horizontal filter is positioned at the current pixel, determining a horizontal edge gradient value between each pair of adjacent pixels within the set of horizontally adjacent pixels, and determining a horizontal filtering output value, wherein the horizontal filtering output is based at least partially upon a set of horizontal filtering coefficients and a comparison of the horizontal edge gradient values to a horizontal edge threshold value corresponding to the color of the current pixel;

wherein applying the vertical filtering to each of the red, blue, and green components of the Bayer image data using a vertical filter comprises, for the current pixel, identifying a set of vertically adjacent pixels of the same color as the current pixel, wherein a center tap of the vertical filter is positioned at the current pixel, determining a vertical edge gradient value between each pair of adjacent pixels within the set of vertically adjacent pixels, and determining a vertical filtering output value, wherein the vertical filtering output is based at least partially upon a set of vertical filtering coefficients and a comparison of the vertical edge gradient values to a vertical edge threshold value corresponding to the color of the current pixel; and

wherein a filtered value for the current input pixel is determined based upon the horizontal filtering output value and the vertical filtering output value.

4. The method of claim 3, wherein the horizontal filtering of the current pixel is applied before the vertical filtering of the current pixel.

5. The method of claim 3, wherein the horizontal filter comprises a 7-tap filter, and wherein the vertical filter comprises a 5-tap filter.

6. The method of claim 3, comprising, before applying the horizontal and vertical filtering for the reduction of noise, applying green non-uniformity correction to green components of the Bayer image data by:

for a current green pixel in the raw Bayer image data, identifying a 2×

2 array of pixels in the Bayer image data, the 2×

2 array comprising the current green pixel, a second green pixel, a red pixel, and a blue pixel;

determining a first value equal to the absolute value of the difference between the value of the current green pixel and the value of the second green pixel;

determining whether the first value is less than a green non-uniformity threshold value; and

replacing the value of the current green pixel with a second value equal to the average of the value of the current green pixel and the value of the second green pixel if the first value is less than the green non-uniformity threshold value.

7. One or more tangible computer-readable storage media having instructions encoded thereon for execution by a processor, the instructions comprising:

code to identify, in an image processing pipeline, defective pixels in raw image data received by the image processing pipeline, the raw image data corresponding to an image scene captured using a digital image sensor;

code to correct the detected defective pixels;

code to reduce noise in the raw image data;

code to correct lens shading distortion in the raw image data;

code to demosaic the raw image data to produce color image data;

code to correct color values of each pixel of the color image data using color correction coefficients stored in a color correction matrix;

code to adjust gamma values of each pixel of the color image data based upon a set gamma-corrected values stored in separate look-up tables for each color component of the color image data; and

code to convert the color image data into a corresponding set of image data in a luma and chroma color space.

8. The one or more tangible computer-readable storage media of claim 7, wherein the code to reduce noise in the raw image data is executed after the defective pixels are corrected, wherein the code to correct lens shading distortion is executed after the noise is reduced, and wherein the code to demosaic the raw image data is executed after the lens shading distortion is corrected.

9. The one or more tangible computer-readable storage media of claim 7, wherein the luma and chroma color space comprises the YCbCr color space.

10. The one or more tangible computer-readable storage media of claim 7, comprising:

code to sharpen the luma image data using a multi-scale unsharp mask having at least two low pass Gaussian filters with different radii;

code to apply at least one of brightness, contrast, color, and hue adjustments to the luma and chroma image data; and

code to cause a visual representation of the captured image scene based on the luma and chroma image data to be displayed on a display device.

11. The one or more tangible computer-readable storage media of claim 7, comprising code to apply chroma suppression to the chroma image data based at least partially upon luma sharpness values obtained from sharpening the luma image data.

12. An image signal processing system comprising:

image processing pipeline circuitry configured to process raw image data corresponding to an image scene captured by a digital image sensor, wherein the image processing pipeline circuitry comprises;

raw image data processing logic configured to convert the raw image data into a corresponding set of RGB image data, wherein the raw image data processing logic comprises defective pixel detection and correction logic configured to identify and correct defective pixels in the raw image data, noise reduction logic configured to reduce noise in the raw image data, lens shading correction logic configured to correct lens shading distortion in the raw image data, and demosaicing logic configured to convert the raw image data into the RGB image data; and

RGB processing logic configured to apply color correction and gamma adjustment to the RGB image data, and to convert the RGB image data into a corresponding set of image data in a luma and chroma color space;

wherein the RGB processing logic comprises color correction logic configured to apply color correction to the RGB image data by applying a color correction matrix to current red, blue, and green values associated with a current RGB pixel to determine corrected red, blue, and green values for the current RGB pixel, wherein the color correction matrix comprises a 3×

3 color correction matrix having a first row comprising first, second, and third color correction coefficients for correcting red color components, a second row comprising fourth, fifth, and sixth color correction coefficients for correcting green color components, and a third row comprising seventh, eighth, and ninth color correction coefficients for correcting blue color components;

wherein the corrected red value is determined by summing the product of the first color correction coefficient and the current red value, the product of the second color correction coefficient and the current green value, and the product of the third color correction coefficient and the current blue value;

wherein the corrected green value is determined by summing the product of the fourth color correction coefficient and the current red value, the product of the fifth color correction coefficient and the current green value, and the product of the sixth color correction coefficient and the current blue value; and

wherein the corrected blue value is determined by summing the product of the seventh color correction coefficient and the current red value, the product of the eighth color correction coefficient and the current green value, and the product of the ninth color correction coefficient and the current blue value.

13. The image signal processing system of claim 12, wherein the values of the color correction coefficients in each of the first, second, and third rows of the 3×

3 color correction matrix are selected such that the sum of the color correction coefficients in each row is equal to approximately 1.

14. The image signal processing system of claim 12, comprising a front-end statistics processing unit configured to process the raw image data to determine one or more sets of statistics information relating to at least one of the digital image sensor or the image scene before the raw image data is processed by the raw image data processing logic, and wherein the one or more sets of statistics data are used to determine the values of the 3×

3 color correction matrix.

15. The image signal processing system of claim 12, wherein the RGB processing logic comprises gamma adjustment logic comprising:

a first bank of look-up tables comprising first, second, and third look-up tables containing gamma-corrected values for red, blue, and green values, respectively, for RGB pixels of a current frame of the RGB image data;

wherein the gamma adjustment logic is configured to apply gamma correction to each RGB pixel in the current frame of the RGB image data by using the first, second, and third look-up tables to map the red, green, and blue value for each pixel in the current frame to corresponding gamma-corrected red, blue, and green values.

16. The image signal processing system of claim 15, wherein if a red, green, or blue value of a current RGB pixel in the current frame falls between two entries in a corresponding one of the first, second, or third look-up tables, the gamma-corrected value corresponding to the current RGB pixel is determined by linearly interpolating the gamma-corrected values corresponding to the two entries.

17. The image signal processing system of claim 15, comprising a second bank of look-up tables comprising fourth, fifth, and sixth look-up tables containing gamma-corrected values for red, blue, and green values, respectively, for a subsequent frame of the RGB image data, wherein the gamma-corrected values for the second bank of look-up tables may be updated while the gamma adjustment logic is applying gamma correction to the pixels of the current frame using the first bank of look-up tables.

18. The image signal processing system of claim 15, wherein the gamma adjustment logic applies gamma correction after the color correction logic applies color correction to the RGB image data.

19. An image signal processing system comprising:

image processing pipeline circuitry configured to process raw image data corresponding to an image scene captured by a digital image sensor, wherein the image processing pipeline circuitry comprises;

raw image data processing logic configured to convert the raw image data into a corresponding set of RGB image data, wherein the raw image data processing logic comprises defective pixel detection and correction logic configured to identify and correct defective pixels in the raw image data, noise reduction logic configured to reduce noise in the raw image data, lens shading correction logic configured to correct lens shading distortion in the raw image data, and demosaicing logic configured to convert the raw image data into the RGB image data; and

RGB processing logic configured to apply color correction and gamma adjustment to the RGB image data, and to convert the RGB image data into a corresponding set of image data in a luma and chroma color space; and

an image front-end processing unit, wherein the raw image data is first processed by the image front-end processing unit to acquire one or more sets of imaging statistics prior to being received by the image processing pipeline, and wherein the one or more sets of imaging statistics is utilized by the image processing pipeline for at least one of converting the raw image data into the RGB data and converting the RGB image data into the luma and chroma image data.

20. An electronic device, comprising:

at least one digital image sensor, wherein the digital image sensor comprises at least one of a Bayer color array filter, an RGBW color array filter, a CYGM color array filter, or any combination thereof;

an interface configured to communicate with the at least one digital image sensor;

a memory device;

a display device configured to display a visual representation of one or more image scenes corresponding to raw image data acquired by the at least one digital image sensor; and

an imaging signal processing sub-system comprising;

front-end processing logic configured to receive the raw image data and to process the raw image data to acquire one or more sets of statistics data relating to at least one of the raw image data or the digital image sensor; and

an image processing pipeline configured to receive the raw image data from the front-end processing unit and to convert the raw image data into a corresponding set of image data in a luma and chroma color space, wherein the image processing pipeline comprises;

a raw image data processing circuit configured to correct defective pixels in the raw image data, then reduce noise in the raw image data, then correct lens shading distortion in the raw image data, and then demosaic the raw image data into a corresponding set of RGB image data; and

an RGB image data processing circuit configured to apply at least one of color correction and gamma adjustment to the RGB image data, and to convert the RGB image data into the corresponding set of luma and chroma image data;

wherein the image processing pipeline is configured to output the luma and chroma image data to the memory device for storage or to the display device.

21. The electronic device of claim 20, wherein the at least one digital image sensor comprises a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via the interface, or some combination thereof.

22. The electronic device of claim 20, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof.

23. The electronic device of claim 20, wherein the displayed visual representation comprises still image data or video image data.

24. An electronic device, comprising:

at least one digital image sensor;

an interface configured to communicate with the at least one digital image sensor;

a memory device;

a display device configured to display a visual representation of one or more image scenes corresponding to raw image data acquired by the at least one digital image sensor; and

an imaging signal processing sub-system comprising;

front-end processing logic configured to receive the raw image data and to process the raw image data to acquire one or more sets of statistics data relating to at least one of the raw image data or the digital image sensor; and

an image processing pipeline configured to receive the raw image data from the front-end processing unit and to convert the raw image data into a corresponding set of image data in a luma and chroma color space, wherein the image processing pipeline comprises;

a raw image data processing circuit configured to correct defective pixels in the raw image data, then reduce noise in the raw image data, then correct lens shading distortion in the raw image data, and then demosaic the raw image data into a corresponding set of RGB image data; and

an RGB image data processing circuit configured to apply at least one of color correction and gamma adjustment to the RGB image data, and to convert the RGB image data into the corresponding set of luma and chroma image data;

wherein the image processing pipeline is configured to output the luma and chroma image data to the memory device for storage or to the display device; and

wherein the one or more sets of statistics data is utilized by the image processing pipeline for converting the raw image data into the RGB image data and the luma and chroma image data.

25. An electronic device, comprising:

at least one digital image sensor;

an interface configured to communicate with the at least one digital image sensor;

a memory device;

a display device configured to display a visual representation of one or more image scenes corresponding to raw image data acquired by the at least one digital image sensor; and

an imaging signal processing sub-system comprising;

front-end processing logic configured to receive the raw image data and to process the raw image data to acquire one or more sets of statistics data relating to at least one of the raw image data or the digital image sensor, wherein the one or more sets of statistics data comprises at least one of auto-exposure statistics, auto-white balance statistics, auto-focus statistics, flicker detection statistics, or any combination thereof; and

an image processing pipeline configured to receive the raw image data from the front-end processing unit and to convert the raw image data into a corresponding set of image data in a luma and chroma color space, wherein the image processing pipeline comprises;

a raw image data processing circuit configured to correct defective pixels in the raw image data, then reduce noise in the raw image data, then correct lens shading distortion in the raw image data, and then demosaic the raw image data into a corresponding set of RGB image data; and

an RGB image data processing circuit configured to apply at least one of color correction and gamma adjustment to the RGB image data, and to convert the RGB image data into the corresponding set of luma and chroma image data;

wherein the image processing pipeline is configured to output the luma and chroma image data to the memory device for storage or to the display device.

26. An electronic device, comprising:

at least one digital image sensor;

an interface configured to communicate with the at least one digital image sensor, wherein the interface comprises a Mobile Industry Processor Interface (MIPI), a Standard Mobile Imaging Architecture (SMIA) interface, or some combination thereof;

a memory device;

a display device configured to display a visual representation of one or more image scenes corresponding to raw image data acquired by the at least one digital image sensor; and

an imaging signal processing sub-system comprising;

front-end processing logic configured to receive the raw image data and to process the raw image data to acquire one or more sets of statistics data relating to at least one of the raw image data or the digital image sensor; and

an image processing pipeline configured to receive the raw image data from the front-end processing unit and to convert the raw image data into a corresponding set of image data in a luma and chroma color space, wherein the image processing pipeline comprises;

a raw image data processing circuit configured to correct defective pixels in the raw image data, then reduce noise in the raw image data, then correct lens shading distortion in the raw image data, and then demosaic the raw image data into a corresponding set of RGB image data; and

an RGB image data processing circuit configured to apply at least one of color correction and gamma adjustment to the RGB image data, and to convert the RGB image data into the corresponding set of luma and chroma image data;

wherein the image processing pipeline is configured to output the luma and chroma image data to the memory device for storage or to the display device.

27. The electronic device of claim 26, comprising an encoder, wherein the luma and chroma image data is encoded by the encoder or is output to the display device.

28. The electronic device of claim 26, wherein the encoder comprises at least one of a JPEG encoder or an H.264 encoder, or a combination thereof.