Image signal processor front-end image data processing system and method

• US 8,330,772 B2
• Filed: 05/28/2010
• Issued: 12/11/2012
• Est. Priority Date: 10/20/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cote, Guy, Frederiksen, Jeffrey E.
• Primary Examiner(s): McDowell, Jr., Maurice L
• Application Number:
  US12/789,810
• Publication Number:
  US 20110090380A1
• Time in Patent Office: 928 Days
• Field of Search: 345/611, 348/246
• US Class Current: 345/611
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 25/611 Correction of chromatic aberration
• CPC Class Codes: H04N 25/683 by defect estimation performed on the scene signal, e.g. real time or on the fly detection

Abstract

Various techniques are provided herein for processing raw image data in front-end processing logic of an image signal processing system. In one embodiment, the front-end processing logic includes a statistics processing unit configured to process raw image data acquired by an image sensor to obtain one or more sets of statistics. The statistics processing unit may first correct defective pixels in the raw image data and then correct lens shading errors in the raw image data prior to extracting the statistics information. In certain embodiments, black level compensation may be applied between the defective pixel correction and lens shading correction steps, and inverse black level compensation may be applied between the lens shading correction step and the extraction of the statistics information. The acquired statistics information may be utilized by an image signal processing pipeline for converting the raw image data into a color (e.g., RGB) and/or luma (e.g., YCbCr) image.

Claims

1. A method for processing raw image data using front-end processing logic of an image signal processor comprising:

using an image processor;

receiving the raw image data, the raw image data having been acquired using an image sensor;

processing the raw image data to detect and correct defective pixels in the raw image data;

processing the raw image data to correct for lens shading distortion after detecting and correcting the defective pixels, wherein correcting for the lens shading distortion comprises applying a lens shading gain to each pixel of the raw image data, the lens shading gain being determined based upon based upon one or more grid points in a gain grid;

after the correcting the raw image data for lens shading distortion, processing the raw image data to determine one or more sets of statistics indicative of one or more properties of the image sensor;

applying black level compensation to the raw image data after detecting and correcting the defective pixels but before correcting the lens shading distortion; and

applying inverse black level compensation to the raw image data after correcting for the lens shading distortion but before determining the one or more sets of statistics.

2. The method of claim 1, wherein applying the black level compensation to the raw image data comprises:

for each pixel of the raw image data;

applying an offset to a current pixel;

applying a digital gain to the current pixel after applying the offset; and

after applying the digital gain, applying a clipping factor to the current pixel if the value of the current pixel is above a particular maximum value or below a particular minimum value.

3. The method of claim 2, wherein the raw image data comprises multiple color components, and wherein the offset, digital gain, and clipping factor are dependent upon the color component to which the current pixel corresponds.

4. The method of claim 2, wherein applying the inverse black level compensation to the raw image data comprises:

for each pixel of the raw image data;

applying the digital gain to a current pixel

applying an offset to the current pixel after applying the digital gain; and

after applying the offset, applying a clipping factor to the current pixel if the value of the current pixel is above a particular maximum value or below a particular minimum value.

5. The method of claim 1, wherein the one or more sets of statistics determined by the front-end processing logic comprises at least one of auto-exposure statistics, auto-white balance statistics, auto-focus statistics, flicker detection statistics, or some combination thereof.

6. The method of claim 1, wherein the raw image data comprises Bayer image data.

7. The method of claim 1, comprising converting the raw image data into a color space corresponding to a display device, and displaying the image data on a display device.

8. A method for processing raw image data using front-end processing logic of an image signal processor comprising:

using an image processor;

receiving the raw image data, the raw image data having been acquired using an image sensor;

processing the raw image data to detect and correct defective pixels in the raw image data;

processing the raw image data to correct for lens shading distortion after detecting and correcting the defective pixels, wherein correcting for the lens shading distortion comprises applying a lens shading gain to each pixel of the raw image data, the lens shading gain being determined based upon based upon one or more grid points in a gain grid; and

after the correcting the raw image data for lens shading distortion, processing the raw image data to determine one or more sets of statistics indicative of one or more properties of the image sensor;

wherein detecting and correcting the defective pixels in the raw image data comprises;

receiving a current input pixel of the raw image data;

identifying four horizontal neighboring pixels of the same color as the current input pixel including two horizontal neighboring pixels located on a first side of the current input pixel and two horizontal neighboring pixels located on a second side of the current input pixel;

determining pixel-to-pixel gradient values with respect to the current input pixel for each of the four horizontal neighboring pixels located within a processing region defined within an imaging frame of the image sensor;

determining a count of the number of gradient values that are less than or equal to a defective pixel threshold value;

determining whether the count is less than a maximum count value; and

if the count is less than the maximum count value, identifying the current input pixel as a defective pixel and correcting the defective pixel.

9. The method of claim 8, wherein correcting the defective pixel comprises:

determining the value of a first neighboring pixel having the same color as the current input pixel and located immediately adjacent to a first side of the input pixel in the imaging frame, wherein the first neighboring pixel is a pixel that has already been processed for defective pixel detection and correction;

determining whether the first neighboring pixel is within the processing region of the imaging frame;

replacing the value of the current input pixel with the value of the first neighboring pixel if the first neighboring pixel is within the processing region;

if the first neighboring pixel is not within the processing region, determining the value of a second neighboring pixel having the same color as the current input pixel and located immediately adjacent to a second side of the input pixel in the imaging frame, wherein the second side is opposite the first side, and wherein the second neighboring pixel is a pixel that has not yet been processed for defective pixel detection and correction; and

replacing the value of the current input pixel with the value of the second neighboring pixel.

10. An imaging processing system comprising:

a memory device; and

front-end processing logic comprising;

a front-end statistics processing unit configured to receive raw image data acquired using an image sensor, to process the raw image data to acquire a set of statistics for each frame of the raw image data processed by the front-end statistics processing unit; and

a front-end pixel processing unit configured to process the raw image data for at least one of reducing noise in the raw image data or scaling the raw image data and to provide the raw image data processed by the front-end pixel processing unit to image pipeline processing logic coupled downstream from the front-end processing logic;

wherein the sets of statistics data acquired by the front-end statistics processing unit are stored in the memory device and utilized by the image pipeline processing logic to convert the raw image data into at least one of an RGB image or a luma image;

wherein the front-end statistics processing unit comprises;

defective pixel detection and correction logic configured to detect and correct defective pixels in the raw image data;

lens shading correction logic configured to correct for lens shading error in the raw image data after the defective pixels are detected and corrected by the defective pixel detection and correction logic; and

statistics collection logic configured to acquire the sets of statistics data after the lens shading error in the raw image data is corrected by the lens shading correction logic; and

wherein the first front-end statistics processing unit comprises;

black level compensation logic configured to apply black level compensation to the raw image data after the defective pixels are detected and corrected by the defective pixel detection and correction logic, but before the lens shading error is corrected by the lens shading correction logic; and

inverse black level compensation logic configured to apply inverse black level compensation to the raw image data after the lens shading error is corrected by the lens shading correction logic, but before the sets of statistics data are acquired by the statistics collection logic.

11. The image processing system of claim 10, wherein each set of statistics data comprises at least one of auto-exposure statistics, auto-white balance statistics, flicker detection statistics, and auto-focus statistics, or any combination thereof, pertaining to a respective frame of the raw image data.

12. The image processing system of claim 10, wherein the statistics collection logic comprises a memory interface by which the sets of statistics data may be written to the memory device.

13. An imaging processing system comprising:

a memory device; and

front-end processing logic comprising;

a front-end statistics processing unit configured to receive raw image data acquired using an image sensor, to process the raw image data to acquire a set of statistics for each frame of the raw image data processed by the front-end statistics processing unit; and

a front-end pixel processing unit configured to process the raw image data for at least one of reducing noise in the raw image data or scaling the raw image data and to provide the raw image data processed by the front-end pixel processing unit to image pipeline processing logic coupled downstream from the front-end processing logic;

wherein the sets of statistics data acquired by the front-end statistics processing unit are stored in the memory device and utilized by the image pipeline processing logic to convert the raw image data into at least one of an RGB image or a luma image;

wherein the front-end pixel processing unit is configured to reduce noise in the raw image data by applying a temporal filter to the raw image data.

14. An imaging processing system comprising:

a memory device; and

front-end processing logic comprising;

a front-end statistics processing unit configured to receive raw image data acquired using an image sensor, to process the raw image data to acquire a set of statistics for each frame of the raw image data processed by the front-end statistics processing unit; and

a front-end pixel processing unit configured to process the raw image data for at least one of reducing noise in the raw image data or scaling the raw image data and to provide the raw image data processed by the front-end pixel processing unit to image pipeline processing logic coupled downstream from the front-end processing logic;

wherein the sets of statistics data acquired by the front-end statistics processing unit are stored in the memory device and utilized by the image pipeline processing logic to convert the raw image data into at least one of an RGB image or a luma image;

wherein the front-end pixel processing unit is configured to scale the raw image data by applying a binning compensation filter utilizing polyphase filters in horizontal and vertical directions.

15. The image processing system of claim 14, wherein the one or more sets of statistics is used to convert the raw image data into at least one of color image data and luma and chroma image data.

16. One or more tangible computer-readable storage media having instructions encoded thereon for execution by a processor, the instructions comprising:

code to detect defective pixels in raw image data acquired using an image sensor;

code to correct the detected defective pixels;

code to correct lens shading error in the raw image data; and

code to determine one or more sets of statistics corresponding to one or more image properties of the raw image data;

wherein the code to correcting the lens shading error in the raw image data comprises;

code to determine a color component of a current input pixel of the raw image data;

code to determine the location of the current input pixel relative to a gain grid comprising a plurality of grid points, wherein each of the plurality of grid points is associated with a lens shading gain selected based upon the color of the current input pixel;

code to applying a lens shading gain associated with a grid point to the current input pixel if the location of the current input pixel corresponds directly to the grid point; and

code to identifying a nearest set of grid points that encloses the current input pixel, determining a lens shading gain by interpolating the lens shading gains associated with each of the identified set of grid points, and applying the interpolated lens shading gain to the current input pixel if the location of the input pixel does not correspond directly to one of the plurality of grid points.

17. The one or more tangible computer-readable storage media of claim 16, wherein the code to correct the lens shading error is executed after the code to correct the defective pixels.

18. The one or more tangible computer-readable storage media of claim 17, wherein the code to determine the one or more sets of statistics is executed after the code to correct the lens shading error.

19. The one or more tangible computer-readable storage media of claim 16, wherein the gain grid comprises a radial gain grid, and wherein the code to interpolate the lens shading gain uses bi-linear interpolation of the lens shading gains associated with each of the identified set of grid points that enclose the input pixel in radial angular directions.

20. An electronic device comprising:

a memory device;

a first image sensor comprising a first sensor interface and configured to acquire a first set of raw image data; and

an image processing system comprising;

front-end image processing logic comprising a first front-end statistics processing unit configured to receive the first set of raw image data acquired by the first image sensor and to acquire a first set of statistics for each frame of the first set of raw image data processed by the first front-end statistics processing unit, wherein the first front-end statistics processing unit comprises;

a first defective pixel detection and correction logic configured to detect and correct defective pixels in the first set of raw image data;

a first lens shading correction logic configured to correct for lens shading error in the first set of raw image data after the defective pixels are detected and corrected by the first defective pixel detection and correction logic; and

a first statistics collection logic configured to acquire the first sets of statistics data after the lens shading error in the first set of raw image data is corrected by the first lens shading correction logic and to write the first sets of statistics data to the memory device; and

a display device configured to display an image corresponding to the first set of raw image data acquired by the first image sensor after the first set of raw image data is processed by the image processing system;

wherein the front-end image processing logic comprises;

a second image sensor comprising a second sensor interface and configured to acquire a second set of raw image data;

a second front-end statistics processing unit configured to receive the second raw image data acquired by the second image sensor and to acquire a second set of statistics for each frame of the raw image data processed by the second front-end statistics processing unit, wherein the second front-end statistics processing unit comprises;

a second defective pixel detection and correction logic configured to detect and correct defective pixels in the second set of raw image data;

a second lens shading correction logic configured to correct for lens shading error in the second set of raw image data after the defective pixels are detected and corrected by the second defective pixel detection and correction logic; and

a second statistics collection logic configured to acquire the second sets of statistics data after the lens shading error in the second set of raw image data is corrected by the second lens shading correction logic and to write the second sets of statistics data to the memory device;

wherein the display device is configured to display an image corresponding to the second set of raw image data acquired by the second image sensor after the second set of raw image data is processed by the image processing system.

21. The electronic device of claim 20, wherein the first image sensor and the second image sensors are each configured to provide the first set raw image data and the second set of raw image data, respectively, either directly to the memory device or directly to the first and second front-end statistics processing units, respectively, using the first and second sensor interfaces, respectively.

22. The electronic device of claim 20, wherein the first sensor interface and the second sensor interface comprises a Mobile Industry Processor Interface (MIPI), a Standard Mobile Imaging Architecture (SMIA) interface, or some combination thereof.

23. The electronic device of claim 20, wherein the display device comprises an LED display, an LCD display, plasma display, OLED display, or some combination thereof.

24. The electronic device of claim 23, wherein the display device comprises a touchscreen interface.

25. The electronic device of claim 20,

wherein the memory device comprises a memory device having a direct memory access (DMA) interface.

26. An electronic device comprising:

a memory device;

a first image sensor comprising a first sensor interface and configured to acquire a first set of raw image data; and

an image processing system comprising;

front-end image processing logic comprising a first front-end statistics processing unit configured to receive the first set of raw image data acquired by the first image sensor and to acquire a first set of statistics for each frame of the first set of raw image data processed by the first front-end statistics processing unit, wherein the first front-end statistics processing unit comprises;

a first defective pixel detection and correction logic configured to detect and correct defective pixels in the first set of raw image data;

a first lens shading correction logic configured to correct for lens shading error in the first set of raw image data after the defective pixels are detected and corrected by the first defective pixel detection and correction logic; and

a first statistics collection logic configured to acquire the first sets of statistics data after the lens shading error in the first set of raw image data is corrected by the first lens shading correction logic and to write the first sets of statistics data to the memory device; and

a display device configured to display an image corresponding to the first set of raw image data acquired by the first image sensor after the first set of raw image data is processed by the image processing system;

comprising a selection circuit configured to select an input image signal from the memory device or from the first sensor interface, and to provide the selected input image signal to the first front-end statistics processing unit.

27. An electronic device comprising:

a memory device;

a first image sensor comprising a first sensor interface and configured to acquire a first set of raw image data; and

an image processing system comprising;

front-end image processing logic comprising a first front-end statistics processing unit configured to receive the first set of raw image data acquired by the first image sensor and to acquire a first set of statistics for each frame of the first set of raw image data processed by the first front-end statistics processing unit, wherein the first front-end statistics processing unit comprises;

a first defective pixel detection and correction logic configured to detect and correct defective pixels in the first set of raw image data;

a first lens shading correction logic configured to correct for lens shading error in the first set of raw image data after the defective pixels are detected and corrected by the first defective pixel detection and correction logic; and

a first statistics collection logic configured to acquire the first sets of statistics data after the lens shading error in the first set of raw image data is corrected by the first lens shading correction logic and to write the first sets of statistics data to the memory device; and

a display device configured to display an image corresponding to the first set of raw image data acquired by the first image sensor after the first set of raw image data is processed by the image processing system;

wherein the electronic device comprises a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a personal digital assistant, a portable digital media player, or some combination thereof.