Image signal processor line buffer configuration for processing ram image data

• US 8,508,612 B2
• Filed: 09/30/2010
• Issued: 08/13/2013
• Est. Priority Date: 09/30/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Ct, Guy, Frederiksen, Jeffrey E., Bratt, Joseph P.
• Primary Examiner(s): Velez, Roberto
• Assistant Examiner(s): Tissire, Abdelaaziz
• Application Number:
  US12/895,396
• Publication Number:
  US 20120081578A1
• Time in Patent Office: 1,048 Days
• Field of Search: 348/272
• US Class Current: 348/222.1
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 23/632 for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/673 based on contrast or high frequency components of image signals, e.g. hill climbing method
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

The present disclosure provides techniques relates to the implementation of a raw pixel processing unit using a set of line buffers. In one embodiment, the set of line buffers may include a first subset and second subset. Various logical units of the raw pixel processing unit may be implemented using the first and second subsets of line buffers in a shared manner. For instance, in one embodiment, defective pixel correction and detection logic may be implemented using the first subset of line buffers. The second subset of line buffers may be used to implement lens shading correction logic, gain, offset, and clamping logic, and demosaicing logic. Further, noise reduction may also be implemented using at least a portion of each of the first and second subsets of line buffers.

Claims

1. A method for processing image data using an image signal processing system comprising:

using a digital image sensor to acquire raw image data comprising a plurality of raw pixels representative of an image scene;

providing the raw pixels to a raw image processing pipeline comprising a set of line buffers;

processing the raw pixels using the raw image processing pipeline, wherein processing the raw pixels using the raw image processing pipeline comprises;

applying a first set of gain, offset, and clamping parameters to the raw pixels using gain, offset, and clamping logic implemented in a first line of logic;

using defective pixel correction logic implemented using the first line of logic and a first subset of the set of line buffers to apply a defective pixel correction operation to the raw pixels;

using noise reduction logic implemented using the set of line buffers to apply noise reduction to the raw pixels, comprising;

using a horizontal filter implemented using at least one line buffer from the first subset of line buffers to apply horizontal filtering to the raw pixels; and

using a vertical filter implemented using at least one line buffer from the first subset of line buffers and at least a portion of the second set of line buffers to apply vertical filtering;

using lens shading correction logic implemented using a second subset of the set of line buffers to apply lens shading correction to the raw pixels;

applying a second set of gain, offset, and clamping parameters to the raw pixels using the second subset of line buffers; and

using demosaicing logic implemented using the second subset of line buffers to demosaic the raw pixels into corresponding color RGB pixels.

2. The method of claim 1, wherein the raw pixels comprise raw Bayer pixels having a first green color component and a second green color component.

3. The method of claim 2, wherein using the noise reduction logic implemented using the set of line buffers to apply noise reduction to the raw pixels further comprises using green non-uniformity (GNU) correction logic implemented using at least one line buffer from the first subset of line buffers to apply green non-uniformity correction to the raw pixels.

4. The method of claim 3, wherein applying green non-uniformity correction to the raw pixels comprises:

for a first green pixel, identifying a 2×

2 array of pixels in the raw Bayer pixels, the 2×

2 array comprising the first green pixel and a second green pixel, wherein the first green pixel corresponds to the first green color component and the second green pixel corresponds to the second green color component;

determining an absolute value of the difference between the value of the first green pixel and the value of the second green pixel;

determining whether the absolute value is less than a green non-uniformity threshold value; and

replacing the value of the first green pixel with a value equal to the mean of the value of the first green pixel and the value of the second green pixel if the absolute value is less than the green non-uniformity threshold value.

5. The method of claim 1, wherein the horizontal filter comprises a 7-tap filter, and wherein applying horizontal filtering comprises applying coefficients of the 7-tap filter to seven horizontally adjacent same-colored raw pixels.

6. The method of claim 1, wherein the vertical filter comprises a vertical filter configured to operate in a recursive mode and a non-recursive mode.

7. The method of claim 6, wherein applying vertical filtering to the raw pixels comprises operating the vertical filter as a 5-tap recursive filter if the vertical filter is operating in the recursive mode and operating the vertical filter as a 3-tap non-recursive filter if the vertical filter is operating in the non-recursive mode.

8. An image signal processing system comprising:

an image processing pipeline comprising a raw pixel processing unit, wherein the raw pixel processing unit is configured to process and convert raw pixels acquired using a digital image sensor into corresponding color RGB pixels and comprises a first row of logic and a plurality of line buffers comprising a first subset of line buffers and a second subset of line buffers, wherein the raw pixel processing unit comprises;

a first gain, offset, and clamping logic implemented using the first row of logic;

defective pixel correction and detection unit implemented using the first row of logic, the first line buffer, the second line buffer, the third line buffer, and the fourth line buffer;

noise reduction logic implemented using the first row of logic, the first line buffer, the second line buffer, the third line buffer, the fourth line buffer, the fifth line buffer, the sixth line buffer, the seventh line buffer, the eighth line buffer, the ninth line buffer, and the tenth line buffer;

lens shading correction logic implemented using the sixth line buffer, the seventh line buffer, the eighth line buffer, the ninth line buffer, and the tenth line buffer;

a second gain, offset, and clamping logic implemented using the sixth line buffer, the seventh line buffer, the eighth line buffer, the ninth line buffer, and the tenth line buffer; and

demosaicing logic implemented using the sixth line buffer, the seventh line buffer, the eighth line buffer, the ninth line buffer, and the tenth line buffer;

wherein the first subset of line buffers comprises a first line buffer configured to receive a first output from the first row of logic, a second line buffer configured to receive a second output from the first line buffer, a third line buffer configured to receive a third output from the second line buffer, a fourth line buffer configured to receive a fourth output from the third line buffer, and a fifth line buffer configured to receive a fifth output from the fourth line buffer; and

wherein the second subset of line buffers comprises a sixth line buffer configured to receive a sixth output from the fifth line buffer, a seventh line buffer configured to receive a seventh output from the sixth line buffer, an eighth line buffer configured to receive an eighth output from the seventh line buffer, a ninth line buffer configured to receive a ninth output from the eighth line buffer, and a tenth line buffer configured to receive a tenth output from the ninth line buffer.

9. The image signal processing system of claim 8, wherein the noise reduction logic comprises:

green non-uniformity (GNU) correction logic implemented using the first row of logic and the first line buffer;

a horizontal noise reduction filter implemented using the second line buffer; and

a vertical noise reduction filter implemented using the second line buffer, the third line buffer, the fourth line buffer, the fifth line buffer, the sixth line buffer, the seventh line buffer, the eighth line buffer, the ninth line buffer, and the tenth line buffer.

10. The image signal processing system of claim 9, wherein the vertical noise reduction filter is configured to operate in an infinite impulse response (IIR) mode and a finite impulse response (FIR) mode.

11. The image signal processing system of claim 10, wherein when operating in the IIR mode, the vertical noise reduction filter operates as a 5-tap filter implemented using the second line buffer, the third line buffer, the fourth line buffer, the fifth line buffer, the sixth line buffer, the seventh line buffer, the eighth line buffer, the ninth line buffer, and the tenth line buffer;

and

wherein when operating in the FIR mode, the vertical noise reduction filter operates as a 3-tap filter implemented using the second line buffer, the third line buffer, the fourth line buffer, the fifth line buffer, and the sixth line buffer.

12. The image signal processing system of claim 8, wherein the image processing pipeline comprises:

an RGB pixel processing unit configured to receive the color RGB pixels provided by the raw pixel processing unit and to convert the color RGB pixels into corresponding luma and chroma pixels of a luma and chroma color space; and

a luma and chroma pixel processing unit configured to receive and process the luma and chroma pixels provided by the RGB pixel processing unit.

13. A method for implementing a raw processing pipeline of an image signal processing system comprising:

providing a first line of logic configured to receive raw image pixels acquire using an image sensor, wherein the first line of logic is used to implement a first gain, offset, and clamping logic, a first portion of defective pixel correction logic, and a first portion of green non-uniformity (GNU) correction logic;

providing an arrangement of line buffers comprising a first subset of line buffers and a second subset of line buffers, wherein the first subset of line buffers comprises a first line buffer, a second line buffer, a third line buffer, a fourth line buffer, and a fifth line buffer, and wherein the second subset of logic comprises a sixth line buffer, a seventh line buffer, an eighth line buffer, a ninth line buffer, and a tenth line buffer, wherein the first subset of line buffers is used to implement a second portion of the defective pixel correction logic, a second portion of the GNU correction logic, a horizontal noise reduction filter, and a first portion of a vertical noise reduction filter and wherein the second subset of line buffers is used to implement a second portion of the vertical noise reduction filter, lens shading correction logic, a second gain, offset, and clamping logic, and demosaicing logic; and

communicatively coupling an output of the first line of logic to an input of the first line buffer, an output of the first line buffer to an input of the second line buffer, an output of the second line buffer to an input of the third line buffer, an output of the third line buffer to an input of the fourth line buffer, an output of the fourth line buffer to an input of the fifth line buffer, an output of the fifth line buffer to an input of the sixth line buffer, an output of the sixth line buffer to an input of the seventh line buffer, an output of the seventh line buffer to an input of the eighth line buffer, an output of the eighth line buffer to an input of the ninth line buffer, and an output of the ninth line buffer to an input of the tenth line buffer;

wherein the raw pixel processing pipeline is configured to process the raw image pixels to produce corresponding RGB image pixels.

14. The method of claim 13, wherein the defective pixel correction logic is implemented using the first line of logic, the first line buffer, the second line buffer, the third line buffer, and the fourth line buffer.

15. The method of claim 13, wherein the horizontal noise reduction filter is implemented using the second line buffer.

16. The method of claim 13, wherein the vertical noise reduction filter is implemented using the second, third, fourth, and fifth line buffers of the first subset of line buffers and the sixth, seventh, eighth, ninth, and tenth line buffers of the second subset of line buffers.

17. An electronic device comprising:

an image sensor configured to acquire raw image data comprising a plurality of raw pixels representative of an image scene;

an image signal processing sub-system; and

a sensor interface configured to provide the raw pixels from the image sensor to the image signal processing sub-system, wherein the image signal processing sub-system comprises an image pixel processing pipeline having a raw pixel processing unit comprising;

a first gain, offset, and clamping logic implemented using a first row of logic;

a first subset of line buffers comprising a first line buffer, a second line buffer, a third line buffer, a fourth line buffer, and a fifth line buffer;

a second subset of line buffers comprising a sixth line buffer, a seventh line buffer, an eighth line buffer, a ninth line buffer, and a tenth line buffer;

defective pixel correction and detection unit implemented using the first row of logic and at least a portion of the first subset of line buffers;

noise reduction logic implemented using the first row of logic, the first subset of line buffers, and the second subset of line buffers;

lens shading correction logic implemented using the second subset of line buffers;

a second gain, offset, and clamping logic implemented using the second subset of line buffers; and

demosaicing logic implemented using the second subset of line buffers, wherein the demosaicing logic is configured to convert the raw pixels into RGB pixels, wherein an output of the first row of logic is provided to an input of the first line buffer, wherein an output of the first line buffer is provided to an input of the second line buffer, wherein an output of the second line buffer is provided to an input of the third line buffer, wherein an output of the third line buffer is provided to an input of the fourth line buffer, wherein an output of the fourth line buffer is provided to an input of the fifth line buffer, wherein an output of the fifth line buffer is provided to an input of the sixth line buffer, wherein an output of the sixth line buffer is provided to an input of the seventh line buffer, wherein an output of the seventh line buffer is provided to an input of the eighth line buffer, wherein an output of the eighth line buffer is provided to an input of the ninth line buffer, wherein an output of the ninth line buffer is provided to an input of the tenth line buffer.

18. An electronic device comprising:

an image sensor configured to acquire raw image data comprising a plurality of raw pixels representative of an image scene;

an image signal processing sub-system; and

a sensor interface configured to provide the raw pixels from the image sensor to the image signal processing sub-system, wherein the image signal processing sub-system comprises an image pixel processing pipeline having a raw pixel processing unit comprising;

a first gain, offset, and clamping logic implemented using a first row of logic;

a first subset of line buffers comprising a first line buffer, a second line buffer, a third line buffer, a fourth line buffer, and a fifth line buffer;

a second subset of line buffers comprising a sixth line buffer, a seventh line buffer, an eighth line buffer, a ninth line buffer, and a tenth line buffer;

defective pixel correction and detection unit implemented using the first row of logic and at least a portion of the first subset of line buffers;

noise reduction logic implemented using the first row of logic, the first subset of line buffers, and the second subset of line buffers;

lens shading correction logic implemented using the second subset of line buffers;

a second gain, offset, and clamping logic implemented using the second subset of line buffers; and

demosaicing logic implemented using the second subset of line buffers, wherein the noise reduction logic comprises;

a horizontal filter implemented using at least one line buffer from the first subset of line buffers; and

a vertical filter implemented using the second set of line buffers and at least a portion of the first subset of line buffers.

19. An electronic device comprising:

an image sensor configured to acquire raw image data comprising a plurality of raw pixels representative of an image scene;

an image signal processing sub-system; and

a sensor interface configured to provide the raw pixels from the image sensor to the image signal processing sub-system, wherein the image signal processing sub-system comprises an image pixel processing pipeline having a raw pixel processing unit comprising;

a first gain, offset, and clamping logic implemented using a first row of logic;

a first subset of line buffers comprising a first line buffer, a second line buffer, a third line buffer, a fourth line buffer, and a fifth line buffer;

a second subset of line buffers comprising a sixth line buffer, a seventh line buffer, an eighth line buffer, a ninth line buffer, and a tenth line buffer;

defective pixel correction and detection unit implemented using the first row of logic and at least a portion of the first subset of line buffers;

noise reduction logic implemented using the first row of logic, the first subset of line buffers, and the second subset of line buffers;

lens shading correction logic implemented using the second subset of line buffers;

a second gain, offset, and clamping logic implemented using the second subset of line buffers; and

demosaicing logic implemented using the second subset of line buffers, comprising a control register, wherein the vertical filter is configured to operate in a recursive mode if the control register indicates a first state and in a non-recursive mode if the control register indicates a second state.

20. The electronic device of claim 19, wherein when operating in the recursive mode, the vertical filter functions as a 5-tap filter implemented using the second, third, fourth, fifth, sixth, seventh, eighth, ninth, and tenth line buffers, and wherein when operating in the non-recursive mode, the vertical filter functions as a 3-tap filter implemented using the second, third, fourth, fifth, and sixth line buffers.

21. The electronic device of claim 19, wherein the image sensor comprises a Bayer color filter array, and wherein the raw pixels comprise Bayer pixels.

22. The electronic device of claim 19, wherein image sensor comprises at least one of a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via the interface, or some combination thereof.

23. The electronic device of claim 19, wherein the sensor interface comprises a Standard Mobile Imaging Architecture (SMIA) interface, a serial interface, a parallel interface, or some combination thereof.

24. The electronic device of claim 19, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof.