BINNING COMPENSATION FILTERING TECHNIQUES FOR IMAGE SIGNAL PROCESSING

• US 20120026368A1
• Filed: 07/29/2010
• Published: 02/02/2012
• Est. Priority Date: 07/29/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Côté, Guy, Frederiksen, Jeffrey E., Bratt, Joseph P.
• Granted Patent: US 8,525,895 B2
• Time in Patent Office: Days
• US Class Current: 348/242
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns

Abstract

Various techniques for applying binning compensation filtering to binned raw image data acquired by an image sensor are provided. In one embodiment, a binning compensation filter (BCF) includes separate digital differential analyzers (DDA) for vertical and horizontal scaling. A current position of an output pixel is determined by incrementing the DDA based upon a step size. Using the known output pixel position, a center source input pixel and an index corresponding to the between-pixel fractional position of the output pixel position relative to the input pixels may be selected for filtering. Using the selected center input pixel, one or more same-colored neighboring source pixels may be selected. The number of selected source pixels may depend on the number of taps used by the scaling logic, and may depend on whether horizontal or vertical scaling is being applied. Using the selected index, a set of filter coefficients may be selected from a filter coefficient lookup table, applied to the selected source pixels, and the results may be summed to determine a value for an output pixel having a position corresponding to the current position of the DDA. This process may be repeated for each input pixel and may be performed in both vertical and horizontal directions, thus ultimately producing a re-sampled set of image data that is spatially evenly distributed.

Claims

1. A method for processing raw image data comprising:

using a binning compensation filter of an image signal processor;

using a position value stored by a position register to determine a plurality of output pixel locations, wherein each of the plurality of output pixel locations is determined by incrementing the position value using a step value;

for each output pixel location corresponding to a current position value stored in the position register;

using the current position value to select a center input pixel from the raw image data and to select an index value, wherein the index value represents a fractional position of a current output pixel at the current position value between two input pixels of the same color, and wherein the raw image data comprises multiple color components in a single plane and corresponds to an image scene captured by a digital image sensor;

selecting a set of input pixels adjacent to the center input pixel, wherein the adjacent input pixels and the center input pixel form a set of source input pixels;

selecting a set of filtering coefficients from a filter coefficient lookup table using the index value;

filtering the set of source input pixels using scaling logic to apply a respective one of the set of filtering coefficients to each of the set of source input pixels to determine an output value for the current output pixel at the current position value; and

processing the output pixels provided by the binning compensation filter to display a visual representation of the image scene on a display device.

2. The method of claim 1, wherein selecting the center input pixel comprises:

determining if the current position value corresponds to an even or odd output pixel position;

if the current position value corresponds to an even output pixel position, incrementing the current position value by 1 and rounding the result to the nearest even input pixel, wherein the nearest even input pixel is selected as the center input pixel; and

if the current position value corresponds to an odd output pixel position, rounding the current position value to the nearest odd input pixel, wherein the nearest odd input pixel is selected as the center input pixel.

3. The method of claim 2, wherein determining whether the current position value is even or odd comprises:

determining the value of the least significant bit of the position register, the least significant bit being defined by the step value;

identifying the current position value as an even output pixel position if the least significant bit is 0; and

identifying the current position value as an odd output pixel position if the least significant bit is 1.

4. The method of claim 1, wherein the position register is configured to store a first number of bits corresponding to an integer portion of the current position value, and a second number of bits corresponding to a fraction portion of the current position value.

5. The method of claim 1, wherein the position register comprises a 32-bit register, and wherein the first number of bits comprises 16 bits and the second number of bits comprises 16 bits.

6. The method of claim 4, wherein each input pixel of the raw image data is separated from an adjacent input pixel of the same color by a particular number of index steps, and wherein determining the index value comprises:

determining if the current position value corresponds to an even or odd output pixel position;

if the current position value corresponds to an even output pixel position, incrementing the current position value by one index step and evaluating the lowest order bit of the integer portion and the two highest order bits of the fraction portion to determine the index value; and

if the current position value corresponds to an odd output pixel position, incrementing the current position value by a value equivalent to the sum of 1 and one index step and evaluating the lowest order bit of the integer portion and the two highest order bits of the fraction portion to determine the index value.

7. The method of claim 6, wherein the particular number of index steps between adjacent input pixels of the same color is eight.

8. The method of claim 1, wherein filtering the set of source input pixels comprises using horizontal and vertical scaling logic.

9. An image signal processing system comprising:

a binning compensation filter comprising;

an input configured to receive raw image data acquired by a digital image sensor;

a digital differential analyzer configured to indicate a position of a current output pixel based upon a step size;

binning compensation logic configured to select a center input pixel and an index value using the position of the current output pixel; and

scaling logic configured to apply multi-tap polyphase filtering to a set of source input pixels selected from the raw image data based upon a set of filtering coefficients selected using the index value, and to determine an output value for the current output pixel based upon the filtering, wherein the center input pixel is positioned at the center of the source input pixels.

10. The image signal processing system of claim 9, wherein the raw image data is binned by the digital image sensor prior to being provided to the input of the binning compensation filter.

11. The image signal processing system of claim 10, wherein the digital image sensor is configured to apply 2×

2 binning to a captured set of full resolution image data to provide the binned raw image data.

12. The image signal processing system of claim 11, wherein each binned pixel of the binned raw image data is determined by:

identifying first, second, third, and fourth pixels of a particular color component, wherein each of the first, second, third, and fourth pixels are spatially separated from one another by one pixel of a different color component, and wherein the first, second, third, and fourth pixels form the top left, top right, bottom left, and bottom right corners, respectively of a square; and

using the values of the first, second, third, and fourth to interpolate a value for the binned pixel, the binned pixel being spatially located at the center of the square.

13. The image signal processing system of claim 12, wherein the value of the binned pixel is determined as the mean of the values of the first, second, third, and fourth pixel.

14. The image signal processing system of claim 9, wherein the center pixel is selected by determining whether the position of the current output pixel is odd or even based on the value of the bit in the differential analyzer corresponding to the least significant bit based on the step size, selecting the nearest even input pixel as the center input pixel if the position of the current output pixel is even, and selecting the nearest odd input pixel as the center input pixel if the position of the current output pixel is odd.

15. The image signal processing system of claim 9, wherein the binning compensation filter is configured to apply binning compensation filtering on the raw image data prior to a demosaicing operation.

16. The image signal processing system of claim 9, wherein the raw image data received by the input of the binning compensation filter is not evenly spatially distributed, and wherein the output of the binning compensation filter comprises a re-sampled set of raw image data that is evenly spatially distributed.

17. One or more tangible computer-readable storage media having instructions encoded thereon for execution by a processor, the instructions comprising:

code to determine a position of a current output pixel based upon a current position value stored in a digital differential analyzer;

code to select a center input pixel from a set of raw image data received by a binning compensation filter using the position of the current output pixel;

code to select an index value using the position of the current output pixel relative to two input pixels of the same color from the set of raw image data, the position of the current output pixel being located spatially between the positions of the two input pixels;

code to select a set of input pixels having the same color and being adjacent to the selected center input pixel;

code to select a set of filtering coefficients using the index value; and

code to apply scaling to each of the center input pixel and the adjacent input pixels to determine an output value for the position of the current output pixel by applying a respective one of the filtering coefficients to each of the center input pixel and the adjacent input pixels.

18. The one or more tangible computer-readable storage media of claim 17, wherein the code to determine the output value for the current output pixel is configured to multiply each of the center input pixel and the adjacent input pixels by a respective filtering coefficient, and to sum the results to determine the output value for the current output pixel.

19. The one or more tangible computer-readable storage media of claim 17, wherein the code to apply scaling comprises code to apply horizontal scaling using a 5-tap horizontal polyphase filter, wherein the adjacent input pixels comprise two adjacent input pixels of the same color located spatially to the left of the center input pixel and two adjacent input pixels of the same color located spatially to the right of the center input pixel.

20. The one or more tangible computer-readable storage media of claim 17, wherein the code to apply scaling comprises code to apply vertical scaling using a 3-tap vertical polyphase filter, wherein the adjacent input pixels comprise one adjacent input pixel of the same color located spatially above the center input pixel and one adjacent input pixel of the same color located spatially below the center input pixel.

21. The one or more tangible computer-readable storage media of claim 17, comprising code to increment the digital differential analyzer by a particular step size greater than 1 to determine a position of a next output pixel.

22. An electronic device, comprising:

at least one digital image sensor;

an interface configured to communicate with the at least one digital image sensor;

a memory device;

a display device configured to display a visual representation of one or more image scenes corresponding to raw image data acquired by the at least one digital image sensor; and

an imaging signal processing sub-system comprising;

a binning compensation filter comprising;

a digital differential analyzer configured to indicate a position of a current output pixel based upon a step size;

an input configured to receive raw image data acquired by a digital image sensor;

binning compensation logic configured to select a center input pixel and an index value using the position of the current output pixel; and

scaling logic configured to apply multi-tap polyphase filtering to a set of source input pixels comprising an even number of adjacent input pixels distributed about the center input pixel and having the same color as the center input pixel using a set of filtering coefficients selected using the index value, and to determine an output value for the current output pixel based upon the filtering.

23. The electronic device of claim 22, wherein the at least one digital image sensor comprises at least one of a Bayer color array filter, an RGBW color array filter, a CYGM color array filter, or any combination thereof.

24. The electronic device of claim 22, wherein the at least one digital image sensor comprises binning logic configured to bin full resolution image data to produce a set of binned raw image data, wherein the binned raw image data is provided to the input of the binning compensation filter.

25. The electronic device of claim 24, wherein the binning logic is configured to apply 2×

2 binning, 3×

3 binning, horizontal binning, vertical binning, skipping, or some combination thereof.

26. The electronic device of claim 22, wherein the image signal processing subsystem comprises an image processing pipeline configured to receive the output pixels from the binning compensation filter, to convert the output pixels into a corresponding set of RGB image data, and to convert the RGB image data into a corresponding set of luma and chroma image data;

wherein the image processing pipeline is configured to output the luma and chroma image

data to the memory device for storage or to the display device.

27. The electronic device of claim 22, wherein the at least one digital image sensor comprises a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via the interface, or some combination thereof.

28. The electronic device of claim 22, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof.

29. The electronic device of claim 22, wherein the interface comprises a Mobile Industry Processor Interface (MIPI), a Standard Mobile Imaging Architecture (SMIA) interface, or some combination thereof.

30. The electronic device of claim 22, wherein the digital differential analyzer is incremented by the step size after the value of the output pixel is determined, and wherein the binning compensation filter determines an output value corresponding to the next output pixel located at a position corresponding to the incremented value stored in the digital differential analyzer.