Sensor data rescaler for image signal processing

• US 9,756,266 B2
• Filed: 12/21/2015
• Issued: 09/05/2017
• Est. Priority Date: 07/31/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Mills, Christopher L., Lin, Sheng, Pope, David R., Silverstein, D. Amnon, Lim, Suk Hwan
• Primary Examiner(s): Monk, Mark T
• Application Number:
  US14/977,384
• Publication Number:
  US 20160110843A1
• Time in Patent Office: 624 Days
• Field of Search: None
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 25/447 by preserving the colour pattern with or without loss of information
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

An input rescale module that performs cross-color correlated downscaling of sensor data in the horizontal and vertical dimensions. The module may perform a first-pass demosaic of sensor data, apply horizontal and vertical scalers to resample and downsize the data in the horizontal and vertical dimensions, and then remosaic the data to provide horizontally and vertically downscaled sensor data as output for additional image processing. The module may, for example, act as a front end scaler for an image signal processor (ISP). The demosaic performed by the module may be a relatively simple demosaic, for example a demosaic function that works on 3×3 blocks of pixels. The front end of module may receive and process sensor data at two pixels per clock (ppc); the horizontal filter component reduces the sensor data down to one ppc for downstream components of the input rescale module and for the ISP pipeline.

Claims

1. An apparatus, comprising:

an image signal processor comprising an image processing pipeline for processing images captured by an image sensor; and

an input rescaler configured to;

receive sensor data for an image at two pixels per clock according to an image sensor format that represents different color channels as separate pixels;

demosaic the received sensor data at two pixels per clock to generate color space data according to a color space that includes two or more color channels in each pixel;

resample the color space data to downscale the color space data and to reduce the color space data to one pixel per clock;

remosaic the downscaled color space data at one pixel per clock to generate downscaled sensor data according to the image sensor format; and

output the downscaled sensor data to the image processing pipeline of the image signal processor at one pixel per clock.

2. The apparatus as recited in claim 1, wherein the sensor data comprises pixel pairs according to the sensor format, each pixel pair including one pixel of a first color channel and one pixel of one of two other color channels, wherein, to demosaic the received sensor data at two pixels per clock, the input rescale module processes pixels of the first color channel on a first processing path and pixels of the two other color channels on a second processing path.

3. The apparatus as recited in claim 1, wherein, to generate color space data for a given pixel, the demosaic function is configured to interpolate from other pixels in a respective 3×

3 pixel block to estimate missing color channels for the pixel in the block and apply a blending function to the estimates to generate the color space data for the given pixel.

4. The apparatus as recited in claim 1, wherein the input rescale module is further configured to preprocess the received sensor data at two pixels per clock to detect and correct problems in the input sensor data prior to the demosaic.

5. The apparatus as recited in claim 1, wherein, to resample the color space data to downscale the color space data and to reduce the color space data to one pixel per clock, the input rescale module is configured to apply a horizontal filter to the color space data to horizontally downscale the color space data, wherein the horizontal filter receives pixels at two pixels per clock and outputs pixels at one pixel per clock.

6. The apparatus as recited in claim 5, wherein, to resample the color space data, the input rescale module is further configured to apply a vertical filter to the color space data output from the horizontal filter to vertically downscale the color space data.

7. A method, comprising:

converting, by an input rescale module, sensor data received at two pixels per clock according to an image sensor format that represents different color channels as separate pixels to color space data according to a color space that includes two or more color channels in each pixel, wherein said converting is performed at two pixels per clock;

resampling, by the input rescale module, the color space data to downscale the color space data and reduce the color space data to one pixel per clock;

converting, by the input rescale module, the downscaled color space data to the image sensor format to produce downscaled sensor data according to the image sensor format; and

outputting the downscaled sensor data to an image processing pipeline of an image signal processor at one pixel per clock.

8. The method as recited in claim 7, wherein the sensor data comprises pixel pairs according to the image sensor format, each pixel pair including one pixel of a first color channel and one pixel of one of two other color channels, wherein said converting comprises converting pixels of the first color channel on a first processing path and converting pixels of the two other color channels on a second processing path.

9. The method as recited in claim 7, wherein said converting the sensor data comprises applying a demosaic function to 3×

3 pixel blocks of the sensor data to estimate missing color channels for the pixels, wherein the demosaic function interpolates from other pixels in a given 3×

3 pixel block to estimate the missing color channels of a center pixel in the block and blends the estimates to generate the color space data for the center pixel.

10. The method as recited in claim 7, further comprising preprocessing, by the input rescale module at two pixels per clock, the sensor data prior to said converting the sensor data from the image sensor format to color space data.

11. The method as recited in claim 7, wherein the preprocessing comprises at least one of:

detecting and correcting pixel defects in the sensor data;

detecting and correcting green non-uniformity in the sensor data;

or

performing highlight recovery to prevent color bleeding in the horizontally and vertically downscaled sensor data.

12. The method as recited in claim 7, wherein said resampling comprises applying a horizontal filter to the color space data to horizontally downscale the color space data, wherein the horizontal filter receives pixels at two pixels per clock and outputs pixels at one pixel per clock.

13. The method as recited in claim 12, wherein said resampling further comprises applying a vertical filter to the horizontally downscaled color space data output by the horizontal filter to vertically downscale the color space data.

14. The method as recited in claim 7, wherein the image sensor format is a format in which each block of four sensor pixels includes two green pixels, one red pixel, and one blue pixel, wherein said converting the downscaled color space data to the image sensor format to produce downscaled sensor data according to the image sensor format comprises outputting one of the color channels for each pixel in the downscaled color space data, wherein the other two color channels are discarded for the pixel.

15. A device, comprising:

an image sensor configured to capture images according to a filter format that represents different color channels as separate pixels;

an image signal processor configured to process images in the filter format captured by the image sensor to generate output images in a different format, the image signal processor comprising an image processing pipeline that implements a plurality of stages each configured to perform one or more operations on image data passing through the pipeline; and

an input rescale module located between the image sensor and the image processing pipeline and configured to;

receive filter format image data at two pixels per clock;

apply a function at two pixels per clock to the filter format image data to convert the image data to another color space;

resample the color space data to downscale the color space data and to reduce the color space data to one pixel per clock apply another function at one pixel per clock to convert the downscaled color space data to the filter format; and

output the downscaled filter format data to a stage of the image processing pipeline at one pixel per clock.

16. The device as recited in claim 15, wherein the filter format image data comprises pixel pairs, each pixel pair including one green pixel and either one red pixel or one blue pixel, wherein, to apply the function at two pixels per clock to the filter format image data to convert the image data to the other color space, the input rescale module is configured to process green pixels on a first processing path and red and blue pixels on a second processing path.

17. The device as recited in claim 15, wherein the input rescale module is further configured to preprocess the received filter format image data at two pixels per clock to detect and correct problems in the filter format image data prior to applying the function.

18. The device as recited in claim 15, wherein, to resample the color space data, the input rescale module is configured to apply a horizontal filter to the color space data to horizontally downscale the color space data, wherein the horizontal filter receives pixels at two pixels per clock and outputs pixels at one pixel per clock.

19. The device as recited in claim 15, wherein, to resample the color space data, the input rescale module is further configured to apply a vertical filter to the horizontally downscaled color space data to vertically downscale the color space data.

20. The device as recited in claim 15, further comprising a memory configured to store raw images as captured by the image sensor, wherein the input rescale module is configured to receive the filter format image data from the image sensor or from the memory.