Image signal processing involving geometric distortion correction

• US 8,917,336 B2
• Filed: 05/31/2012
• Issued: 12/23/2014
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cote, Guy, Butler, Simon Wolfenden, Bratt, Joseph P.
• Primary Examiner(s): Velez, Roberto
• Assistant Examiner(s): COLEMAN, STEPHEN P
• Application Number:
  US13/484,842
• Publication Number:
  US 20130321674A1
• Time in Patent Office: 936 Days
• Field of Search: 348/241, 348/246, 348/220, 348/231.99, 382/167
• US Class Current: 348/241
• CPC Class Codes: G06T 5/006 Geometric correction correction of chromatic aberrations adapted to be used in an image pickup device containing an electronic image sensor H04N23/10; detecting, correcting, reducing or removing artefacts resulting only from the lens unit, e.g. flare, shading, vignetting or "cos4" H04N25/61
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/673 based on contrast or high frequency components of image signals, e.g. hill climbing method
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/136 using complementary colours
• CPC Class Codes: H04N 25/611 Correction of chromatic aberration
• CPC Class Codes: H04N 9/64 Circuits for processing colour signals H04N9/77 takes precedence; camera processing pipelines for processing colour signals H04N23/84

Abstract

Systems and methods for correcting geometric distortion are provided. In one example, an electronic device may include an imaging device, which may obtain image data of a first resolution, and geometric distortion and scaling logic. The imaging device may include a sensor and a lens that causes some geometric distortion in the image data. The geometric distortion correction and scaling logic may scale and correct for geometric distortion in the image data by determining first pixel coordinates in uncorrected or partially corrected image data that, when resampled, would produce corrected output image data at second pixel coordinates. The geometric distortion correction and scaling logic may resample pixels around the image data at the first pixel coordinates to obtain the corrected output image data at the second pixel coordinates. The corrected output image data may be of a second resolution.

Claims

1. An image signal processing system comprising:

chromatic aberration correction logic configured to at least partially correct for chromatic aberration in image data of a first resolution while the image data is in a Bayer raw format; and

geometric distortion correction logic configured to at least partially correct for geometric distortion in the image data while the image data is in a YCC format and to scale the image data while the image data is in a YCC format, wherein the geometric distortion correction logic comprises;

a first plurality of line buffers configured to store lines of luma component image data;

a vertical luma scaler configured to scale and correct for geometric distortion in a vertical direction using the lines of luma component image data, wherein the vertical luma scaler comprises;

vertical luma coordinate generation logic configured to determine vertically correcting pixel coordinates that, when resampled, would produce vertically geometrically corrected pixels; and

a vertical luma resampling filter configured to resample pixels across a plurality of lines in the line buffers at the vertically correcting pixel coordinates to obtain the vertically geometrically corrected pixels, wherein the vertically geometrically corrected pixels are stored in the line buffers;

a horizontal luma scaler configured to scale and correct for geometric distortion in a horizontal direction using the lines of luma component image data, wherein the horizontal luma scaler comprises;

horizontal luma coordinate generation logic configured to determine horizontally correcting pixel coordinates that, when resampled, would produce horizontally and vertically geometrically corrected pixels; and

a horizontal luma resampling filter configured to resample a plurality of horizontal pixels of a line in the line buffers at the horizontally correcting pixel coordinates to obtain the horizontally and vertically geometrically corrected pixels;

a second plurality of line buffers configured to store lines of chroma component image data;

a vertical chroma scaler configured to scale and correct for geometric distortion in the vertical direction using the lines of chroma component image data, wherein the vertical chroma scaler comprises;

vertical chroma coordinate generation logic configured to determine vertically correcting pixel coordinates that, when resampled, would produce vertically geometrically corrected pixels; and

a vertical chroma resampling filter configured to resample pixels across a plurality of lines in the line buffers at the vertically correcting pixel coordinates to obtain the vertically geometrically corrected pixels, wherein the vertically geometrically corrected pixels are stored in the line buffers; and

a horizontal chroma scaler configured to scale and correct for geometric distortion in the horizontal direction using the lines of chroma component image data, wherein the horizontal chroma scaler comprises;

horizontal chroma coordinate generation logic configured to determine horizontally correcting pixel coordinates that, when resampled, would produce horizontally and vertically geometrically corrected pixels; and

a horizontal chroma resampling filter configured to resample a plurality of horizontal pixels of a line in the line buffers at the horizontally correcting pixel coordinates to obtain the horizontally and vertically geometrically corrected pixels.

2. The image signal processing system of claim 1, wherein the chromatic aberration correction logic comprises a component of a raw processing pipeline configured to perform a plurality of operations on Bayer raw image data.

3. The image signal processing system of claim 1, wherein the geometric distortion correction logic comprises a component of a YCC processing pipeline configured to perform a plurality of operations on YCC image data.

4. The image signal processing system of claim 1, wherein the chromatic aberration correction logic and the geometric distortion logic are separated by RGB processing logic configured to perform a plurality of operations on the image data while the image data is in an RGB format.

5. An electronic device comprising:

an imaging device configured to obtain image data of a first resolution, wherein the imaging device comprises a sensor and a lens that causes some geometric distortion in the image data; and

geometric distortion correction and scaling logic configured to scale and correct for geometric distortion in the image data by;

storing lines of luma component image data of the uncorrected image frame in a first plurality of line buffers;

scaling and correcting for geometric distortion in a vertical direction using the lines of the luma component image data by;

using vertical luma coordinate generation logic to determine vertically correcting luma pixel coordinates that when resampled, would produce geometrically corrected luma pixels;

using a vertical luma resampling filter to resample pixels across a plurality of lines in the first plurality of line buffers at the vertically correcting luma pixel coordinates to obtain the vertically geometrically corrected luma pixels; and

storing the vertically geometrically corrected luma pixels in the first plurality of line buffers;

scaling and correcting for geometric distortion in a horizontal direction using the lines of the luma component image data by;

using horizontal luma coordinate generation logic to determine horizontally correcting luma pixel coordinates that, when resampled, would produce horizontally and vertically geometrically corrected luma pixels; and

using a horizontal luma resampling filter to resample pixels across a plurality of lines in the first plurality of line buffers at the horizontally correcting luma pixel coordinates to obtain the horizontally and vertically geometrically corrected luma component pixels;

storing lines of chroma component image data of the uncorrected image frame in a second plurality of line buffers;

scaling and correcting for geometric distortion in the vertical direction using the lines of the chroma component image data by;

using vertical chroma coordinate generation logic to determine vertically correcting chroma pixel coordinates that, when resampled, would produce vertically geometrically corrected chroma pixels;

using a vertical chroma resampling filter to resample pixels across a plurality of lines in the second plurality of line buffers at the vertically correcting chroma pixel coordinates to obtain the vertically geometrically corrected chroma pixels; and

storing the vertically geometrically corrected pixels in the second plurality of line buffers; and

scaling and correcting for geometric distortion in the horizontal direction using the lines of the chroma component image data by;

using horizontal chroma coordinate generation logic to determine horizontally correcting chroma pixel coordinates that, when resampled, would produce horizontally and vertically geometrically corrected chroma pixels; and

using a horizontal chroma resampling filter to resample pixels across a plurality of lines in the second plurality of line buffers at the horizontally correcting chroma pixel coordinates to obtain the horizontally and vertically geometrically corrected chroma pixels.

6. The electronic device of claim 5, wherein the geometric distortion correction and scaling logic is configured to operate on image data that has been substantially corrected of chromatic aberration.

7. The electronic device of claim 5, wherein the geometric distortion correction and scaling logic is configured to resample the image data to obtain corrected output image data of the second resolution and corrected output image data of a third resolution.

8. The electronic device of claim 5, wherein the vertical luma coordinate generation logic is configured to determine fractional pixel coordinates.

9. The electronic device of claim 8, wherein the vertical luma resampling filter is configured to resample the fractional pixel coordinates using coefficients that vary depending on the fractional value of the fractional pixel coordinates.

10. The electronic device of claim 8, wherein the fractional pixel coordinates comprise a fractional component that is a multiple of ½

.

11. The electronic device of claim 10, wherein the fractional component is of a granularity of 1/16 or smaller.

12. A method comprising:

receiving a plurality of pixels of a frame of image data into logic of an image signal processor, wherein the frame of image data has geometric distortion due to a lens through which the image data was obtained;

computing a radius of a spatial location of an output pixel from an optical center of the image data aligned with the center of the lens using the logic of the image signal processor;

determining a displacement of the spatial location of the output pixel due to geometric distortion based at least in part on the radius, using the logic of the image signal processor;

determining correcting pixel coordinates in the uncorrected image frame using the logic of the image signal processor, wherein the correcting pixel coordinates represent a spatial location within the uncorrected image frame that contains image information associated with a corrected version of the output pixel; and

resampling the uncorrected frame using a multi-tap filter that includes a pixel closest to the correcting pixel coordinates and other pixels nearby the pixel closest to the correcting pixel coordinates to obtain the output pixel using the logic of the image signal processor, wherein the output pixel is at least partially corrected of geometric distortion;

wherein determining the correcting pixel coordinates and resampling the uncorrected frame comprises;

storing lines of luma component image data of the uncorrected image frame in a first plurality of line buffers;

scaling and correcting for geometric distortion in a vertical direction using the lines of the luma component image data by;

using vertical luma coordinate generation logic to determine vertically correcting luma pixel coordinates that when resampled, would produce geometrically corrected luma pixels;

using a vertical luma resampling filter to resample pixels across a plurality of lines in the first plurality of line buffers at the vertically correcting luma pixel coordinates to obtain the vertically geometrically corrected luma pixels; and

storing the vertically geometrically corrected luma pixels in the first plurality of line buffers;

scaling and correcting for geometric distortion in a horizontal direction using the lines of the luma component image data by;

using horizontal luma coordinate generation logic to determine horizontally correcting luma pixel coordinates that, when resampled, would produce horizontally and vertically geometrically corrected luma pixels; and

using a horizontal luma resampling filter to resample pixels across a plurality of lines in the first plurality of line buffers at the horizontally correcting luma pixel coordinates to obtain the horizontally and vertically geometrically corrected luma component pixels;

storing lines of chroma component image data of the uncorrected image frame in a second plurality of line buffers;

scaling and correcting for geometric distortion in the vertical direction using the lines of the chroma component image data by;

using vertical chroma coordinate generation logic to determine vertically correcting chroma pixel coordinates that, when resampled, would produce vertically geometrically corrected chroma pixels;

using a vertical chroma resampling filter to resample pixels across a plurality of lines in the second plurality of line buffers at the vertically correcting chroma pixel coordinates to obtain the vertically geometrically corrected chroma pixels; and

storing the vertically geometrically corrected pixels in the second plurality of line buffers; and

scaling and correcting for geometric distortion in the horizontal direction using the lines of the chroma component image data by;

using horizontal chroma coordinate generation logic to determine horizontally correcting chroma pixel coordinates that, when resampled, would produce horizontally and vertically geometrically corrected chroma pixels; and

using a horizontal chroma resampling filter to resample pixels across a plurality of lines in the second plurality of line buffers at the horizontally correcting chroma pixel coordinates to obtain the horizontally and vertically geometrically corrected chroma pixels.

13. The method of claim 12, wherein the displacement is determined using a lookup table relating the radius to the displacement.

14. An image signal processing system comprising:

YCC scaling logic configured to scale image data and at least partially correct for geometric distortion in the image data comprising;

a first plurality of line buffers configured to store lines of luma component image data;

a vertical luma scaler configured to scale and correct for geometric distortion in a vertical direction using the lines of luma component image data, wherein the vertical luma scaler comprises;

vertical luma coordinate generation logic configured to determine vertically correcting pixel coordinates that, when resampled, would produce vertically geometrically corrected pixels; and

a vertical luma resampling filter configured to resample pixels across a plurality of lines in the line buffers at the vertically correcting pixel coordinates to obtain the vertically geometrically corrected pixels, wherein the vertically geometrically corrected pixels are stored in the line buffers;

a horizontal luma scaler configured to scale and correct for geometric distortion in a horizontal direction using the lines of luma component image data, wherein the vertical luma scaler comprises;

horizontal luma coordinate generation logic configured to determine horizontally correcting pixel coordinates that, when resampled, would produce horizontally and vertically geometrically corrected pixels; and

a horizontal luma resampling filter configured to resample a plurality of horizontal pixels of a line in the line buffers at the horizontally correcting pixel coordinates to obtain the horizontally and vertically geometrically corrected pixels;

a second plurality of line buffers configured to store lines of chroma component image data;

a vertical chroma scaler configured to scale and correct for geometric distortion in the vertical direction using the lines of chroma component image data, wherein the vertical chroma scaler comprises;

vertical chroma coordinate generation logic configured to determine vertically correcting pixel coordinates that, when resampled, would produce vertically geometrically corrected pixels; and

a vertical chroma resampling filter configured to resample pixels across a plurality of lines in the line buffers at the vertically correcting pixel coordinates to obtain the vertically geometrically corrected pixels, wherein the vertically geometrically corrected pixels are stored in the line buffers; and

a horizontal chroma scaler configured to scale and correct for geometric distortion in the horizontal direction using the lines of chroma component image data, wherein the horizontal chroma scaler comprises;

horizontal chroma coordinate generation logic configured to determine horizontally correcting pixel coordinates that, when resampled, would produce horizontally and vertically geometrically corrected pixels; and

a horizontal chroma resampling filter configured to resample a plurality of horizontal pixels of a line in the line buffers at the horizontally correcting pixel coordinates to obtain the horizontally and vertically geometrically corrected pixels.

15. The image signal processing system of claim 14, wherein the line buffers are configurable to hold a plurality of different numbers of lines or a plurality of different line widths, or both.

16. The image signal processing system of claim 14, wherein the line buffers are configurable to hold 48 lines, 24 lines, or 12 lines.

17. The image signal processing system of claim 14, wherein the line buffers are configurable to hold lines of 4096 pixels, 2048 pixels, or 1024 pixels.

18. The image signal processing system of claim 14, comprising barrel shifters to select a subset of the lines held in the line buffers to be provided to the vertical luma scaler.

19. The image signal processing system of claim 18, wherein the barrel shifters are configured to select at least as many of the lines held in the line buffers as a maximum span of lines employed by the luma vertical resampling filter.

20. The image signal processing system of claim 18, wherein the barrel shifters are configured to select at least 20 lines to be provided to the vertical luma scaler.

21. The image signal processing system of claim 14, comprising a shared line buffer controller to control the first plurality of line buffers and the second plurality of line buffers.