Image processing apparatus

• US 20050058361A1
• Filed: 09/10/2004
• Published: 03/17/2005
• Est. Priority Date: 09/12/2003
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Tajima, Kaori
• Granted Patent: US 7,477,803 B2
• Time in Patent Office: Days
• US Class Current: 382/254
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

An image processing system has an input unit for inputting an image signal obtained with offset sampling per pixel, a correlation detecting unit for, based on the input image signal, producing a coefficient depending on an arbitrary direction in which correlation between a plurality of pixels near a target pixel to be interpolated is high, and an interpolation circuit for producing an interpolation image signal in accordance with the coefficient produced by the correlation detecting unit, and interpolating the image signal input from the input unit by using the interpolation image signal.

Claims

1. An image processing apparatus comprising:

input means for inputting an image signal obtained with offset sampling per pixel;

correlation detecting means for, based on the image signal input from said input means, producing a coefficient depending on an arbitrary direction in which correlation between a plurality of pixels near a target pixel to be interpolated is high; and

interpolation means for producing an interpolation image signal in accordance with the coefficient produced by said correlation detecting means, wherein the interpolation means uses the interpolation image signal to interpolate the image signal input from said input means.

2. An image processing apparatus according to claim 1, wherein said correlation detecting means detects a correlation degree in the vertical direction and a correlation degree in the horizontal direction, and produces the coefficient based on the detected correlation degree in the vertical and horizontal direction.

3. An image processing apparatus according to claim 2, wherein said correlation detecting means computes the coefficient based on a ratio of vertical correlation data representing the correlation degree in the vertical direction and horizontal correlation data representing the correlation degree in the horizontal direction.

4. An image processing apparatus according to claim 2, wherein said correlation detecting means compares a value of the vertical correlation data representing the correlation degree in the vertical direction with a value of the horizontal correlation data representing the correlation degree in the horizontal direction, and

wherein the correlation detecting means produces the coefficient such that when the quotient resulting from dividing the smaller correlation data by the larger correlation data is larger than a predetermined threshold, said correlation detecting means produces the coefficient based on the quotient, and when the quotient is not larger than the threshold, said correlation detecting means produces the coefficient having a predetermined value corresponding to the horizontal direction or the vertical direction.

5. An image processing apparatus according to claim 1, wherein said interpolation means comprises

horizontal interpolation data producing means for filtering pixels in the horizontal direction and producing first interpolation data,

vertical interpolation data producing means for filtering pixels in the vertical direction and producing second interpolation data, and

two-dimensional interpolation data producing means for filtering pixels in the vertical and horizontal directions and producing third interpolation data, and said interpolation means produces the interpolation image signal by synthesizing the third interpolation data and the first or second interpolation data in accordance with the coefficient.

6. An image processing apparatus according to claim 5, wherein said two-dimensional interpolation data producing means includes a filter having a frequency characteristic allowing passage of signals in a region within a quadrilateral defined, on a frequency plane having vertical and horizontal coordinate axes, by connecting a position of Nyquist frequency in the vertical direction and a position of Nyquist frequency in the horizontal direction, but cutting off signals in a region outside said quadrilateral.

7. An image processing apparatus according to claim 1, wherein said interpolation means comprises

horizontal interpolation data producing means for filtering pixels in the horizontal direction and producing first interpolation data,

vertical interpolation data producing means for filtering pixels in the vertical direction and producing second interpolation data,

third interpolation data producing means for executing a filtering process with a cutoff characteristic in a 45°

-direction relative to the horizontal direction and producing third interpolation data, and

fourth interpolation data producing means for executing a filtering process with a cutoff characteristic in a 135°

-direction relative to the horizontal directions and producing fourth interpolation data, and said interpolation means produces the interpolation image signal by synthesizing any one of the third interpolation data and the fourth interpolation data and the first or second interpolation data.

8. An image processing apparatus according to claim 7, wherein said correlation detecting means detects a correlation degree in the 45°

-direction and a correlation degree in the 135°

-direction, and produces a control signal for selecting any one of the third interpolation data and the fourth interpolation data in accordance with the correlation degree in the 45°

-direction and the correlation degree in the 135°

-direction.

9. An image processing apparatus according to claim 1, wherein the image signal has one of color components of red, green and blue, and the image signal of the green component is subjected to offset sampling per pixel.

10. An image processing apparatus according to claim 9, wherein said correlation detecting means produces the coefficient by using the image signal having one of the color components of red, green and blue, and said interpolation means produces the interpolation image signal by using the image signal of the green component.

11. An image processing method comprising the steps of:

an input step of inputting an image signal obtained with offset sampling per pixel;

a correlation detecting step of using the input image signal to produce a coefficient in an arbitrary direction that has a high correlation between a plurality of pixels near a target pixel; and

an interpolation signal producing step of producing an interpolation image signal in accordance with the coefficient produced; and

an interpolating step of interpolating the input image signal input by using the interpolation image signal.

12. An image processing method according to claim 11, wherein said correlation detecting step detects a correlation degree in the vertical direction and a correlation degree in the horizontal direction, and produces the coefficient based on the correlation degree in the vertical direction and the correlation degree in the horizontal direction.

13. An image processing method according to claim 12, wherein said correlation detecting step computes the coefficient based on a ratio of vertical correlation data representing the correlation degree in the vertical direction and horizontal correlation data representing the correlation degree in the horizontal direction.

14. An image processing method according to claim 12, wherein said correlation detecting step

compares a value of the vertical correlation data representing the correlation degree in the vertical direction with a value of the horizontal correlation data representing the correlation degree in the horizontal direction, and

produces the coefficient such that when the quotient resulting from dividing the smaller correlation data by the larger correlation data is larger than a predetermined threshold, said correlation detecting step produces the coefficient based on the quotient, and

when the quotient is not larger than the threshold, said correlation detecting step produces the coefficient having a predetermined value corresponding to the horizontal direction or the vertical direction.

15. An image processing method according to claim 11, wherein said interpolation signal producing step comprises

a horizontal interpolation data producing step of filtering pixels in the horizontal direction and producing first interpolation data,

a vertical interpolation data producing step of filtering pixels in the vertical direction and producing second interpolation data, and

a two-dimensional interpolation data producing step of filtering pixels in the vertical and horizontal directions and producing third interpolation data, and said interpolation signal producing step produces the interpolation image signal by synthesizing the third interpolation data and the first or second interpolation data in accordance with the coefficient.

16. An image processing method according to claim 15, wherein said two-dimensional interpolation data producing step executes a filtering process with a frequency characteristic allowing passage of signals within a quadrilateral region, on a frequency plane having vertical and horizontal coordinate axes, by connecting a position of Nyquist frequency in the vertical direction and a position of Nyquist frequency in the horizontal direction, but cutting off signals outside said quadrilateral region.

17. An image processing method according to claim 11, wherein said interpolation signal producing step comprises

a horizontal interpolation data producing step of filtering pixels in the horizontal direction and producing first interpolation data,

a vertical interpolation data producing step of filtering pixels in the vertical direction and producing second interpolation data,

a third interpolation data producing step of executing a filtering process with a cutoff characteristic in a 45°

-direction relative to the horizontal directions and producing third interpolation data, and

a fourth interpolation data producing step of executing a filtering process with a cutoff characteristic in a 135°

-direction relative to the horizontal directions and producing fourth interpolation data, and said interpolation signal producing step produces the interpolation image signal by synthesizing any one of the third interpolation data and the fourth interpolation data and the first or second interpolation data.

18. An image processing method according to claim 17, wherein said correlation detecting step detects a correlation degree in the 45°

-direction and a correlation degree in the 135°

-direction, and produces a control signal for selecting any one of the third interpolation data and the fourth interpolation data in accordance with the correlation degree in the 45°

-direction and the correlation degree in the 135°

-direction.

19. An image processing method according to claim 11, wherein the image signal has one of color components of red, green and blue, and the image signal of the green component is subjected to offset sampling per pixel.

20. An image processing method according to claim 19, wherein said correlation detecting step produces the coefficient by using the image signal having one of the color components of red, green and blue, and said interpolation signal producing step produces the interpolation image signal by using the image signal of the green component.

21. An image processing apparatus comprising:

input means for receiving an image signal obtained with offset sampling per pixel;

correlation detecting means for, based on the input image signal, detecting degrees of correlation in horizontal and vertical directions between a plurality of pixels near a target pixel, and obtaining correlation values representing the detected correlation degrees;

coefficient producing means for producing a coefficient based on the correlation value in the horizontal direction and the correlation value in the vertical direction obtained by said correlation detecting means;

said coefficient producing means setting a value of the coefficient to a particular value when the target pixel contains a frequency component near Nyquist frequency in the horizontal or vertical direction, and determining the coefficient by using the correlation value in the horizontal direction and the correlation value in the vertical direction when the target pixel does not contain the frequency component near the Nyquist frequency; and

interpolation means for producing an interpolation image signal in accordance with the coefficient produced by said coefficient producing means, and interpolating the image signal input from said input means by using the interpolation image signal.

22. An image processing apparatus according to claim 21, wherein said coefficient producing means determines based on larger one of the correlation value in the horizontal direction and the correlation value in the vertical direction whether the target pixel contains the frequency component near the Nyquist frequency.

23. An image processing apparatus according to claim 21, wherein when the target pixel does not contain the Nyquist frequency component, said coefficient producing means produces, based on the correlation value in the horizontal direction and the correlation value in the vertical direction, the coefficient depending on an arbitrary direction in which correlation between a plurality of pixels near the target pixel is high.

24. An image processing apparatus according to claim 21, wherein said interpolation means comprises

horizontal interpolation data producing means for filtering pixels in the horizontal direction and producing first interpolation data,

vertical interpolation data producing means for filtering pixels in the vertical direction and producing second interpolation data, and

two-dimensional interpolation data producing means for filtering pixels in the vertical and horizontal directions and producing third interpolation data, and said interpolation means produces the interpolation image signal by synthesizing the third interpolation data and the first or second interpolation data in accordance with the coefficient.

25. An image processing apparatus according to claim 24, wherein said two-dimensional interpolation data producing means includes a filter having a frequency characteristic allowing passage of signals within a quadrilateral region, on a frequency plane having vertical and horizontal coordinate axes, by connecting a position of Nyquist frequency in the vertical direction and a position of Nyquist frequency in the horizontal direction, but cutting off signals in a region outside said quadrilateral region.

26. An image processing apparatus according to claim 21, wherein the image signal has one of color components of red, green and blue, and the image signal of the green component is subjected to offset sampling per pixel.

27. An image processing apparatus according to claim 26, wherein said input means includes an image pickup device of a primary-color Bayer array, which has three kinds of color filters of red, green and blue, the green filters being arranged in a checkered pattern.

28. An image processing apparatus comprising:

input means for inputting an image signal obtained with offset sampling per pixel;

correlation detecting means for, based on the input image signal, detecting degrees of correlation in horizontal and vertical directions for a target pixel to be interpolated, and for obtaining correlation values representing the detected correlation degrees;

coefficient producing means for producing a coefficient based on the correlation value in the horizontal direction and the correlation value in the vertical direction obtained by said correlation detecting means;

said coefficient producing means setting a value of the coefficient to a particular value when each of the correlation value in the horizontal direction and the correlation value in the vertical direction is smaller than a first threshold and is not present between a second threshold and a third threshold, and producing the coefficient through computation based on a differential between the correlation value in the horizontal direction and the correlation value in the vertical direction when each of the correlation value in the horizontal direction and the correlation value in the vertical direction is smaller than the first threshold and is present between the second threshold and the third threshold; and

interpolation means for producing an interpolation image signal in accordance with the coefficient produced by said coefficient producing means, and interpolating the image signal input from said input means by using the interpolation image signal.

29. An image processing apparatus according to claim 28, wherein said coefficient producing means further produces the coefficient through computation based on a ratio of the correlation value in the horizontal direction to the correlation value in the vertical direction when at least one of the correlation value in the horizontal direction and the correlation value in the vertical direction is larger than the first threshold and is present between a fourth threshold and a fixed threshold.

30. An image processing method comprising the steps of:

an input step of inputting an image signal obtained with offset sampling per pixel;

a correlation detecting step of, based on the input image signal, detecting degrees of correlation in horizontal and vertical directions between a plurality of pixels near a target pixel to be interpolated, and obtaining correlation values representing the detected correlation degrees;

a coefficient producing step of producing a coefficient based on the correlation value in the horizontal direction and the correlation value in the vertical direction obtained in said correlation detecting step;

said coefficient producing step setting a value of the coefficient to a particular value when the target pixel to be interpolated contains a frequency component near Nyquist frequency in the horizontal or vertical direction, and computing the coefficient through computation using the correlation value in the horizontal direction and the correlation value in the vertical direction when the target pixel does not contain the frequency component near the Nyquist frequency; and

an interpolation step of producing an interpolation image signal in accordance with the coefficient produced in said coefficient producing step, and interpolating the input image signal by using the interpolation image signal.

31. An image processing method comprising:

inputting an image signal obtained with offset sampling per pixel;

detecting, based on the input image signal, degrees of correlation in horizontal and vertical directions for a target pixel to be interpolated, and obtaining correlation values representing the detected correlation degrees;

producing a coefficient based on the detected correlation value in the horizontal direction and the correlation value in the vertical direction;

wherein said producing a coefficient further comprises setting a value of the coefficient to a particular value when each of the correlation value in the horizontal direction and the correlation value in the vertical direction is smaller than a first threshold and is not present between a second threshold and a third threshold, and producing the coefficient based on a differential between the correlation value in the horizontal direction and the correlation value in the vertical direction when each of the correlation value in the horizontal direction and the correlation value in the vertical direction is smaller than the first threshold and is present between the second threshold and the third threshold; and

producing an interpolation image signal in accordance with the coefficient produced in said coefficient producing step, and interpolating, using the interpolation image signal, the input image signal.