Image processing apparatus and image processing program

• US 7,454,057 B2
• Filed: 09/15/2005
• Issued: 11/18/2008
• Est. Priority Date: 03/28/2003
• Status: Expired due to Fees

• Current Assignee: Olympus Corporation
• Original Assignee: Olympus Corporation
• Inventors: Tsukioka, Taketo
• Primary Examiner(s): Alavi; Amir
• Application Number:
  US11/226,944
• Publication Number:
  US 20060012841A1
• Time in Patent Office: 1,160 Days
• Field of Search: 382/162, 382/164, 382/167, 382/218, 382/219, 382/254, 382260-264, 382/272, 345/589, 345600-604, 358/515, 358/518
• US Class Current: 382/167
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

An image processing apparatus inputs an image obtained by sensing an optical image by an image sensing system, compensates an omitted color component, and outputs a color image. A weight setting unit sets a weight Wk in each direction Sk (k is an integer from 1 to n) of a plurality of directions S1, S2, . . . , Sn (n is an integer of not less than 1) starting from a pixel of interest in a predetermined neighborhood of the pixel of interest. An average value calculation unit calculates a weighted average of pixel values Vk of pixels having a specific color component and located on each direction Sk (k is an integer from 1 to n) in the predetermined neighborhood by using weights W1 to Wn of the pixel values V1 to Vn. An restoration unit causes the average calculation unit to calculate weighted averages of a plurality of types of color components and restores a value of an omitted color component of the pixel of interest on the basis of the weighted average result and the pixel value of the pixel of interest.

Claims

1. An image processing apparatus which inputs an image obtained by sensing an optical image by an imaging system, compensates for an omitted color component, and outputs a color image, the apparatus comprising:

a weight setting unit which sets a weight Wk in each direction Sk (k is an integer from 1 to n) of a plurality of directions S1, S 2, . . . , Sn (n is an integer of not less than

     1) starting from a pixel of interest in a predetermined neighborhood of the pixel of interest;

an average value calculation unit which calculates a weighted average of pixel values Vk of pixels having a specific color component and located on each direction Sk (k is an integer from 1 to n) in the predetermined neighborhood by using the weights W1 to Wn of the pixel values V1 to Vn; and

a restoration unit which causes the average calculation unit to calculate weighted averages of a plurality of types of color components, and which restores a value of the omitted color component of the pixel of interest on the basis of a weighted average result and a pixel value of the pixel of interest.

2. An image processing apparatus according to claim 1, wherein the weight setting unit selects for a predetermined color component at least one set having two pixels parallel to said each direction Sk in the predetermined neighborhood and having the predetermined color component and calculates the weight Wk for said each direction Sk on the basis of a pixel value difference of each set.

3. An image processing apparatus according to claim 2, wherein the weight setting unit calculates the pixel value differences of said each set for a plurality of types of color components and determines the weight Wk for said each direction Sk on the basis of a sum of products of the pixel value differences and a predetermined coefficient.

4. An image processing apparatus according to claim 3, wherein the imaging system comprises a 1-chip imaging element which has a mosaic filter of a primary color Bayer array, a color component R or B is obtained in the pixel of interest, four directions, upper, lower, right, and left directions of the pixel of interest, are defined as said plurality of directions S1 to S4, said plurality of types of color components are G and a color component obtained in the pixel of interest, and the omitted color component is G.

5. An image processing apparatus according to claim 3, wherein the imaging system comprises a 1-chip imaging element which has a mosaic filter of a primary color Bayer array, a color component G is obtained in the pixel of interest, said plurality of directions are defined as directions connecting pixels having a color component R in the predetermined neighborhood centered on the pixel of interest, said plurality of types of color components are R and G, and the omitted color component is R.

6. An image processing apparatus according to claim 3, wherein the imaging system comprises a 1-chip imaging element which has a mosaic filter of a primary color Bayer array, a color component G is obtained in the pixel of interest, said plurality of directions are defined as directions connecting pixels having a color component B in the predetermined neighborhood centered on the pixel of interest, said plurality of types of color components are B and G, and the omitted color component is B.

7. An image processing apparatus according to claim 1, wherein the weight setting unit comprises an estimation unit which estimates an approximate value of a predetermined color component of the pixel of interest in the predetermined neighborhood, and calculates the weight Wk for said each direction Sk on the basis of the pixel value of the pixel located on said each direction Sk and having the predetermined color component and a difference between the approximate values.

8. An image processing apparatus according to claim 1, wherein the weight setting unit selects a plurality of sets each having the two pixels parallel to said each direction Sk in the predetermined neighborhood and having the predetermined color component and calculates a minimum value of the pixel value differences of each set.

9. An image processing apparatus according to any one of claims 2, 3, and 8, wherein the weight setting unit obtains a maximum value of the pixel value differences after the pixel value difference for said each direction Sk is calculated for all integers of 1 to n and calculates weights so as to equalize weights in the respective directions when the magnitudes of the maximum values are smaller.

10. An image processing apparatus according to any one of claims 3 to 8, wherein the weight setting unit calculates the weight Wk for said each direction so that each weight is in inverse proportion to the pixel value difference for said each direction Sk and the sum of the weights becomes 1.

11. An image processing apparatus according to any one of claims 2, 3, and 8, wherein the weight setting unit has an evaluation unit which evaluates an edge color misregistration amount between color components in the pixel of interest and changes a weight calculation method in accordance with an evaluation result of the color misregistration evaluation unit.

12. An image processing apparatus according to claim 11, wherein the weight setting unit changes the weight calculation method by changing the predetermined coefficient in accordance with an evaluation result in the color misregistration evaluation unit.

13. An image processing apparatus according to claim 12, wherein when the predetermined coefficient is to be changed in accordance with the evaluation result of the color misregistration evaluation unit, the color misregistration evaluation unit identifies the color components whose misregistration amount are large, and the weight setting unit makes the coefficient assigned to the pixel value difference associated with the color components smaller.

14. An image processing apparatus according to claim 1, wherein the restoration unit multiplies predetermined coefficients by the weighted average results for the plurality of color components and the values of the pixel of interest, and adds the products to obtain a value of the omitted color component.

15. An image processing apparatus according to claim 14, wherein the restoration unit comprises a color misregistration evaluation unit which evaluates a color misregistration amount of an edge between color components in the pixel of interest and changes a calculation for the omitted color component in accordance with an evaluation result in the color misregistration evaluation unit.

16. An image processing apparatus according to claim 14, wherein the restoration unit comprises a color misregistration evaluation unit which evaluates a color misregistration amount of an edge between color components in the pixel of interest and changes the predetermined coefficient in accordance with the evaluation result of the color misregistration evaluation unit.

17. An image processing apparatus according to any one of claims 15 and 16, wherein the color misregistration evaluation unit evaluates an edge color misregistration amount between color components on the basis of chromatic aberration of the optical image.

18. An image processing apparatus according to claim 16, wherein when the predetermined coefficient is to be changed in accordance with the evaluation result of the color misregistration evaluation unit, the color misregistration evaluation unit identifies the color components whose misregistration amount are large, and the restoration unit makes the coefficient assigned to the weighted average result associated with the color components smaller.

19. An image processing apparatus according to claim 1, wherein said plurality of directions are set as directions connecting the pixel of interest and pixels each having both the omitted color component at the pixel of interest and another color component obtained at the pixel of interest.

20. An image processing apparatus according to claim 19, wherein the imaging system comprises a 1-chip imaging element which has a mosaic filter of a primary color Bayer array, the omitted color component in the pixel of interest is R or B, and said another color component obtained in the pixel of interest is G.

21. A computer readable medium storing a computer program which causes a computer to input an image obtained by sensing an optical image by an image sensing system, compensate for an omitted color component, and output a color image, the program being executable by the computer to cause the computer to perform functions comprising:

a weight setting function which sets a weight Wk in each direction Sk (k is an integer from 1 to n) of a plurality of directions S1, S 2, . . . , Sn (n is an integer of not less than

     1) starting from a pixel of interest in a predetermined neighborhood of the pixel of interest;

an averaging function which calculates a weighted average of pixel values Vk of pixels having a specific color component and located on each direction Sk (k is an integer from 1 to n) in the predetermined neighborhood by using the weights W1 to Wn of the pixel values V1 to Vn; and

a restoration unit function which causes the averaging function to calculate weighted averages of a plurality of types of color components, and which restores a value of the omitted color component of the pixel of interest on the basis of a weighted average result and a pixel value of the pixel of interest.

22. A computer readable medium according to claim 21, wherein the weight setting function selects, for each color component, at least one set having two pixels parallel to a direction Sk in the predetermined neighborhood and having the color component, and a weight Wk in said each direction Sk is calculated on the basis of a value adjusted by multiplying a predetermined coefficient to a pixel value difference of said each set.

23. A computer readable medium according to claim 21, wherein the restoration function includes a color misregistration evaluation function of evaluating an edge color misregistration amount between color components in the pixel of interest, and a calculation method for the omitted color component is changed in accordance with an evaluation result of the color misregistration evaluation function.

24. A computer readable medium according to claim 21, wherein the restoration function includes a color misregistration evaluation function of evaluating an edge color misregistration amount between color components in the pixel of interest, a coefficient which changes in accordance with the evaluation result of said color misregistration evaluation function is multiplied to the pixel value of the pixel of interest and the weighted average result of the averaging function for said plurality of color components, and the value of the omitted color component is obtained.

25. A computer readable medium according to any one of claims 23 and 24, wherein the color misregistration evaluation function evaluates an edge color misregistration amount between the color components on the basis of chromatic aberration of the optical image.