Method for creating high resolution color image, system for creating high resolution color image and program creating high resolution color image

• US 7,515,747 B2
• Filed: 10/08/2003
• Issued: 04/07/2009
• Est. Priority Date: 01/31/2003
• Status: Expired due to Fees

• Current Assignee: Tokyo Institute of Technology (National University Corporation Tokyo Institute of Technology)
• Original Assignee: The Circle for the Promotion of Science and Engineering
• Inventors: Goto, Tomomasa, Okutomi, Masatoshi
• Primary Examiner(s): Couso; Jose L
• Application Number:
  US10/543,846
• Publication Number:
  US 20060038891A1
• Time in Patent Office: 2,008 Days
• Field of Search: 382/176, 382/299, 382/312, 348/266, 348/268, 348/272, 348/273
• US Class Current: 382/167
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

A limitation in the physical resolution of an image sensor offers a motivation to improve the resolution of an image. Super-resolution is mainly applied to gray scale images, and it has not been thoroughly investigated yet that a high resolution color image is reconstructed from an image sensor having a color filter array. An object of the invention is to directly reconstruct a high resolution color image from color mosaic obtained by an image sensor having a color filter array. A high resolution color image reconstruction method according to the invention is based on novel technique principles of color image reconstruction that an increase in resolution and demosaicing are performed at the same time. The verification and effective implement of the invention are also described.

Claims

1. A high resolution color image reconstruction method which reconstructs a high resolution color image based on a single input image or a sequence of multiple input images captured by an image capturing device employing an image sensor and a color filter array,

said high resolution color image reconstruction method characterized by

reconstructing said high resolution color image having a predetermined resolution equal to or larger than a physical resolution of said image capturing device;

and

comprising the step of;

conducting an image reconstruction process based on prior knowledge that relates to a model of an image capturing system or a subject including known or unknown displacement or deformation generated between said sequence of multiple input images, and then reconstructing said high resolution color image in which even a high spatial frequency component contained in said subject is truly reproduced and occurrence of false color is suppressed.

2. A high resolution color image reconstruction method according to claim 1, wherein in said image reconstruction process, said input image which is obtained from said image capturing device and is a raw data not color interpolated is directly processed.

3. A high resolution color image reconstruction method according to claim 1, wherein in said image reconstruction process, an optimization process of an evaluation function relating to said model of said image capturing system or a color component is conducted.

8. A high resolution color image reconstruction method according to claim 1, further comprising the step of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

9. A high resolution color image reconstruction apparatus characterized in that it is a single-chip image capturing device employing a single-chip capturing element and using said high resolution color image reconstruction method according to claim 1.

10. A high resolution color image reconstruction apparatus according to claim 9, wherein said single-chip capturing element is a CCD.

19. A high resolution color image reconstruction method according to claim 2, further comprising the step of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

20. A high resolution color image reconstruction method according to claim 3, further comprising the step of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

25. A high resolution color image reconstruction apparatus characterized in that it is a single-chip image capturing device employing a single-chip capturing element and using said high resolution color image reconstruction method according to claim 2.

26. A high resolution color image reconstruction apparatus characterized in that it is a single-chip image capturing device employing a single-chip capturing element and using said high resolution color image reconstruction method according to claim 3.

31. A high resolution color image reconstruction apparatus characterized in that it is a single-chip image capturing device employing a single-chip capturing element and using said high resolution color image reconstruction method according to claim 8.

4. A high resolution color image reconstruction method which reconstructs a high resolution color image based on a single input image or a sequence of multiple input images captured by an image capturing device employing an image sensor and a color filter array,

said high resolution color image reconstruction method characterized by

reconstructing said high resolution color image having a predetermined resolution equal to or larger than a physical resolution of said image capturing device;

and

comprising the step of;

conducting an image reconstruction process based on prior knowledge that relates to a model of an image capturing system or a subject including known or unknown displacement or deformation generated between said sequence of multiple input images, and then reconstructing said high resolution color image in which even a high spatial frequency component contained in said subject is truly reproduced and occurrence of false color is suppressed,

wherein in said image reconstruction process, an optimization process of an evaluation function relating to said model of said image capturing system or a color component is conducted,

wherein said evaluation function comprises;

a first term which evaluates fidelity of said high resolution color image being an estimated image with respect to said input images by using an image formation model expressing generation of said high resolution color image from said input images;

a second term which evaluates smoothness in said estimated image by penalizing said estimated image within which neighboring pixel values change abruptly in said input images; and

a third term which evaluates smoothness of chrominance of said estimated image by penalizing said estimated image within which chrominance components change abruptly in said input images.

5. A high resolution color image reconstruction method according to claim 4, wherein said second term serves to obtain a smooth estimated image with preserved edges;

and said third term serves to obtain a smooth estimated image with local chrominance continuity and preserved edges.

6. A high resolution color image reconstruction method according to claim 5, wherein:

said evaluation function is determined by following equation;

$\hat{z}=\arg <br><br>\underset{z}{\min }<br><br>\left\{f_1<br><br>\left(z\right)+{\mathrm{af}}_2<br><br>\left(z\right)+{\mathrm{bf}}_3<br><br>\left(z\right)\right\};<br><br>$

said first term is determined by following equation;

$f_1<br><br>\left(z\right)=\underset{k=1}{\overset{M}{\sum <br><br>}}<br><br><br><br>{<br><br>y_k-A_k<br><br>z<br><br>}^2;<br><br>$

said second term is determined by following equation;

f₂(z)=∥

Pz∥

²;

said third term is determined by following equation;

f₃(z)=∥

QTz∥

²,

where

y_k is a vector which contains pixel values in the k-th image in an input image sequence obtained from said image sensor and said color filter array,

M represents the number of images in said input image sequence,

z is a vector which contains pixel values in said high resolution color image to be estimated and is composed of vectors z_R, z_G, z_B which contain pixel values in red, green, and blue channel of z respectively,

{circumflex over (z)} is a vector which contains pixel values of said high resolution color image obtained by said high resolution color image reconstruction method,

A_k is a matrix which relates y_k to z and is determined by properties of said image capturing device,

said properties comprises motion between different said input images y₁, y₂, . . . , y_M, a point spread function of said image capturing device, down-sampling factor by said image sensor and said color filter array,

P and Q are matrices which serve as adaptive or non-adaptive high-pass filter,

T is a matrix which transforms RGB signals to chrominance signals, and

a, b are weight parameters which control influence of f₂(z) and f₃(z) respectively.

21. A high resolution color image reconstruction method according to claim 4, further comprising the step of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

22. A high resolution color image reconstruction method according to claim 5, further comprising the step of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

23. A high resolution color image reconstruction method according to claim 6, further comprising the step of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

27. A high resolution color image reconstruction apparatus characterized in that it is a single-chip image capturing device employing a single-chip capturing element and using said high resolution color image reconstruction method according to claim 4.

28. A high resolution color image reconstruction apparatus characterized in that it is a single-chip image capturing device employing a single-chip capturing element and using said high resolution color image reconstruction method according to claim 5.

29. A high resolution color image reconstruction apparatus characterized in that it is a single-chip image capturing device employing a single-chip capturing element and using said high resolution color image reconstruction method according to claim 6.

7. A high resolution color image reconstruction method which reconstructs a high resolution color image based on a single input image or a sequence of multiple input images captured by an image capturing device employing an image sensor and a color filter array,

said high resolution color image reconstruction method characterized by

reconstructing said high resolution color image having a predetermined resolution equal to or larger than a physical resolution of said image capturing device;

and

comprising the step of;

conducting an image reconstruction process based on prior knowledge that relates to a model of an image capturing system or a subject including known or unknown displacement or deformation generated between said sequence of multiple input images, and then reconstructing said high resolution color image in which even a high spatial frequency component contained in said subject is truly reproduced and occurrence of false color is suppressed,

wherein in said image reconstruction process, an optimization process of an evaluation function relating to said model of said image capturing system or a color component is conducted,

wherein said evaluation function is determined by following equations;

$\hat{z}=\arg <br><br><br><br>\underset{z}{\min }<br><br>\left\{f_1<br><br>\left(z\right)+f_p<br><br>\left(z\right)\right\},<br><br>f_1<br><br>\left(z\right)=\underset{k=1}{\overset{M}{\sum <br><br>}}<br><br><br><br>{<br><br>y_k-A_k<br><br>z<br><br>}^2,<br><br>f_p<br><br>\left(z\right)=f_2<br><br>\left(z_Y\right)+f_3<br><br>\left(z_{\mathrm{Cb}},z_{\mathrm{Cr}}\right),<br><br>f_2<br><br>\left(z_Y\right)=\underset{d\in <br><br>D}{\sum <br><br>}<br><br><br><br>{<br><br>{\mathrm{\Lambda <br><br>}}_d<br><br>P_d<br><br>z_Y<br><br>}^2,<br><br>f_3<br><br>\left(z_{\mathrm{Cb}},z_{\mathrm{Cr}}\right)={\mathrm{\lambda <br><br>}}_c<br><br>\left({<br><br>{\mathrm{Hz}}_{\mathrm{Cb}}<br><br>}^2+{<br><br>{\mathrm{Hz}}_{\mathrm{Cr}}<br><br>}^2\right)$

where;

y_k is a vector which contains pixel values in the k-th image in an input image sequence obtained from said image sensor and said color filter array,

M represents the number of images in said input image sequence,

z is a vector which contains pixel values in said high resolution color image to be estimated and is composed of vectors z_R, z_G, z_B which contain pixel values in red, green, and blue channel of z respectively,

{circumflex over (z)} is a vector which contains pixel values of said high resolution color image obtained by said high resolution color image reconstruction method,

A_k is a matrix which relates y_k to z and is determined by properties of said image capturing device,

said properties comprises motion between different said input images y₁, y₂, . . . , y_M, a point spread function of said image capturing device, down-sampling factor by said image sensor and said color filter array,

z_y is a luminance signal of z, f₂(z_y) is a regularization term for z_y,

z_Cb, z_Cr are chrominance signals of z, f₂(z_Cb, z_Cr) is a regularization term for z_Cb, z_Cr,

d is a edge direction in each pixel of z_y, i.e. represents any one direction of horizontal, vertical, and diagonal (two ways) directions, no direction (equal to all directions),

P_d is a high-pass operator that evaluates smoothness in said edge direction d,

Λ

_d is a diagonal matrix whose element is a weight with respect to a high-pass operation in said edge direction d,

H is an isotropic high-pass filter, and

λ

_c is a weighting parameter.

24. A high resolution color image reconstruction method according to claim 7, further comprising the step of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

30. A high resolution color image reconstruction apparatus characterized in that it is a single-chip image capturing device employing a single-chip capturing element and using said high resolution color image reconstruction method according to claim 7.

11. A high resolution color image reconstruction computer program embodied in a computer-readable medium which reconstructs a high resolution color image based on a single input image or a sequence of multiple input images captured by an image capturing device employing an image sensor and a color filter array,

said high resolution color image reconstruction computer program is executable with a computer, comprising the function of:

reconstructing said high resolution color image having a predetermined resolution equal to or larger than a physical resolution of said image capturing device;

and further comprising the function of;

conducting an image reconstruction process based on prior knowledge that relates to a model of an image capturing system or a subject including known or unknown displacement or deformation generated between said sequence of multiple input images, and then reconstructing said high resolution color image in which even a high spatial frequency component contained in said subject is truly reproduced and occurrence of false color is suppressed.

12. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 11, wherein in said image reconstruction process, said input image which is obtained from said image capturing device and is a raw data not color interpolated is directly processed.

13. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 11, wherein in said image reconstruction process, an optimization process of an evaluation function relating to said model of said image capturing system or a color component is conducted.

14. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 13, wherein said evaluation function comprises:

a first term which evaluates fidelity of said high resolution color image being an estimated image with respect to said input images by using an image formation model expressing generation of said high resolution color image from said input images;

a second term which evaluates smoothness in said estimated image by penalizing said estimated image within which neighboring pixel values change abruptly in said input images; and

a third term which evaluates smoothness of chrominance of said estimated image by penalizing said estimated image within which chrominance components change abruptly in said input images.

15. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 14, wherein said second term serves to obtain a smooth estimated image with preserved edges;

and

said third term serves to obtain a smooth estimated image with local chrominance continuity and preserved edges.

16. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 15, wherein:

said evaluation function is determined by following equation;

$\hat{z}=\arg <br><br>\underset{z}{\min }<br><br>\left\{f_1<br><br>\left(z\right)+{\mathrm{af}}_2<br><br>\left(z\right)+{\mathrm{bf}}_3<br><br>\left(z\right)\right\};<br><br>$

said first term is determined by following equation;

$f_1<br><br>\left(z\right)=\underset{k=1}{\overset{M}{\sum <br><br>}}<br><br><br><br>{<br><br>y_k-A_k<br><br>z<br><br>}^2;<br><br>$

said second term is determined by following equation;

f₂(z)=∥

Pz∥

²; and

said third term is determined by following equation;

f₃(z)=∥

QTz∥

²,

where

y_k is a vector which contains pixel values in the k-th image in an input image sequence obtained from said image sensor and said color filter array,

M represents the number of images in said input image sequence,

z is a vector which contains pixel values in said high resolution color image to be estimated and is composed of vectors z_R, z_G, z_B which contain pixel values in red, green, and blue channel of z respectively,

{circumflex over (z)} is a vector which contains pixel values of said high resolution color image obtained by said high resolution color image reconstruction method,

A_k is a matrix which relates y_k to z and is determined by properties of said image capturing device,

said properties comprises motion between different said input images y₁, y₂, . . . , y_M, a point spread function of said image capturing device, down-sampling factor by said image sensor and said color filter array,

P and Q are matrices which serve as adaptive or non-adaptive high-pass filter,

T is a matrix which transforms RGB signals to chrominance signals, and

a, b are weight parameters which control influence of f₂(z) and f₃(z) respectively.

17. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 13, wherein said evaluation function is determined by following equations:

$\hat{z}=\arg <br><br><br><br>\underset{z}{\min }<br><br>\left\{f_1<br><br>\left(z\right)+f_p<br><br>\left(z\right)\right\},<br><br>f_1<br><br>\left(z\right)=\underset{k=1}{\overset{M}{\sum <br><br>}}<br><br><br><br>{<br><br>y_k-A_k<br><br>z<br><br>}^2,<br><br>f_p<br><br>\left(z\right)=f_2<br><br>\left(z_Y\right)+f_3<br><br>\left(z_{\mathrm{Cb}},z_{\mathrm{Cr}}\right),<br><br>f_2<br><br>\left(z_Y\right)=\underset{d\in <br><br>D}{\sum <br><br>}<br><br><br><br>{<br><br>{\mathrm{\Lambda <br><br>}}_d<br><br>P_d<br><br>z_Y<br><br>}^2,<br><br>f_3<br><br>\left(z_{\mathrm{Cb}},z_{\mathrm{Cr}}\right)={\mathrm{\lambda <br><br>}}_c<br><br>\left({<br><br>{\mathrm{Hz}}_{\mathrm{Cb}}<br><br>}^2+{<br><br>{\mathrm{Hz}}_{\mathrm{Cr}}<br><br>}^2\right)$

where;

y_k is a vector which contains pixel values in the k-th image in an input image sequence obtained from said image sensor and said color filter array,

M represents the number of images in said input image sequence,

z is a vector which contains pixel values in said high resolution color image to be estimated and is composed of vectors z_R, z_G, z_B which contain pixel values in red, green, and blue channel of z respectively,

{circumflex over (z)} is a vector which contains pixel values of said high resolution color image obtained by said high resolution color image reconstruction method,

A_k is a matrix which relates y_k to z and is determined by properties of said image capturing device,

said properties comprises motion between different said input images y₁, y₂, . . . , y_M, a point spread function of said image capturing device, down-sampling factor by said image sensor and said color filter array,

z_y is a luminance signal of z, f₂(z_y) is a regularization term for z_y,

z_Cb,z_Cr are chrominance signals of z, f₂(z_Cb,z_Cr) is a regularization term for z_Cb,z_Cr,

d is a edge direction in each pixel of z_y, i.e. represents any one direction of horizontal, vertical, and diagonal (two ways) directions, no direction (equal to all directions),

P_d is a high-pass operator that evaluates smoothness in said edge direction d,

Λ

_d is a diagonal matrix whose element is a weight with respect to a high-pass operation in said edge direction d,

H is an isotropic high-pass filter, and

λ

_c is a weighting parameter.

18. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 11, further comprising the function of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

32. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 12, further comprising the function of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

33. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 13, further comprising the function of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

34. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 14, further comprising the function of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

35. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 15, further comprising the function of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

36. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 16, further comprising the function of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.

37. A high resolution color image reconstruction computer program embodied in a computer-readable medium according to claim 17, further comprising the function of:

estimating said displacement or deformation between said input images at the same time as said high resolution color image.