Processing of mosaic digital images

• US 20050169521A1
• Filed: 01/31/2004
• Published: 08/04/2005
• Est. Priority Date: 01/31/2004
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Hel-Or, Yacov
• Granted Patent: US 7,916,940 B2
• Time in Patent Office: Days
• US Class Current: 382/167
• CPC Class Codes: G06T 2200/12 involving antialiasing
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 1/58 Edge or detail enhancement; Noise or error suppression, e.g. colour misregistration correction H04N1/62 takes precedence
• CPC Class Codes: H04N 1/6005 with luminance or chrominance signals, e.g. LC1C2, HSL or YUV
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

A full color output image is estimated from a mosaic digital image, the output image is transformed from an original color space into a luminance/chrominance color space, chrominance components of the transformed output image are smoothed, the image is re-transformed back to its original color space, and corresponding pixels in the re-transformed image are re-set to sampled values.

Claims

1. A method of producing a full color digital image from a mosaic digital image having sampled values, the method comprising:

estimating a full color output image from the mosaic image;

transforming the output image from an original color space into a luminance/chrominance color space;

smoothing chrominance components of the transformed image;

re-transforming the image with the smoothed chrominance components back to the original color space; and

re-setting corresponding pixels in the re-transformed image to the sampled values.

2. The method of claim 1, wherein estimating the full color output image includes using a single algorithm to estimate missing information in all color planes of the output image.

3. The method of claim 1, wherein estimating the full color output image includes using a first algorithm to estimate missing information in a first color plane, and using a second algorithm to estimate missing information in remaining color planes of the output image;

and wherein the method further comprises setting corresponding pixels in the re-transformed image to the estimated values in the first color plane.

4. The method of claim 3, wherein the first color plane is more densely sampled than the remaining color planes.

5. The method of claim 4, wherein a green color plane is most densely sampled;

and wherein missing information in the red and blue color planes is estimated by the steps of transforming, smoothing, re-transforming, and re-setting.

6. The method of claim 1, wherein the luminance/chrominance color space is (l,c₁,c₂) color space where

$\left(\begin{array}{c}l\\ c_1\\ c_2\end{array}\right)=T$

( R G B ) ⁢

 

⁢

and ⁢

 

⁢

where ⁢

 

$T=\left(\begin{array}{ccc}1/\sqrt{3}& 0& 0\\ 0& 1/\sqrt{2}& 0\\ 0& 0& 1/\sqrt{6}\end{array}\right)<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\begin{array}{ccc}1& 1& 1\\ 1& -1& 0\\ 1& 1& -2\end{array}\right).$

7. The method of claim 1, wherein the luminance/chrominance color space is YIQ color space.

8. The method of claim 1, further comprising smoothing a luminance component of the transformed image.

9. The method of claim 8, wherein the smoothing of the luminance component is light and the smoothing of the chrominance components is heavy.

10. The method of claim 1, wherein a linear algorithm is used to estimate the output image;

wherein the smoothing of the chrominance components is linear; and

wherein the method is performed as a single linear operation.

11. The method of claim 1, further comprising at least one additional iteration, each iteration including transforming the output image, smoothing the chrominance components of the transformed output image, re-transforming the image, and re-setting corresponding pixel values.

12. The method of claim 1, further comprising performing post-processing, the post-processing including sharpening the full color image.

13. The method of claim 1, wherein the mosaic image corresponds to a Bayer CFA.

14. An article comprising memory encoded with an image produced by the method of claim 1.

15. Apparatus for performing the method of claim 1.

16. An article for a processor, the article comprising memory encoded with data for causing the processor to perform the method of claim 1.

17. An article comprising memory encoded with filter kernels for image processing according to the method of claim 1.

18. The article of claim 16, wherein the filter kernels are generated by treating the method as block-shift invariant.

19. Apparatus comprising a processor for processing a mosaic digital image, the processing including estimating a full color output image from the mosaic image;

transforming the output image from an original color space into a luminance/chrominance color space;

smoothing chrominance components of the transformed image;

re-transforming the image back to the original color space; and

re-setting corresponding pixels in the re-transformed image to sampled values of the mosaic image.

20. The apparatus of claim 19, wherein estimating the full color output image includes using the same algorithm to estimate missing information in all color planes of the output image.

21. The apparatus of claim 19, wherein estimating the full color output image includes using a first algorithm to estimate missing information in a first color plane, and using a second algorithm to estimate missing information in remaining color planes of the output image;

and wherein the method further comprises setting corresponding pixels in the re-transformed image to the estimated values in the first color plane.

22. The apparatus of claim 19, wherein the first color plane is more densely sampled than the remaining color planes.

23. The apparatus of claim 19, wherein a green color plane is most densely sampled;

and wherein missing information in the red and blue color planes is estimated by the steps of transforming, smoothing, re-transforming, and re-setting the original samples.

24. The apparatus of claim 19, wherein the luminance/chrominance color space is (l,c₁,c₂) color space.

25. The apparatus of claim 19, wherein the processing further includes smoothing a luminance component of the transformed image.

26. The apparatus of claim 25, wherein the smoothing of the luminance component is light and the smoothing of the chrominance components is heavy.

27. The apparatus of claim 19, wherein a linear algorithm is used to estimate the full color output image;

wherein the smoothing of the chrominance components is linear; and

wherein the method is performed as a single linear operation.

28. The apparatus of claim 19, wherein filter kernels are used to perform the single linear operation.

29. The apparatus of claim 19, wherein the processing further includes performing at least one additional iteration, where each iteration includes transforming the output image, smoothing the chrominance components of the transformed output image, re-transforming the image, and re-setting corresponding pixels.

30. The apparatus of claim 19, wherein the processing further includes sharpening the full color image during post-processing.

31. The apparatus of claim 19, further comprising a capture device for generating the mosaic image.

32. A digital camera comprising:

a photosensor array;

a processor; and

memory encoded with data, the data used by the processor to process an output of the photosensor array, the processing including estimating a full color output image from the array output;

smoothing chrominance components of the transformed image; and

re-setting corresponding pixels in the output image to the original sampled values.

33. The digital camera of claim 32, wherein estimating the full color output image includes using the same algorithm to estimate missing information in all color planes of the output image.

34. The digital camera of claim 32, wherein estimating the full color output image includes using a first algorithm to estimate missing information in a first color plane, and using a second algorithm to estimate missing information in remaining color planes of the output image;

and wherein the method further comprises setting corresponding pixels in the re-transformed image to the estimated values in the first color plane.

35. The digital camera of claim 32, wherein the processing is linear, and wherein the data causes the processor to perform the demosaicing in a single linear operation.

36. The digital camera of claim 35, wherein the data includes filter kernels.

37. The digital camera of claim 32, wherein the data includes a program for instructing the processor to perform the processing.

38. Apparatus comprising:

means for accessing sampled values of an undersampled digital image; and

means for processing the undersampled image to produce a full color image, the processing including estimating a full color output image from the undersampled image;

transforming the output image from an original color space into a luminance/chrominance color space;

smoothing chrominance components of the transformed image;

re-transforming the smoothed image back to the original color space; and

re-setting corresponding pixels in the re-transformed image to the sampled values.

39. An article for a processor, the article comprising data used by the processor to process undersampled images, the processing including estimating a full color output image of an undersampled image being processed;

smoothing chrominance components of the transformed image; and

re-setting corresponding pixels in the output image to sampled values in the undersampled image being processed.

40. The article of claim 39, wherein the data includes filter kernels.

41. The article of claim 39, wherein the data includes a demosaicing program.

42. Apparatus comprising a processor for generating data, the data used to process a mosaic image, the processing including the steps of estimating a full color output image from the mosaic image;

transforming the output image from an original color space into a luminance/chrominance color space;

smoothing chrominance components of the transformed image;

re-transforming the smoothed image back to the original color space; and

re-setting corresponding pixels in the re-transformed image to the original sampled values.

43. The apparatus of claim 42, wherein the data comprises a set of filter kernels.