Image mosaic data reconstruction

• US 7,071,978 B2
• Filed: 07/18/2001
• Issued: 07/04/2006
• Est. Priority Date: 07/18/2001
• Status: Active Grant

• Current Assignee: Dolby International AB (Dolby Laboratories Incorporated)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Hunter, Andrew Arthur, Pollard, Stephen B
• Primary Examiner(s): Ho, Tuan
• Assistant Examiner(s): Henn, Timothy J.
• Application Number:
  US09/906,625
• Publication Number:
  US 20030025814A1
• Time in Patent Office: 1,812 Days
• Field of Search: 348/222.1, 348/234, 348/235, 348/252, 348/266, 348/272, 348/273, 348/280, 348/281
• US Class Current: 348/266
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

This invention relates to the reconstruction of a full colour image from image mosaic data in particular where the image mosaic data is unequally weighted between different colours. The image mosaic is composed of a plurality of image pixels. Each image pixel has one of at least three colour values and has a luminance value representing the intensity of the colour value for that pixel in the image mosaic. The pixels of each colour value are interleaved across the image mosaic with pixels of different colour values to form the image mosaic. For each colour value, both a low spatial frequency monochrome image and a high spatial frequency luminance image are generated, the high spatial frequency luminance image extending only across pixels locations of the image mosaic for that colour value. Each of the high spatial frequency luminance images is then combined with a corresponding low spatial frequency monochrome image to form the full colour image.

Claims

1. A method of de-mosaicing an image mosaic to form a full colour image, the image mosaic being composed of a plurality of image pixels and each image pixel having one of at least three colour values and having a luminance value representing the intensity of the colour value for that pixel in the image mosaic, the pixels of each colour value being interleaved across the image mosaic with pixels of different colour values to form the image mosaic, the method comprising the steps of:

i) for each colour value, generating from the pixels for that colour value a low spatial frequency monochrome image;

ii) for each colour value, generating from the pixels for that colour value a high spatial frequency luminance image that extends only across pixel locations of the image mosaic for that colour value;

iii) combining each of the high spatial frequency luminance images with each of the corresponding low spatial frequency monochrome images to form the full colour image.

2. A method as claimed in claim 1, comprising the step of combining the three or more high spatial frequency images to form a high spatial frequency luminance image that extends across all pixel locations of the image mosaic.

3. A method as claimed in claim 1, in which in step i) the low spatial frequency monochrome image extends across all pixel locations of the image mosaic.

4. A method as claimed in claim 1, in which in step i) the low spatial frequency monochrome image is calculated separately in local areas of the image mosaic.

5. A method as claimed in claim 1, comprising the steps of:

iv) forming a plurality of mosaic data sets, one for each colour value, and each such data set comprising elements that represent the luminance values of the pixels which have the corresponding colour value;

v) generating, from each image mosaic data set, a smoothed image data set representative of the corresponding low spatial frequency monochrome image, and each such data set comprising elements that represent smoothed luminance values across all pixel locations of the image mosaic;

vi) generating, from each image mosaic data set, an image data set representative of the corresponding high spatial frequency image, and each such data set comprising elements that represent high frequency luminance values of the pixels which have the corresponding colour value;

vii) combining each high frequency image data with a smoothed image data set to form a plurality of de-mosaiced image data sets, one for each colour value, and each such data set comprising elements that represent de-mosaiced luminance values of the pixels which have the corresponding colour value.

6. A method as claimed in claim 1, in which each of the low spatial frequency monochrome images is formed by two-dimensional interpolation combined with low pass filtering.

7. A method as claimed in claim 1, in which each of the high spatial frequency images is formed for each of the colour values from a difference between the luminance values of the image mosaic pixels for said colour value and corresponding portions of the low spatial frequency monochrome image for said colour value.

8. A method as claimed in claim 1, in which additional gain is applied to the high frequency image prior to combining it with the smoothed image data for each colour value in order to sharpen the resulting full colour image.

9. A method as claimed in claim 1, in which the full colour image is formed for each of the colour values by summing each of the high spatial frequency images with corresponding portions of the low spatial frequency monochrome images for said colour value.

10. A method as claimed in claim 1, in which there are three colour values, namely red, green and blue, and there is a predominance of pixels having a green colour value.

11. A method as claimed in claim 10, in which the colour values of pixels in the image mosaic are arranged in a Bayer pattern.

12. A method as claimed in claim 1, wherein the step of combining comprises addition of each of the high spatial frequency luminance images to a corresponding low spatial frequency monochrome images to form the full colour image.

14. A device as claimed in claim 12, wherein the processor, software and memory are operable to combine the high frequency image data from the three or more colour values to form composite high frequency image data representative of a high spatial frequency luminance image that extends across all pixel locations of the image mosaic.

13. A device for de-mosaicing an image mosaic to form a full colour image, the device comprising a processor, software, and a memory, in which the memory stores image mosaic data representative of an image mosaic having a plurality of image pixels, said image mosaic data including for each pixel one of at least three colour values and a luminance value representing the intensity of the colour value for that pixel in the image mosaic, the pixels of each colour value being interleaved across the image mosaic with pixels of different colour values to form the image mosaic, wherein the processor, software and memory are operable to:

a) generate from the image mosaic data, for each colour value, smoothed image data representative of a low spatial frequency monochrome image;

b) generate from the image mosaic data, for each colour value, image data representative of a high spatial frequency image for that colour value that extends only across pixels locations of the image mosaic for that colour value;

c) combine the high frequency image data with the smoothed image data for each colour value to form de-mosaiced image data representative of the full colour image.

15. A device as claimed in claim 13, wherein the processor, software and memory are operable to form each of the high spatial frequency images for each of the colour values from a difference between the luminance values of the image mosaic pixels for said colour value and corresponding portions of the low spatial frequency monochrome image for said colour value.

16. A device as claimed in claim 13, wherein the processor, software and memory are operable to:

d) form a plurality of mosaic data sets, one for each colour value, and each such data set comprising elements that represent the luminance values of the pixels which have the corresponding colour value;

e) generate, from each image mosaic data set, a smoothed image data set representative of the corresponding low spatial frequency monochrome image, and each such data set comprising elements that represent smoothed luminance values across all pixel locations of the image mosaic;

f) generate, from each image mosaic data set, sri image data set representative of the corresponding high spatial frequency image, and each such data set comprising elements that represent high frequency luminance values of the pixels which have the corresponding colour value;

g) combine each high frequency image data with a smoothed image data set to form a plurality of de-mosaiced image data sets, one for each colour value, and each such data set comprising elements that represent de-mosaiced luminance values of the pixels which have the corresponding colour value.

17. A device as claimed in claim 13, wherein the processor, software and memory are operable to form each of the low spatial frequency monochrome images is formed by two-dimensional interpolation combined with low pass filtering.

18. A device as claimed in claim 13, wherein the processor, software and memory are operable to apply additional gain and to the high frequency image prior to combining it with the smoothed image data for each colour value in order to sharpen the resulting full colour image.

19. A device as claimed in claim 13, wherein the processor, software and memory are operable to form the full color image for each of the colour values by summing each of the high spatial frequency images with corresponding portions of the low spatial frequency monochrome images for said colour value.

20. The device of claim 13 wherein the combining operation comprises adding each of the high spatial frequency luminance images to a corresponding low spatial frequency monochrome images to form the full colour image.