Image mosaic data reconstruction

• US 7,006,686 B2
• Filed: 07/18/2001
• Issued: 02/28/2006
• Est. Priority Date: 07/18/2001
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Hunter, Andrew Arthur, Pollard, Stephen B
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): Hung, Yubin
• Application Number:
  US09/906,786
• Publication Number:
  US 20030063201A1
• Time in Patent Office: 1,686 Days
• Field of Search: 382/162, 382/167, 382/263, 382/264, 382/268, 382/269, 382/279, 348/222.1, 348/241, 348/246, 348/272, 348/273, 348/280, 348234-238, 348/263, 345/606, 345/611, 345/615, 345/616, 358/518
• US Class Current: 382/162
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

This invention relates to the reconstruction of a full colour image from an image mosaic composed of a plurality of image pixels that have one of at least three colour values. The pixels are interleaved in rows and columns across the mosaic with pixels of different colour values to form the mosaic in such a way that each row and column contains pixels of at least two colours. A high spatial frequency luminance image is generated from the mosaic that extends across all pixel locations of the image mosaic. The luminance values of pixels in the high frequency image are adjusted to reduce zippering effects. The adjustment is performed on pixels generated from a particular colour value at least in a first row and/or a first column of the high frequency luminance image according to the luminance values for said pixels and the luminance values of pixels generated from another colour value. A low spatial frequency luminance image is generated from the mosaic for each colour value. After the luminance adjustment the high frequency luminance pixels are combined with each of the low frequency luminance images in order to generate a de-mosaiced image for each of the colour values corresponding to said low frequency luminance images.

Claims

1. An image processing method of de-mosaicing an image mosaic, the image mosaic including a plurality of image pixels and each image pixel having one of at least three colour values and having an intensity value representing the intensity of the colour value for that pixel in the image mosaic, the pixels of each colour value being interleaved in rows and columns across the image mosaic with pixels of different colour values and thereby form the image mosaic, the method comprising the steps of:

i) generating from the image mosaic a high spatial frequency image that extends across all pixel locations of the image mosaic, each pixel having an intensity value;

ii) reducing mosaic artifacts by adjusting the intensity values of pixels in the high spatial frequency image, said adjustment being performed on pixels generated from a particular colour value at least in a first row and/or a first column of the high frequency image according to the intensity values for said pixels and the intensity values of pixels generated from another colour value;

iii) generating for each colour value from the image mosaic a low spatial frequency intensity image, each pixel having an intensity value;

iv) after the adjustment of step ii), generating a demosaiced image for each of the colour values corresponding to said low frequency images by combining across all pixel locations the high frequency pixels with each of the low frequency intensity images;

v) in step i), forming the high spatial frequency image by first generating, for each colour value, from the pixels for that colour value, a partial high spatial frequency image that extends only across pixel locations of the image mosaic for that colour value;

vi) reducing mosaic artifacts by processing each of the partial high spatial frequency images according to step ii), and then using each of said processed partial high spatial frequency images in combination with each of the low frequency images to form a full colour demosaiced image.

2. An image processing method as claimed in claim 1, wherein step ii) includes adjusting the intensity values of the high frequency image in response to differences between neighbouring pixels in said first row and/or column of the high frequency image being below a first threshold value.

3. A method as claimed in claim 1, in which in step iii), the low spatial frequency image extends across all pixel locations of the image mosaic.

4. A method as claimed in claim 1, in which in step iii), the low spatial frequency image is calculated separately in local areas of the image mosaic.

5. A method as claimed in claim 1, comprising the step of:

vii) for each colour value, generating from the image mosaic pixels for that colour value a low spatial frequency image that extends across all pixel locations of the image mosaic.

6. An image processing method of de-mosaicing an image mosaic, the image mosaic including a plurality of image pixels and each image pixel having one of at least three colour values and having an intensity value representing the intensity of the colour value for that pixel in the image mosaic, the pixels of each colour value being interleaved in rows and columns across the image mosaic with pixels of different colour values and thereby form the image mosaic, the method comprising the steps of:

i) generating from the image mosaic a high spatial frequency image that extends across all pixel locations of the image mosaic, each pixel having an intensity value;

ii) reducing mosaic artifacts by adjusting the intensity values of pixels in the high spatial frequency image, said adjustment being performed on pixels generated from a particular colour value at least in a first row and/or a first column of the high frequency image according to the intensity values for said pixels and the intensity values of pixels generated from another colour value;

iii) generating for each colour value from the image mosaic a low spatial frequency intensity image, each pixel having an intensity value; and

iv) after the adjustment of step ii), generating a demosaiced image for each of the colour values corresponding to said low frequency images by combining across all pixel locations the high frequency pixels with each of the low frequency intensity images,

wherein step ii) includes adjusting the intensity values of the high frequency image in response to differences between neighbouring pixels in said first row and/or column of the high frequency image being below a first threshold value, wherein step ii) includes adjusting the intensity values of the high frequency image in response to differences between neighbouring pixels in said row(s) and column(s) of the high frequency image being below a second threshold value, said adjustment being performed on pixels generated from a particular colour value in both row(s) and column(s) of the high frequency image, and the second threshold value being less than the first threshold value.

7. An image processing method as claimed in claim 6, in which there is a predominance of pixels of one colour value as compared with the other colour values, each row and column having image pixels of both the predominant colour and one of the other colours.

8. An image processing method as claimed in claim 7, wherein step ii) includes assessing differences between one or more adjacent pairs of pixels that have been derived from the same non-predominant colour value, and differences between one or more adjacent pairs of pixels that have been derived from the predominant colour value.

9. An image processing method as claimed in claim 7, wherein the adjustment of step ii) includes scaling the intensity values of at least some of the pixels of the non-predominant colour value according to the intensity value of the low spatial frequency image of the pixels derived from the predominant colour value.

10. A method as claimed in claim 7, in which pixels with the predominant colour value alternate with pixels of one other colour in said row and/or column, and the difference in intensity values which is evaluated includes differences in intensity values between five immediately adjacent pixels in said row or column, said five pixels consisting of a first central pixel having the non-predominant colour, a second pixel and a third pixel each having the predominant colour, said second pixel being on one side of the first pixel and said third pixel being on the opposite side of the first pixel, and a fourth pixel and a fifth pixel each having the non-predominant colour, said fourth pixel being adjacent the second pixel and said fifth pixel being adjacent the third pixel.

11. An image processing method of de-mosaicing an image mosaic, the image mosaic including a plurality of image pixels and each image pixel having one of at least three colour values and having an intensity value representing the intensity of the colour value for that pixel in the image mosaic, the pixels of each colour value being interleaved in rows and columns across the image mosaic with pixels of different colour values and thereby form the image mosaic, the method comprising the steps of:

i) generating from the image mosaic a high spatial frequency image that extends across all pixel locations of the image mosaic, each pixel having an intensity value;

ii) reducing mosaic artifacts by adjusting the intensity values of pixels in the high spatial frequency image, said adjustment being performed on pixels generated from a particular colour value at least in a first row and/or a first column of the high frequency image according to the intensity values for said pixels and the intensity values of pixels generated from another colour value;

iii) generating for each colour value from the image mosaic a low spatial frequency intensity image, each pixel having an intensity value; and

iv) after the adjustment of step ii), generating a demosaiced image for each of the colour values corresponding to said low frequency images by combining across all pixel locations the high frequency pixels with each of the low frequency intensity images,

wherein step ii) includes adjusting the intensity values of the high frequency image in response to differences between neighbouring pixels in said first row and/or column of the high frequency image being below a first threshold value, in which there is a predominance of pixels of one colour value as compared with the other colour values, each row and column having image pixels of both the predominant colour and one of the other colours, wherein the adjustment of step ii) includes scaling the intensity values of at least some of the pixels of the non-predominant colour value according to the intensity value of the low spatial frequency image of the pixels derived from the predominant colour value, in which the intensity value of at least some of the pixels derived from the non-predominant colour value are scaled according to the intensity value of the low spatial frequency image of the pixels derived from the predominant colour value by adding the difference between the intensity value derived from said non-predominant colour value pixels and the low spatial frequency image derived from said non-predominant colour value pixels to intensity values from corresponding portions of said low spatial frequency image derived from the predominant colour value.

12. An image processing method of de-mosaicing an image mosaic, the image mosaic including a plurality of image pixels and each image pixel having one of at least three colour values and having an intensity value representing the intensity of the colour value for that pixel in the image mosaic, the pixels of each colour value being interleaved in rows and columns across the image mosaic with pixels of different colour values and thereby form the image mosaic, the method comprising the steps of:

i) generating from the image mosaic a high spatial frequency image that extends across all pixel locations of the image mosaic, each pixel having an intensity value;

ii) reducing mosaic artifacts by adjusting the intensity values of pixels in the high spatial frequency image, said adjustment being performed on pixels generated from a particular colour value at least in a first row and/or a first column of the high frequency image according to the intensity values for said pixels and the intensity values of pixels generated from another colour value;

iii) generating for each colour value from the image mosaic a low spatial frequency intensity image, each pixel having an intensity value; and

iv) after the adjustment of step ii), generating a demosaiced image for each of the colour values corresponding to said low frequency images by combining across all pixel locations the high frequency pixels with each of the low frequency intensity images

wherein step ii) includes adjusting the intensity values of the high frequency image in response to differences between neighbouring pixels in said first row and/or column of the high frequency image being below a first threshold value,

in which there is a predominance of pixels of one colour value as compared with the other colour values, each row and column having image pixels of both the predominant colour and one of the other colours

in which pixels with the predominant colour value alternate with pixels of one other colour in said row and/or column, and the difference in intensity values which is evaluated includes differences in intensity values between five immediately adjacent pixels in said row or column, said five pixels consisting of a first central pixel having the non-predominant colour, a second pixel and a third pixel each having the predominant colour, said second pixel being on one side of the first pixel and said third pixel being on the opposite side of the first pixel, and a fourth pixel and a fifth pixel each having the non-predominant colour, said fourth pixel being adjacent the second pixel and said fifth pixel being adjacent the third pixel, and

in which the difference in intensity values which includes a first difference between the second pixel and the third pixel, a second difference between the first pixel and the fourth pixel, and a third difference between the first pixel and the fifth pixel.

13. An image processing method of de-mosaicing an image mosaic, the image mosaic including a plurality of image pixels and each image pixel having one of at least three colour values and having an intensity value representing the intensity of the colour value for that pixel in the image mosaic, the pixels of each colour value being interleaved in rows and columns across the image mosaic with pixels of different colour values and thereby form the image mosaic, the method comprising the steps of:

i) generating from the image mosaic a high spatial frequency image that extends across all pixel locations of the image mosaic, each pixel having an intensity value;

ii) reducing mosaic artifacts by adjusting the intensity values of pixels in the high spatial frequency image, said adjustment being performed on pixels generated from a particular colour value at least in a first row and/or a first column of the high frequency image according to the intensity values for said pixels and the intensity values of pixels generated from another colour value;

iii) generating for each colour value from the image mosaic a low spatial frequency intensity image, each pixel having an intensity value; and

iv) after the adjustment of step ii), generating a demosaiced image for each of the colour values corresponding to said low frequency images by combining across all pixel locations the high frequency pixels with each of the low frequency intensity images,

wherein step ii) includes adjusting the intensity values of the high frequency image in response to differences between neighbouring pixels in said first row and/or column of the high frequency image being below a first threshold value,

wherein step ii) includes adjusting the intensity values of the high frequency image in response to differences between neighbouring pixels in said row(s) and column(s) of the high frequency image being below a second threshold value, said adjustment being performed on pixels generated from a particular colour value in both row(s) and column(s) of the high frequency image, and the second threshold value being less than the first threshold value, in which there is a predominance of pixels of one colour value as compared with the other colour values, each row and column including image pixels of both the predominant colour and one of the other colours.

14. An image processing method as claimed in claim 13, wherein step ii) includes assessing differences between one or more adjacent pairs of pixels that have been derived from the same non-predominant colour value, and differences between one or more adjacent pairs of pixels that have been derived from the predominant colour value.

15. An image processing method as claimed in claim 13, wherein the adjustment of step ii) includes scaling the intensity values of at least some of the pixels of the non-predominant colour value according to the intensity value of the low spatial frequency image of the pixels derived from the predominant colour value.

16. A method as claimed in claim 15, in which the intensity value of at least some of the pixels derived from the non-predominant colour value are scaled according to the intensity value of the low spatial frequency image of the pixels derived from the predominant colour value by adding the difference between the intensity value derived from said non-predominant colour value pixels and the low spatial frequency image derived from said non-predominant colour value pixels to intensity values from corresponding portions of said low spatial frequency image derived from the predominant colour value.

17. A method as claimed in claim 13, in which pixels with the predominant colour value alternate with pixels of one other colour in said row and/or column, and the difference in intensity values which is evaluated includes differences in intensity values between five immediately adjacent pixels in said row or column, said five pixels consisting of a first central pixel having the non-predominant colour, a second pixel and a third pixel each having the predominant colour, said second pixel being on one side of the first pixel and said third pixel being on the opposite side of the first pixel, and a fourth pixel and a fifth pixel each having the non-predominant colour, said fourth pixel being adjacent the second pixel and said fifth pixel being adjacent the third pixel.

18. A method as claimed in claim 17, in which the difference in intensity values which includes a first difference between the second pixel and the third pixel, a second difference between the first pixel and the fourth pixel, and a third difference between the first pixel and the fifth pixel.

19. An image processing method of de-mosaicing an image mosaic, the image mosaic including a plurality of image pixels and each image pixel having one of at least three colour values and having an intensity value representing the intensity of the colour value for that pixel in the image mosaic, the pixels of each colour value being interleaved in rows and columns across the image mosaic with pixels of different colour values and thereby form the image mosaic, the method comprising the steps of:

i) generating from the image mosaic a high spatial frequency image that extends across all pixel locations of the image mosaic, each pixel having an intensity value;

ii) reducing mosaic artifacts by adjusting the intensity values of pixels in the high spatial frequency image, said adjustment being performed on pixels generated from a particular colour value at least in a first row and/or a first column of the high frequency image according to the intensity values for said pixels and the intensity values of pixels generated from other colour values;

iii) generating for each colour value from the image mosaic a low spatial frequency intensity image, each pixel having intensity value; and

iv) after the adjustment of step ii), generating a demosaiced image for each of the colour values corresponding to said low frequency images by combining across all pixel locations the high frequency pixels with each of the low frequency intensity images,

v) in step i), forming the high spatial frequency image by first generating, for each colour value, from the pixels for that colour value, a partial high spatial frequency image that extends only across pixel locations of the image mosaic for that colour value;

vi) reducing mosaic artifacts by processing each of the partial high spatial frequency images according to step ii), and then using each of said processed partial high spatial frequency images in combination with each of the low frequency images to form a full colour de-mosaiced image.

20. A method as claimed in claim 19, comprising the steps of:

vii) for each colour value, generating from the image mosaic pixels for that colour value a low spatial frequency image that extends across all pixel locations of the image mosaic.

21. A device for de-mosaicing an image mosaic to form a full colour image, the device comprising a processor arrangement storing image mosaic data representative of an image mosaic having a plurality of image pixels, said image mosaic data including for each pixel one of at least three colour values and an intensity value representing the intensity of the colour value for that pixel in the image mosaic, the pixels of each colour value being interleaved in rows and columns across the image mosaic with pixels of different colour values to form the image mosaic, wherein the processor arrangement is operable to:

a) generate from the image mosaic a high spatial frequency image that extends across all pixel locations of the image mosaic, each pixel having an intensity value;

b) adjust the intensity values of pixels in the high spatial frequency image to reduce mosaic effects, said adjustment being performed on pixels generated from a particular colour value at least in a first row and/or a first column of the high frequency image according to the intensity values for said pixels and the intensity values of pixels generated from another colour value;

c) generate for each colour value from the image mosaic a low spatial frequency image, each pixel having an intensity value;

d) after the adjustment of the intensity value, combine across all pixel locations the high frequency pixels with each of the low frequency images in order to generate a demosaiced image for each of the colour values corresponding to said low frequency images;

e) in operation a), form the high spatial frequency image by first generating, for each colour value from the pixels for that colour value, a partial high spatial frequency image that extends only across pixel locations of the image mosaic for that colour value;

f) reduce mosaic artifacts by processing each of the partial high spatial frequency images according to operation b), and then using each of said processed partial high spatial frequency images in combination with each of the low frequency images to form a full colour de-mosaiced image.

22. The device as claimed in claim 21, wherein the processor arrangement is operable to:

g) generate, for each colour value from the image mosaic pixels for that colour value, a low spatial frequency image that extends across all pixel locations of the image mosaic.

23. An image processing method of de-mosaicing an image mosaic, the image mosaic including a plurality of image pixels and each image pixel having one of at least three colour values and having an intensity value representing the intensity of the colour value for that pixel in the image mosaic, the pixels of each colour value being interleaved in rows and columns across the image mosaic with pixels of different colour values and thereby form the image mosaic, the method comprising the steps of:

i) generating from the image mosaic a high spatial frequency image that extends across all pixel locations of the image mosaic, each pixel having an intensity value;

ii) reducing mosaic artifacts by adjusting the intensity values of pixels in the high spatial frequency image, said adjustment being performed on pixels generated from a particular colour value at least in a first row and/or a first column of the high frequency image according to the intensity values for said pixels and the intensity values of pixels generated from another colour value;

iii) generating for each colour value from the image mosaic a low spatial frequency intensity image, each pixel having an intensity value; and

iv) after the adjustment of step ii), generating a demosaiced image for each of the colour values corresponding to said low frequency images by combining across all pixel locations the high frequency pixels with each of the low frequency intensity images, wherein step ii) includes adjusting the intensity values of the high frequency image in response to differences between neighbouring pixels in said first row and/or column of the high frequency image being below a first threshold value and adjusting the intensity values of the high frequency image in response to differences between neighbouring pixels in said row(s) and column(s) of the high frequency image being between the first threshold value and above a second threshold value, said adjustment being performed on pixels generated from a particular colour value in both row(s) and column(s) of the high frequency image, and the second threshold value being less than the first threshold value.

24. An image processing method of de-mosaicing an image mosaic, the image mosaic including a plurality of image pixels, each image pixel having one of at least three colour values and having an intensity value representing the intensity of the colour value for that pixel in the image mosaic, the pixels of each colour value being interleaved in rows and columns across the image mosaic with pixels of different colour values thereby forming the image mosaic, the method comprising the steps of:

i) generating from the image mosaic a high spatial frequency image that extends across all pixel locations of the image mosaic, each pixel having an intensity value;

ii) reducing mosaic artefacts by adjusting pixel values in the high spatial frequency image, said adjustment being performed on pixels generated from one or more colour values in the original mosaic according to the colour values of at least one of (a) said pixels and (b) pixels generated from other colour values;

iii) generating for each colour value from the image mosaic a low spatial frequency intensity image, each pixel having an intensity value;

iv) after the adjustment of step ii), generating a de-mosaiced image for each of the colour values corresponding to said low frequency images by combining, across all pixel locations, the high frequency pixels with each of the low frequency intensity images;

v) in step i), forming the high spatial frequency image by first generating, for each colour value, from the pixels for that colour value, a partial high spatial frequency image that extends only across pixel locations of the image mosaic for that colour value; and

vi) reducing mosaic artifacts by processing each of the partial high spatial frequency images according to step ii), and then using each of said processed partial high spatial frequency images in combination with each of the low frequency images to form a full colour de-mosaiced image.

25. Apparatus for de-mosaicing an image mosaic, the image mosaic including a plurality of image pixels, each image pixel having one of at least three colour values and having an intensity value representing the intensity of the colour value for that pixel in the image mosaic, the pixels of each colour value being interleaved in rows and columns across the image mosaic with pixels of different colour values thereby forming the image mosaic, the apparatus comprising a processor arrangement for performing the steps of claim 24.