Computational camera using fusion of image sensors

• US 9,654,700 B2
• Filed: 09/16/2014
• Issued: 05/16/2017
• Est. Priority Date: 09/16/2014
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Inventors: Kovtun, Ivan, Musatenko, Yuriy, Kysenko, Volodymyr, Proca, Adrian M, Stetson, Philip S, Ivannikov, Yevhen
• Primary Examiner(s): Nazrul, Shahbaz
• Application Number:
  US14/487,785
• Publication Number:
  US 20160080626A1
• Time in Patent Office: 973 Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

A camera device includes monochromatic and color image sensors that capture an image as a clear image in monochrome and as a Bayer image. The camera device implements image processing algorithms to produce an enhanced, high-resolution HDR color image. The Bayer image is demosaiced to generate an initial color image, and a disparity map is generated to establish correspondence between pixels of the initial color image and clear image. A mapped color image is generated to map the initial color image onto the clear image. A denoised clear image is applied as a guide image of a guided filter that filters the mapped color image to generate a filtered color image. The filtered color image and the denoised clear image are then fused to produce an enhanced, high-resolution HDR color image, and the disparity map and the mapped color image are updated based on the enhanced, high-resolution HDR color image.

Claims

1. A camera device, comprising:

one or more monochromatic image sensors configured to capture light of an image as a clear image in monochrome;

one or more HDR color image sensors configured to capture the light of the image as a Bayer image;

a processing system to implement image processing algorithms of an image fusion application that is configured to;

demosaic the Bayer image to generate an initial color image;

generate a disparity map to establish correspondence between color image pixels of the initial color image and clear image pixels of the clear image;

generate a mapped color image based on the disparity map to map the initial color image onto the clear image;

denoise the clear image to generate a denoised clear image;

apply the denoised clear image as a guide image of a guided filter that filters the mapped color image to generate a filtered color image; and

fuse the filtered color image and the denoised clear image to produce an enhanced, high-resolution HDR color image.

2. The camera device as recited in claim 1, wherein the image fusion application is configured to reiterate to update the disparity map and the mapped color image based on the enhanced, high-resolution HDR color image to map the enhanced, high-resolution HDR color image onto the clear image.

3. The camera device as recited in claim 2, wherein the image fusion application is configured to utilize the clear image to said demosaic the Bayer image based on the reiterated use of the enhanced, high-resolution HDR color image to said update the disparity map and the mapped color image.

4. The camera device as recited in claim 1, wherein the mapped color image is a red, green, blue (RGB) color image from which the filtered color image is generated from the guided filter and used to produce the enhanced, high-resolution HDR color image.

5. The camera device as recited in claim 1, wherein the mapped color image is a luma and chrominance (YUV) components image from which the filtered color image is generated from the guided filter and used to produce the enhanced, high-resolution HDR color image.

6. The camera device as recited in claim 1, wherein the image fusion application is configured to said generate the enhanced, high-resolution HDR color image having a high signal-to-noise ratio derived from the denoised clear image.

7. The camera device as recited in claim 1, wherein the image fusion application is configured for pixel correction by images rectification of the initial color image and the clear image to said generate the disparity map.

8. The camera device as recited in claim 1, wherein the image fusion application reiterates to said update the disparity map to improve color sampling over the initial color image.

9. The camera device as recited in claim 1, wherein the image fusion application reiterates to update a minimum spanning tree to said map the initial color image onto the clear image.

10. A method, comprising:

capturing light of an image as both a Bayer image and a clear image in monochrome;

demosaicing the Bayer image to generate an initial color image;

generating a disparity map to establish correspondence between color image pixels of the initial color image and clear image pixels of the clear image;

generating a mapped color image based on the disparity map to map the initial color image onto the clear image;

denoising the clear image to generate a denoised clear image;

applying the denoised clear image as a guide image of a guided filter that filters the mapped color image to generate a filtered color image; and

fusing the filtered color image and the denoised clear image to produce an enhanced, high-resolution HDR color image.

11. The method as recited in claim 10, further comprising:

reiterating to update the disparity map and the mapped color image based on the enhanced, high-resolution HDR color image to map the enhanced, high-resolution HDR color image onto the clear image.

12. The method as recited in claim 11, wherein the clear image is utilized to demosaic the Bayer image based on said reiterating use of the enhanced, high-resolution HDR color image to update the disparity map and the mapped color image.

13. The method as recited in claim 10, wherein the mapped color image is a red, green, blue (RGB) color image from which the filtered color image is generated from the guided filter and used to produce the enhanced, high-resolution HDR color image.

14. The method as recited in claim 10, wherein the mapped color image is a luma and chrominance (YUV) components image from which the filtered color image is generated from the guided filter and used to produce the enhanced, high-resolution HDR color image.

15. The method as recited in claim 10, wherein the enhanced, high-resolution HDR color image is generated having a high signal-to-noise ratio derived from the denoised clear image.

16. The method as recited in claim 10, further comprising:

rectifying the initial color image and the clear image for pixel correction to generate the disparity map.

17. The method as recited in claim 10, wherein said reiterating to update the disparity map to improve color sampling over the initial color image, and said updating the mapped color image based on the enhanced, high-resolution HDR color image and a minimum spanning tree to map the initial color image onto the clear image.

18. A portable device, comprising:

a camera device configured to capture light of an image as both a Bayer image and a clear image in monochrome;

one or more computer processors; and

one or more computer-readable storage media having executable instructions thereon that, in response to execution by the one or more computer processors, perform operations comprising;

performing a demosaicing algorithm to generate an initial color image from the Bayer image;

performing a dynamic shifts algorithm to generate a disparity map that establishes correspondence between color image pixels of the initial color image and clear image pixels of the clear image;

performing a mapping algorithm to generate a mapped color image based on the disparity map to map the initial color image onto the clear image;

performing a fusion algorithm to apply a denoised clear image as a guide image of a guided filter that filters the mapped color image to generate a filtered color image, and fuse the filtered color image and the denoised clear image to produce an enhanced, high-resolution HDR color image; and

performing the dynamic shifts algorithm to update the disparity map based on the enhanced, high-resolution HDR color image.

19. The portable device as recited in claim 18, wherein the operations further comprise:

performing the mapping algorithm to reiterate generating the mapped color image to map the enhanced, high-resolution HDR color image onto the clear image; and

performing the demosaicing algorithm to utilize the clear image to demosaic the Bayer image based on the reiterated use of the enhanced, high-resolution HDR color image.

20. The portable device as recited in claim 18, wherein performing the fusion algorithm produces the enhanced, high-resolution HDR color image having a high signal-to-noise ratio derived from the denoised clear image.