Low light image registration

• US 9,414,037 B1
• Filed: 09/26/2014
• Issued: 08/09/2016
• Est. Priority Date: 09/26/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Solh, Mashhour
• Primary Examiner(s): Flohre, Jason
• Application Number:
  US14/498,867
• Time in Patent Office: 683 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 1/3871 the composed originals being of different kinds, e.g. low- and high-resolution originals
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/76 by influencing the image signals
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 9/74 for obtaining special effects H04N9/65 - H04N9/73 take precedence

Abstract

Various embodiments provide a method of enhancing images in low lighting conditions wherein a color image is captured with a color camera, a first monochromatic image is captured with a first monochromatic camera, and a second monochromatic image is captured with a second monochromatic camera. Image registration is performed to align the first and second monochromatic images and the luminance information from the registered monochromatic images is fused with chrominance information from the color image.

Claims

1. A computer-implemented method for enhancing images captured in low light, the method comprising:

obtaining first monochromatic image data corresponding to a first monochromatic image of a first field of view;

obtaining second monochromatic image data corresponding to a second monochromatic image of a second field of view, the second field of view at least partially overlapping the first field of view;

obtaining color image data corresponding to a color image of a third field of view, the third field of view at least partially overlapping the first field of view and the second field of view;

determining mono registration data between the first monochromatic image data and the second monochromatic image data, the mono registration data corresponding to a mono registration image;

determining, using the first monochromatic image data and the second monochromatic image data, a depth map to identify foreground pixels and background pixels of a color image corresponding to the color image data, the depth map providing a first depth value for a foreground plane corresponding to the foreground pixels and a second depth value for a background plane corresponding to the background pixels;

applying a first homography to the color image data to modify at least one of a foreground rotation or translation of the color image relative to the mono registration image to generate foreground adjusted color image data, the first homography corresponding to the first depth value;

applying a second homography to the background plane to modify at least one of a background rotation or translation of the color image relative to the mono registration image to generate background adjusted color image data, the second homography corresponding to the second depth value;

comparing the foreground adjusted color image data to the mono registration data to determine a first error value for each pixel between the mono registration data and the foreground adjusted color image data, the first error value corresponding to a foreground difference between a first foreground location of a foreground pixel from the mono registration data and a second foreground location of a corresponding foreground pixel from the foreground adjusted color image data;

comparing the background adjusted color image data to the mono registration data to determine a second error value for each pixel between the mono registration data and the background adjusted color image data, the second error value corresponding to a background difference between a first background location of a background pixel from the mono registration data and a second background location of a corresponding background pixel from the background adjusted color image data;

selecting, for each pixel of the color image, one of the foreground adjusted color image data or the background adjusted color image data having a smallest error value between the first error value and the second error value to generate combined color image data;

removing, from the combined color image data, first luminance image data to generate chrominance image data; and

combining the chrominance image data with second luminance image data from the mono registration data to generate fused image data corresponding to a low-light enhanced color image.

2. The computer-implemented method of claim 1, further comprising:

refining the combined color image data by;

adjusting color image edge information from the combined color data using mono edge information from the mono registration data using a guided filter to generate first smoothed image data;

subtracting the adjusted color image edge information from the first smoothed image data to generate second smoothed image data;

applying a color averaging filter to the second smoothed image data to generate third smoothed image data; and

recombining the third smoothed image data with the adjusted color image edge information.

3. The computer-implemented method of claim 1, wherein the first monochromatic image data and the second monochromatic image data are captured at a first resolution and the color image data is captured at a larger second resolution, and wherein the mono registration data is upsampled from the first image resolution to match the second image resolution of the color image data.

4. A method, comprising:

obtaining, by a computing device, monochromatic image data;

obtaining, by the computing device, color image data;

determining, using the monochromatic image data and the color image data, a depth map to identify a first depth value for a foreground plane and a second depth value for a background plane;

applying, based on the first depth value, a first homography to the foreground plane to generate color foreground data;

applying, based on the second depth value, a second homography to the background plane to generate color background data;

selecting, for each subset of the color image data, a corresponding subset from the color background data or the color foreground data based on a smallest error between the color background data or the color foreground data when compared to the monochromatic image to generate combined color image data; and

combining chrominance image data from the combined color image data with luminance image data from the monochromatic image to generate fused image data.

5. The method of claim 4, wherein the first homography and the second homography are selected from a lookup table of predetermined homographies for a plurality of depth values, the plurality of depth values of the lookup table being determined by:

capturing, for each of the plurality of depth values, mono target image data of a target with a monochromatic camera;

capturing, for each of the plurality of depth values, color target image data of the target with a color camera;

determining a target depth map for each of the plurality of depth values, the target including a foreground target plane and a background target plane;

determining a target foreground homography between the mono target image data and the color target image data for each of the plurality of depth values;

determining a target background homography between the mono target image data and the color target image data for each of the plurality of depth values; and

storing, for each of the plurality of depth values, the target foreground homography and the target background homography in the lookup table.

6. The method of claim 5, wherein the plurality of depth values of the lookup table are further determined by:

determining, using the mono target image data, a first set of foreground plane feature points for each of the plurality of depth values;

determining, using the color target image data, a second set of foreground plane feature points for each of the plurality of depth values; and

determining, for each of the plurality of depth values, the target foreground homography by matching at least a subset of the first set of foreground plane feature points and the second set of foreground plane feature points.

7. The method of claim 6, wherein the plurality of depth values of the lookup table are further determined by:

determining, using the mono target image data, a first set of background plane feature points for each of the plurality of depth values;

determining, using the color target image data, a second set of background plane feature points for each of the plurality of depth values; and

determining, for each of the plurality of depth values, the target background homography by matching at least a subset of the first set of background plane feature points and the second set of background plane feature points.

8. The method of claim 4, further comprising:

removing luminance data from the combined color image data to generate the chrominance image data.

9. The method of claim 4, further comprising:

adjusting color image edge information from the combined color image data using mono edge information from the monochromatic image data using a guided filter to generate first smoothed image data;

subtracting the adjusted color image edge information from the first smoothed image data to generate second smoothed image data;

applying a color averaging filter to the second smoothed image data to generate third smoothed image data; and

recombining the third smoothed image data with the edge information.

10. The method of claim 4, further comprising:

comparing the color foreground data to the monochromatic image data to determine a first error value for each pixel between the monochromatic image data and the color foreground data; and

comparing the color background data to the monochromatic image data to determine a second error value for each pixel between the monochromatic image data and the color background data.

11. The method of claim 10, wherein the first error value and the second error value are determined using a sum of absolute differences algorithm, the first error value corresponding to a foreground difference between a first foreground location of a foreground pixel from the monochromatic image data and a second foreground location of a corresponding foreground pixel from the color foreground data.

12. The method of claim 4, wherein determining the depth map includes:

performing a histogram analysis of the monochromatic image data and the color image data to determine the first depth value of the foreground plane and the second depth value of the background plane.

13. A computing device, comprising:

a processor; and

memory including instructions that, when executed by the processor, cause the computing device to;

obtain first monochromatic image data;

obtain second monochromatic image data;

obtain color image data;

determine mono registration data between the first monochromatic image data and the second monochromatic image data;

determine a depth map of the color image data to identify a foreground plane and background plane;

apply, based on the depth map, a first homography to the color image data for the foreground plane to generate color foreground data;

apply, based on the depth map, a second homography to the color image data for the background plane to generate color background data;

select, for each subset of the color image data, a corresponding subset from the color background data or the color foreground data based on a smallest error between the color background data or the color foreground data when compared to the mono registration data to generate combined color image data;

remove, from the combined color image data, first luminance image data to generate chrominance image data; and

combine the chrominance image data with second luminance image data from the mono registration to generate a fused image.

14. The computing device of claim 13, wherein the first monochromatic image data and the second monochromatic image data are captured at a first image resolution and the color image data is captured at a larger second image resolution, and wherein the mono registration data is upsampled from the first image resolution to match the second image resolution of the color image data.

15. The computing device of claim 14, wherein the first homography is selected from a lookup table of predetermined homographies for a plurality of depth values, the plurality of depth values of the lookup table are determined by:

capturing, for each of the plurality of depth values, first mono target image data of a target with a first monochromatic camera;

capturing, for each of the plurality of depth values, second mono target image data of the target with a second monochromatic camera;

capturing, for each of the plurality of depth values, color target image data of the target with a color camera;

determine mono target registration data between the first mono target image data and the second mono target image data;

determining a target depth map for each of the plurality of depth values, the target including a foreground target plane and a background target plane;

determining a target foreground homography between the mono registration target image data and the color target image data for each of the plurality of depth values;

determining a target background homography between the mono registration target image data and the color target image data for each of the plurality of depth values; and

storing, for each of the plurality of depth values, the target foreground homography and the target background homography in the lookup table.

16. The computing device of claim 15, wherein the plurality of depth values of the lookup table are further determined by:

determining, using the mono registration target image data, a first set of foreground plane feature points for each of the plurality of depth values;

determining, using the color target image data, a second set of foreground plane feature points for each of the plurality of depth values; and

determining, for each of the plurality of depth values, the target foreground homography by matching at least a subset of the first set of foreground plane feature points and the second set of foreground plane feature points.

17. The computing device of claim 13, further comprising:

adjusting color image edge information from the combined image data using mono edge information from the monochromatic image data using a guided filter to generate first smoothed image data;

subtract the adjusted color image edge information from the first smoothed image data to generate second smoothed image data;

apply a color averaging filter to the second smoothed image data to generate third smoothed image data; and

recombine the third smoothed image data with the edge information.

18. The computing device of claim 13, wherein the instructions, when executed by the processor, further enable the computing device to:

compare the color foreground data to the mono registration data to determine a first error value for each pixel between the mono registration data and the color foreground data; and

compare the color background data to the mono registration data to determine a second error value for each pixel between the mono registration data and the color background data, wherein the first error and the second error are determined using a sum of absolute differences algorithm.

19. The computing device of claim 13, wherein determining the depth map includes performing a histogram analysis of the first monochromatic image data and the second monochromatic image data to determine a first depth value of the foreground plane and a second depth value of the background plane.