Methods of image fusion for image stabilization

• US 9,262,684 B2
• Filed: 06/06/2013
• Issued: 02/16/2016
• Est. Priority Date: 06/06/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Tico, Marius, Zhou, Jianping, Nariani Schulze, Anita, Toft, Rolf, Hubel, Paul, Sun, Wei
• Primary Examiner(s): YANG, ANDREW GUS
• Application Number:
  US13/911,740
• Publication Number:
  US 20140363087A1
• Time in Patent Office: 985 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/10 using non-spatial domain filtering
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

Systems, methods, and computer readable media to improve image stabilization operations are described. Novel approaches for fusing non-reference images with a pre-selected reference frame in a set of commonly captured images are disclosed. The fusing approach may use a soft transition by using a weighted average for ghost/non-ghost pixels to avoid sudden transition between neighborhood and almost similar pixels. Additionally, the ghost/non-ghost decision can be made based on a set of neighboring pixels rather than independently for each pixel. An alternative approach may involve performing a multi-resolution decomposition of all the captured images, using temporal fusion, spatio-temporal fusion, or combinations thereof, at each level and combining the different levels to generate an output image.

Claims

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause the programmable control device to:

obtain a first image of a scene captured at a first time, the first image having a plurality of pixels;

perform a multi-resolution decomposition of the first image to generate a first multi-level pyramid representation of the first image;

obtain a second image of the scene, wherein the second image is captured at a different time from the first image, and wherein each of the plurality of pixels in the first image has a corresponding pixel in the second image;

perform a multi-resolution decomposition of the second image to generate a second multi-level pyramid representation of the second image;

generate a level in an output multi-level pyramid representation of the scene for each level in the first and second multi-level pyramid representations of the scene in accordance with instructions to cause the programmable control device to;

identify, for a group of pixels in a level of the first multi-level pyramid representation of the scene, a corresponding group of pixels in the level from the second multi-level pyramid representation of the scene;

determining a weight value associated with each group of pixels in the level from the first multi-level pyramid representation of the scene and the identified corresponding group of pixels in the level from the second multi-level pyramid representation of the scene;

perform a spatial search of the level of the second multi-level pyramid representation of the scene to find a better corresponding group of pixels when the weight value is less than or equal to a specified threshold;

determine a weight value for the better corresponding group of pixels; and

fuse the identified group of pixels from the first multi-level pyramid representations of the scene with the better corresponding group of pixels from the second multi-level pyramid representation when the determined weight value for the better corresponding group of pixels is greater than the specified threshold;

repeat the instructions to cause the programmable control device to generate a level in the output multi-level pyramid representation of the scene for each level in the multi-resolution decomposition of the first image;

combine the output multi-level pyramid representations of the scene to generate a single output image representative of the scene; and

store the output image in a memory.

2. The non-transitory program storage device of claim 1, further comprising instructions to cause the programmable control device to:

not fuse the identified group of pixels from the first multi-level pyramid representations of the scene with the better corresponding group of pixels from the second multi-level pyramid representation when the determined weight value for the better corresponding group of pixels is less than or equal to the specified threshold.

3. The non-transitory program storage device of claim 1, wherein the instructions to cause the programmable control device to determine a weight value associated with each group of pixels in the level from the first multi-level pyramid representation of the scene and the identified corresponding group of pixels in the level from the second multi-level pyramid representation of the scene comprise instructions to cause the programmable control device to:

compare a similarity between each pixel in the group of pixels in the level of the first multi-level pyramid representation of the scene and a corresponding pixel in the identified corresponding group of pixels in the level from the second multi-level pyramid representation of the scene;

obtain a pixel similarity value based on the comparison;

obtain an expected noise content for each pixel in the group of pixels in the level of the first multi-level pyramid representation of the scene; and

calculate the weight value based on the pixel similarity value and the expected noise content.

4. The non-transitory program storage device of claim 1, wherein the better corresponding group of pixels comprises a group of pixels with a weight value greater than the specified threshold.

5. The non-transitory program storage device of claim 1, wherein the instructions to cause the programmable control device to fuse the identified group of pixels from the first multi-level pyramid representations of the scene with the better corresponding group of pixels from the second multi-level pyramid representation comprise instructions to cause the programmable control device to:

estimate a motion field for each second multi-level pyramid representation of the second image, wherein the motion field associates pixels from the first image with pixels from the second image; and

fuse each pixel from the first multi-level pyramid representation with a pixel selected from the motion field.

6. The non-transitory program storage device of claim 1, wherein the instructions to cause the programmable control device to fuse the identified group of pixels from the first multi-level pyramid representations of the scene with the better corresponding group of pixels from the second multi-level pyramid representation comprise instructions to cause the programmable control device to fuse one pixel from the first multi-level pyramid representation with two or more pixels from the better corresponding group of pixels from the second multi-level pyramid representation, wherein spatial coordinates of the two or more pixels are in a pre-determined neighborhood of an identified corresponding pixel to the one pixel in the second multi-level pyramid representation.

7. The non-transitory program storage device of claim 1, wherein the instructions to generate a first multi-level pyramid representation of the first image comprise instructions to cause the programmable control device to generate a high-pass pyramid decomposition of the first image.

8. The non-transitory program storage device of claim 1, wherein the instructions to generate a first multi-level pyramid representation of the first image comprise instructions to cause the programmable control device to generate a wavelet decomposition of the first image.

9. A system comprising:

an image capture device;

memory; and

one or more programmable control devices operable to interact with the image capture device and the memory, and to perform operations comprising;

obtaining a first image of a scene captured at a first time, the first image having a plurality of pixels;

performing a multi-resolution decomposition of the first image to generate a first multi-level pyramid representation of the first image;

obtaining a second image of the scene, wherein the second image is captured at a different time from the first image, and wherein each of the plurality of pixels in the first image has a corresponding pixel in the second image;

performing a multi-resolution decomposition of the second image to generate a second multi-level pyramid representation of the second image;

generating a level in an output multi-level pyramid representation of the scene for each level in the first and second multi-level pyramid representations of the scene in accordance with instructions to cause the programmable control device to perform operations comprising;

identifying, for a group of pixels in a level of the first multi-level pyramid representation of the scene, a corresponding group of pixels in the level from the second multi-level pyramid representation of the scene;

determining a weight value associated with each group of pixels in the level from the first multi-level pyramid representation of the scene and the identified corresponding group of pixels in the level from the second multi-level pyramid representation of the scene;

performing a spatial search of the level of the second multi-level pyramid representation of the scene to find a better corresponding group of pixels when the weight value is less than or equal to a specified threshold;

determining a weight value for the better corresponding group pixels; and

fusing the identified group of pixels from the first multi-level pyramid representations of the scene with the better corresponding group of pixels from the second multi-level pyramid representation of the scene when the determined weight value for the better corresponding group of pixels is greater than the specified threshold;

combining the output multi-level pyramid representations of the scene to generate a single output image representative of the scene; and

storing the output image in a memory.

10. The system of claim 9, wherein the one or more programmable control devices perform operations further comprising:

not fusing the identified group of pixels from the first multi-level pyramid representations of the scene with the better corresponding group of pixels from the second multi-level pyramid representation of the scene when the determined weight value for the better corresponding group of pixels is less than or equal to the specified threshold.

11. The system of claim 9, wherein determining a weight value associated with each group of pixels in the level from the first multi-level pyramid representation of the scene and the identified corresponding group of pixels in the level from the second multi-level pyramid representation of the scene comprises:

comparing a similarity between each pixel in the group of pixels in the level of the first multi-level pyramid representation of the scene and a corresponding pixel in the identified corresponding group of pixels in the level from the second multi-level pyramid representation of the scene;

obtaining a pixel similarity value based on the comparison;

obtaining an expected noise content for the group of pixels in the level of the first multi-level pyramid representation of the scene; and

calculating the weight value based on the pixel similarity value and the expected noise content.

12. The system of claim 9, wherein the better corresponding group of pixels comprises a group of pixels with a weight value greater than the specified threshold.

13. The system of claim 9, wherein fusing the identified group of pixels from the first multi-level pyramid representations of the scene with the better corresponding group of pixels from the second multi-level pyramid representation comprises:

estimating a motion field for each second multi-level pyramid representation of the second image, wherein the motion field associates pixels from the first image with pixels from the second image; and

fusing each pixel from the first multi-level pyramid representation with a pixel selected from the motion field.

14. The system of claim 9, wherein fusing the identified group of pixels from the first multi-level pyramid representations of the scene with the better corresponding group of pixels from the second multi-level pyramid representation of the scene comprises fusing one pixel from the first multi-level pyramid representation with two or more pixels from the better corresponding group of pixels from the second multi-level pyramid representation, wherein spatial coordinates of the two or more pixels are in a pre-determined neighborhood of an identified corresponding pixel to the one pixel in the second multi-level pyramid representation.

15. A method comprising:

obtaining a first image of a scene captured at a first time, the first image having a plurality of pixels;

performing a multi-resolution decomposition of the first image to generate a first multi-level pyramid representation of the first image;

obtaining a second image of the scene, wherein the second image is captured at a different time from the first image, and wherein each of the plurality of pixels in the first image has a corresponding pixel in the second image;

performing a multi-resolution decomposition of the second image to generate a second multi-level pyramid representation of the second image;

generating a level in an output multi-level pyramid representation of the scene for each level in the first and second multi-level pyramid representations of the scene, wherein generating the level comprises;

identifying, for a group of pixels in a level of the first multi-level pyramid representation of the scene, a corresponding group of pixels in the level from the second multi-level pyramid representation of the scene;

determining a weight value associated with each group of pixels in the level from the first multi-level pyramid representation of the scene and the identified corresponding group of pixels in the level from the second multi-level pyramid representation of the scene;

performing a spatial search of the level of the second multi-level pyramid representation of the scene to find a better corresponding group of pixels when the weight value is less than or equal to a specified threshold;

determining a weight value for the better corresponding group of pixels; and

fusing the identified group of pixels from the first multi-level pyramid representations of the scene with the better corresponding group of pixels from the second multi-level pyramid representation of the scene when the determined weight value for the better corresponding group of pixels is greater than the specified threshold;

combining the output multi-level pyramid representations of the scene to generate a single output image representative of the scene; and

storing the output image in a memory.

16. The method of claim 15, further comprising:

not fusing the identified group of pixels from the first multi-level pyramid representations of the scene with the better corresponding group of pixels from the second multi-level pyramid representation of the scene when the determined weight value for the better corresponding group of pixel is less than or equal to the specified threshold.

17. The method of claim 15, wherein determining a weight value associated with each group of pixels in the level from the first multi-level pyramid representation of the scene and the identified corresponding group of pixels in the level from the second multi-level pyramid representation of the scene comprises:

comparing a similarity between each pixel in the group of pixels in the level of the first multi-level pyramid representation of the scene and a corresponding pixel in the identified corresponding group of pixels in the level from the second multi-level pyramid representation of the scene;

obtaining a pixel similarity value based on the comparison;

obtaining an expected noise content for the group of pixels in the level of the first multi-level pyramid representation of the scene; and

calculating the weight value based on the pixel similarity value and the expected noise content.

18. The method of claim 16, wherein the better corresponding group of pixels comprises a group of pixels with a weight value greater than the specified threshold.

19. The method of claim 15, wherein fusing the identified group of pixels from the first multi-level pyramid representations of the scene with the better corresponding group of pixels from the second multi-level pyramid representation comprises:

estimating a motion field for each second multi-level pyramid representation of the second image, wherein the motion field associates pixels from the first image with pixels from the second image; and

fusing each pixel from the first multi-level pyramid representation with a pixel selected from the motion field.

20. The method of claim 15, wherein fusing the identified group of pixels from the first multi-level pyramid representations of the scene with the better corresponding group of pixels from the second multi-level pyramid representation comprises fusing one pixel from the first multi-level pyramid representation with two or more pixels from the better corresponding group of pixels from the second multi-level pyramid representation, wherein spatial coordinates of the two or more pixels are in a pre-determined neighborhood of an identified corresponding pixel to the one pixel in the second multi-level pyramid representation.