Image alignment for burst mode images

• US 10,453,204 B2
• Filed: 08/14/2017
• Issued: 10/22/2019
• Est. Priority Date: 12/06/2016
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Wang, Zhaowen, Jin, Hailin
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US15/676,903
• Publication Number:
  US 20180158199A1
• Time in Patent Office: 799 Days
• Field of Search: 382294
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images

Abstract

The present disclosure is directed towards systems and methods for generating a new aligned image from a plurality of burst image. The systems and methods subdivide a reference image into a plurality of local regions and a subsequent image into a plurality of corresponding local regions. Additionally, the systems and methods detect a plurality of feature points in each of the reference image and the subsequent image and determine matching feature point pairs between the reference image and the subsequent image. Based on the matching feature point pairs, the systems and methods determine at least one homography of the reference image to the subsequent image. Based on the homography, the systems and methods generate a new aligned image that is that is pixel-wise aligned to the reference image. Furthermore, the systems and methods refines boundaries between local regions of the new aligned image.

Claims

1. A computer-implemented method of generating aligned images from burst images, the method comprising:

detecting a plurality of feature points in a reference image and a subsequent image;

determining a feature descriptor for each feature point of the plurality of feature points;

based on the feature descriptor of each feature point, determining matching feature points between the reference image and the subsequent image;

determining at least one homography that maps matching feature points between the reference image and the subsequent image, wherein determining at least one homography comprises applying a homography motion model to a plurality of local regions individually to determine a homography for each local region that maps feature points in a local region of the reference image to feature points in a corresponding local region of the subsequent image; and

warping the subsequent image based on the at least one homography to generate a new aligned image that is aligned to the reference image.

2. The computer-implemented method of claim 1, wherein determining a feature descriptor for each feature point of the plurality of feature points comprises analyzing the detected feature points and generating a descriptor vector for each detected feature point that encodes intensity information, pixel location information, and local neighborhood information regarding its respective feature point.

3. The computer-implemented method of claim 1, wherein determining matching feature points between the reference image and the subsequent image comprises:

predicting for a given feature point having a pixel location in the reference image a corresponding pixel location in the subsequent image;

identifying a set of potentially matching feature points from the subsequent image;

filtering, from the set of potentially matching feature points, any feature points having a descriptor distance above a threshold distance from the given feature point; and

selecting a matching feature point from the set of potentially matching feature points having a highest similarity score as a match to the given feature point.

4. The computer-implemented method of claim 3, wherein identifying the set of potentially matching feature points from the subsequent image comprises comparing a feature descriptor of the given feature point with feature descriptors of feature points having a pixel location within a predetermined number of pixels of the predicted corresponding pixel location in the subsequent image.

5. The computer-implemented method of claim 4, wherein identifying the set of potentially matching feature points comprises:

sorting feature points within the predetermined number of pixels of the predicted corresponding pixel location according to the feature points'"'"' similarities with the given feature point; and

filtering, from the sorted feature points, all but a predetermined number of most similar feature points.

6. The computer-implemented method of claim 5, wherein filtering, from the sorted feature points, all but the predetermined number of most similar feature points comprises filtering, from the sorted feature points, all but a three most similar feature points.

7. The computer-implemented method of claim 5, wherein sorting feature points within the predetermined number of pixels of the predicted corresponding pixel location according to the feature points'"'"' similarities with the given feature point comprises sorting the feature points according to a Hamming distance between the feature descriptors of the feature points and the feature descriptor of the given feature point.

8. The computer-implemented method of claim 3, determining matching feature points between the reference image and the subsequent image further comprises:

determining, from the set of potentially matching feature points, a median displacement vector from the pixel location of the given feature point; and

filtering, from the set of potentially matching feature points, feature points having a displacement that deviates from the median displacement vector by more than a predetermined number of pixels.

9. The computer-implemented method of claim 8, wherein warping the subsequent image based on the at least one homography to generate a new aligned image that is aligned to the reference image comprises warping the subsequent image based on the homography of each local region.

10. A system for generating aligned images from burst images, comprising:

a memory comprising a references image and a subsequent image;

at least one processor communicatively coupled to the memory and storing instructions thereon, that, when executed by the at least one processor, cause the system to;

subdivide the reference image into a plurality of local regions;

subdivide the subsequent image into a plurality of corresponding local regions;

detect a plurality of feature points in the local regions of the reference image;

detect a plurality of feature points the corresponding local regions of the subsequent image;

determine matching feature points between the reference image and the subsequent image;

merge local regions of the reference image that do not include a threshold number of feature points with another local region of the reference image;

based on the matching feature points between the reference image and the subsequent image, determine a homography for each local region and merged region of the reference image, the homography of each local region mapping feature points from a respective local region of the reference image to a respective corresponding local region of the subsequent image, and

warp the subsequent image based on the homography of each local region and merged local region to generate a new aligned image that is aligned to the reference image.

11. The system as recited in claim 10, wherein the instructions when executed by the at least one processor cause the system to determine a homography for each local region of the reference image by performing steps comprising:

determining whether each local region of the plurality of local regions of the reference image includes at least a threshold number of feature points;

if a given local region of the reference image includes at least the threshold number of feature points, determining a homography of the given local region; and

if a given local region of the reference image does not include at least the threshold number of feature points, merging the given local region with an additional local region of the reference image to form a merged local region and determining the homography of the merged local region.

12. The system as recited in claim 11, wherein the instructions when executed by the at least one processor cause the system to determine the homography of the merged local region by performing steps comprising fitting a robust homography motion model to the merged local region.

13. The system as recited in claim 11, wherein the instructions when executed by the at least one processor cause the system to warp the subsequent image by interpolating homography flow vectors of the homographies that are proximate to borders shared by two or more of the corresponding local regions of the subsequent image.

14. The system as recited in claim 10, further comprising instructions that when executed by the at least one processor cause the system to:

determine whether corners of the new aligned image corresponding to contiguous corners of the plurality of local regions of the subsequent image are contiguous;

if the corners of the new aligned image are noncontiguous, determine average coordinates of the noncontiguous corners of the new aligned image; and

determine a homography of each of the local regions of the plurality of local regions of the reference image utilizing the average coordinates as coordinates of the contiguous corners.

15. A computer-implemented method of generating aligned images from burst images, the method comprising:

subdividing a reference image and a subsequent image into a plurality of local regions;

detecting a plurality of feature points in the local regions of the reference image and the subsequent image;

determining matching pairs of feature points between the reference image and the subsequent image;

based on the matching pairs of feature points, performing a step for determining a homography for each local region of the plurality of local regions, the homography of each local region mapping feature points from a local region of the reference image to a corresponding local region of the subsequent image; and

warping the subsequent image based on the homography of each local region of the plurality of local regions to generate a new aligned image that is aligned to the reference image.

16. The computer-implemented method of claim 15, wherein determining matching pairs of feature points between the reference image and the subsequent image comprises determining feature points that are most similar to each other based at least partially on a Hamming distance between feature descriptors of the feature points.

17. The computer-implemented method of claim 15, wherein subdividing each of a reference image and a subsequent image comprises subdividing the reference image into at least four local regions and the subsequent image into at least four corresponding local regions.

18. The computer-implemented method of claim 17, wherein subdividing the reference image into at least four local regions and the subsequent image into at least four corresponding local regions comprises subdividing the reference image and the subsequent image using a 2×

2 spatial grid.

19. The computer-implemented method of claim 15, wherein detecting a plurality of feature points comprises selecting a top number of features points having highest confidence values from each local region of the plurality of local regions.