Image registration methods for still image stabilization

• US 9,384,552 B2
• Filed: 06/06/2013
• Issued: 07/05/2016
• Est. Priority Date: 06/06/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Zhou, Jianping, Hubel, Paul, Tico, Marius, Nariani Schulze, Anita, Toft, Rolf
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): PATEL, PINALBEN V
• Application Number:
  US13/911,793
• Publication Number:
  US 20140363096A1
• Time in Patent Office: 1,125 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

Systems, methods, and computer readable media to improve image stabilization operations are described. A novel approach to pixel-based registration of non-reference images to a reference frame in a set of commonly captured images is disclosed which makes use of pyramid decomposition to more efficiently detect corners. The disclosed pixel-based registration operation may also be combined with motion sensor data-based registration approaches to register non-reference images with respect to the reference frame. When the registered non-reference images are combined with the pre-selected reference image, the resulting image is a quality stabilized image.

Claims

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause the programmable control device to:

obtain a first image and a first one or more additional images;

obtain motion information corresponding to each of the first one or more additional images;

compensate for motion between the first image and each of the first one or more additional images based, at least in part, on each image'"'"'s corresponding motion information to generate a second one or more additional images, by;

determining a transform matrix for each of a plurality of segments of each of the first one or more additional images based, at least in part, on the motion information, and

applying the transform matrix for each of the first one or more additional images to the corresponding first one or more additional image,

wherein each of the second one or more additional images has a corresponding image in the first one or more additional images, and wherein each segment corresponds to a plurality of pixel rows in an image; and

register the second one or more additional images with respect to the first image based, at least in part, on a pixel-level analysis of the first image and each of the second one or more additional images to generate an output image.

2. The non-transitory program storage device of claim 1, further comprising instructions to cause the one or more programmable control devices to identify the first image as a reference image.

3. The non-transitory program storage device of claim 1, wherein the motion information comprises motion information of each of the first one or more additional images with respect to the first image.

4. The non-transitory program storage device of claim 1, wherein the instructions to cause the programmable control device to register the second one or more additional images based, at least in part, on a pixel-level analysis of the first image and one of the second one or more additional images comprise instructions to cause the programmable control device to:

apply pyramid decomposition to the first image and one of the second one or more additional images to generate a plurality of levels for each of the first image and the one of the second one or more additional images;

detect corners at each level of the first image and for each detected corner, wherein the instructions to cause the programmable control device to detect corners comprise instructions to cause the programmable control device to;

search for a corresponding corner in a corresponding level of the one of the second one or more additional images;

generate a transformation matrix based on locations of the corresponding corners found; and

apply the transformation matrix to the one of the second one or more additional images.

5. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause the programmable control device to:

obtain a first image and a first one or more additional images;

obtain motion information corresponding to each of the first one or more additional images;

determine a transform matrix for each of a plurality of segments of each of the first one or more additional images based, at least in part, on the motion information, wherein each segment corresponds to a plurality of pixel rows in an image;

apply the transform matrix for each of the first one or more additional images to the corresponding first one or more additional image; and

register the first one or more additional images with respect to the first image based, at least in part, on a pixel-level analysis of the first image, the first one or more additional images and the transform matrices for each of the first one or more additional images to generate an output image.

6. The non-transitory program storage device of claim 5, wherein the motion information comprises gyroscope data.

7. The non-transitory program storage device of claim 5, wherein the motion information comprises of motion of each of the first one or more additional images with respect to the first image.

8. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause the programmable control device to:

obtain a first image having a plurality of pixels arranged in rows;

obtain a first one or more additional images, wherein a plurality of pixels in the first image have corresponding pixels in each of the first one or more additional images;

obtain motion information corresponding to each of the first one or more additional images;

determine a first transform matrix for each segment of each of the first one or more additional images based, at least in part, on the motion information, wherein each segment corresponds to a one or more rows of pixels;

compensate for motion between the first image and each of the first one or more additional images based, at least in part, on each image'"'"'s corresponding motion information to generate a second one or more additional images, wherein each of the second one or more additional images has a corresponding image in the first one or more additional images;

determine a second transform matrix for each of the first one or more additional images based, at least in part, on a pixel-level analysis of the first image and the second one or more additional images;

combine each image'"'"'s first transform matrices and the corresponding second transform matrices to generate a transformation matrix for each segment of each of the first one or more additional images;

apply the transformation matrices to corresponding segments of the first one or more additional images to generated one or more transformed segments for each of the first one or more additional images; and

combine the transformed segments for each of the first one or more additional images to register the first one or more additional images.

9. The non-transitory program storage device of claim 8, further comprising instructions to cause the one or more programmable control devices to identify the first image as a reference image.

10. The non-transitory program storage device of claim 8, wherein the motion information comprises gyroscope data.

11. The non-transitory program storage device of claim 10, wherein the motion information comprises motion information of each of the first one or more additional images with respect to the first image.

12. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause the programmable control device to:

obtain a first and a second images;

obtain motion information corresponding to the first and second images;

compensate for motion between the first and second images based, at least in part, on the motion information corresponding to the first and second images, wherein the instructions to compensate for motion between the first and second images further comprise instructions to;

apply pyramid decomposition on each of the first and the second images to generate a plurality of levels for each image;

detect corners at each level of the first image;

for each corner detected at each of level the first image, search for a corresponding corner in a corresponding level of the second image; and

generate a transformation matrix for each corresponding corner identified based on locations of the corresponding corners; and

apply the transformation matrices to the second image to register the second image, wherein the second image is registered based, at least in part, on a pixel-level analysis of the first image and the second image.

13. The non-transitory program storage device of claim 12, wherein the instructions to cause the programmable control device to applying pyramid decomposition comprises applying a low pass filter and down sampling the image two or more times to generate a plurality of levels.

14. The non-transitory program storage device of claim 12, wherein a location of corresponding corners identified at a lower level is used as an initial guess for identifying corresponding corners in a subsequent level.

15. The non-transitory program storage device of claim 12, further comprising instructions to cause the one or more programmable control devices to identify and remove outliers in the corresponding corners.