System and method of aligning images

• US 7,756,358 B2
• Filed: 11/30/2004
• Issued: 07/13/2010
• Est. Priority Date: 11/30/2004
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Deng, Yining, Silverstein, D. Amnon
• Primary Examiner(s): Tung; Kee M
• Assistant Examiner(s): AMIN, JWALANT B
• Application Number:
  US11/000,660
• Publication Number:
  US 20060115181A1
• Time in Patent Office: 2,051 Days
• Field of Search: None
• US Class Current: 382/294
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 3/147 using affine transformations
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration

Abstract

A method performed by a processing system. The method includes receiving a sequence of frames, each frame comprising a sequence of at least two images, each image at a different image position in the sequence and each pair of adjacent images having an overlap region. A local alignment value is determined for the images of each pair of adjacent image positions of each frame based on minimizing a difference of a desired parameter between the images in the overlap region. Global alignment values are determined for all images at corresponding pairs of adjacent image positions of each frame based on the local alignment values for the images at the corresponding pair of adjacent image positions of each frame. Each global alignment value is applied to at least one image of the images of the corresponding pair of adjacent image positions of each frame.

Claims

1. A method comprising:

receiving, with a processing system, a sequence of frames, the frames together forming a shot, each frame comprising a sequence of at least two image positions with an image acquired from a corresponding image capturing device at each image position, wherein the images corresponding to each pair of adjacent image positions have an overlap region;

determining, with a processor of the imaging system, a set of local alignment values for each frame of a selected group of at least two frames of the shot, each set of local alignment values including one local alignment value for each pair of adjacent image positions of the frame which is based on minimizing a difference of a desired parameter between the corresponding images in the overlap region;

determining, with the processor, a single set of global alignment values for the shot, the set of global alignment values including one global alignment value for each pair of same adjacent image positions of all frames of the shot, wherein each global alignment value is based on the local alignment value for the corresponding pair of adjacent image positions of all sets of local alignment values; and

applying, with the processing system, each global alignment value to at least one of the images at the corresponding pair of adjacent image positions of every frame of the shot to adjust a position of the at least one image relative to the other image.

2. The method of claim 1, further including:

determining a certainty factor for each local alignment value based on at least one parameter of the images in the overlap region, and wherein determining each of the global alignment values is based on the local alignment values and certainty factors for the images at the corresponding pair of adjacent image positions of the selected frames.

3. The method of claim 2, wherein the global alignment values comprise a weighted average of the local alignment values for the images at the corresponding pair of adjacent images positions of the selected frames, wherein each local alignment value is weighted based on the corresponding certainty factor.

4. The method of claim 1, wherein the selected frames comprise all frames of the sequence of frames.

5. The method of claim 1, wherein the desired parameter for determining the local alignment value comprises values associated with one or more color channels of pixels in the overlap region of each image.

6. The method of claim 1, wherein the desired parameter for determining the local alignment value comprises an intensity of pixels in the overlap region of each image.

7. The method of claim 1, wherein determining a local alignment value for the images of each pair of adjacent image positions includes applying a transform model to the overlap region of the images.

8. The method of claim 7, where the transform model comprises a 2D translational model.

9. The method of claim 7, wherein the transform model comprises a 2D Affine model.

10. The method of claim 7, wherein determining the certainty factor for each local alignment value is based at least partially on a magnitude of the difference of the desired parameter between the corresponding images in the overlap region.

11. The method of claim 1, wherein determining the certainty factor for each local alignment value is based at least partially on an amount of high frequency content of the overlap region of the corresponding images.

12. The method of claim 1, wherein determination of the local alignment values is made relative to a selected reference image in each frame.

13. The method of claim 12, wherein the selected reference image of each frame is at an image position approximately at a center of the sequence of images.

14. The method of claim 12, wherein the selected reference image is at a same image position in each frame.

15. The method of claim 12, wherein the selected reference image comprises an image at a center position of the sequence when each frame comprises three images, with one image being at a right image position and one image being at a left image position relative to the selected reference position at the center position.

16. The method of claim 1, wherein prior to determining local alignment values, the method further comprises:

correcting the image at each image position of each frame for perspective differences based on a viewing angle of an image capture device corresponding to each image position and employed to provide the images at the corresponding image position of each frame.

17. The method of claim 16, wherein correcting each frame for perspective differences includes applying a geometric progression to warp each of the images of a frame to a common sphere.

18. The method of claim 1, further comprises:

receiving a sequence of shots, each shot comprising a sequence of frames; and

determining boundaries between each shot based on a parameter of the images of the frames of each shot.

19. The method of claim 1, wherein determining boundaries is based on a difference between average intensities of pixels comprising the images of each frame of each shot.

20. The method of claim 1, wherein the selected parameter comprises an intensity of pixels in the overlap region.

21. A system comprising:

a memory system including;

a local alignment module;

a global alignment module; and

a combining module; and

a processor, wherein upon the system receiving a shot comprising a sequence of frames, each frame comprising a sequence of at least two image positions with an image at each image position, wherein the images at each pair of adjacent image positions have an overlap region, the processor is configured to;

execute the local alignment module to determine a set of local alignment values for each frame of at least two selected frames of the shot, each set of local alignment values including one local alignment values for each pair of adjacent image positions of the frame which is based on minimizing a difference of a desired parameter between the corresponding images in the overlap region;

execute the global alignment module to determine a single set of global alignment values for the shot, the set of global alignment values including one global alignment value for each pair of same adjacent image positions of all frames of the shot, wherein each global alignment value is based on the local alignment value for the corresponding pair of adjacent image positions of all sets of local alignment values; and

execute the combining module to apply each global alignment value to at least one of the images at the corresponding pair of adjacent image positions of each frame.

22. The system of claim 21, wherein the memory system further includes a certainty estimating module which is executed by the processor, prior to executing the global alignment module to determine a certainty factor for each local alignment value based on at least one parameter of the images in the overlap region, and wherein determining each of the global alignment values is based on the local alignment values and certainty factors for the images at the corresponding pair of adjacent image positions of the selected frames.

23. The system of claim 21, wherein the memory system further includes a perspective correction module which is executed by the processor, prior to execution of the local alignment module, to correct the image at each image position of each frame for perspective differences based on a viewing angle of an image capture device corresponding to each image position and employed to provide the images at the corresponding image position of each frame.

24. The system of claim 21, wherein the memory system further includes a boundary detection module, wherein subsequent to the system receiving a sequence of shots, each shot comprising a sequence of frames, the processor executes the boundary detection module to determine boundaries between each shot based on a parameter of the images of the frames of each shot.

25. A method performed by a processing system on a sequence of frames forming a shot, each frame comprising a sequence of at least two image positions with an image acquired from a corresponding image capturing device at each image position in the sequence, wherein the images at each pair of adjacent image positions have an overlap region, the method comprising:

determining, with a processor of the processing system, a set of local alignment values for each frame of a selected number of frames of the shot, each set of local alignment values having a local alignment value for the images at each pair of adjacent image positions of the frame;

determining, with the processor, a single set of global alignment values for all frames of the shot, one global alignment value for the same pair of adjacent image positions of each frame and which is based on the local alignment values of the corresponding pair of adjacent image positions of the selected frames; and

applying, with the processor, each global alignment value to at least one of the images of the corresponding pair of adjacent image positions of each frame, to adjust a position of the at least one image relative to the other image.

26. A system comprising:

an input/output unit to receive a sequence of frames forming a shot, each frame comprising at least two images, each image at a different image position in the sequence and acquires from a corresponding image capturing device, wherein each pair of adjacent images having an overlap region

a local alignment module executed by a processor of the system to determine for each frame of a selected number of frames of the shot, a set of local alignment values for the images of each pair of adjacent image positions of the frame;

a global alignment module executed by the processor to determine a single set of global alignment values for all frames of the shot including one global alignment value each of the same pairs of adjacent image positions of all frames which is based on each of the local alignment values for the corresponding pair of adjacent image positions of the selected frames; and

a combining module executed by the processor to apply each global alignment value to at least one image of the images at the corresponding pair of adjacent image positions of each frame to adjust a position of the at least one image to the other image.

27. A computer-readable medium including instructions executable by a processing system for performing a method comprising:

receiving a sequence of frames forming a shot, each frame comprising a sequence of at least two images, each image at a different image position in the sequence with the images at each pair of adjacent image positions having an overlap region;

determining a set of local alignment values for each frame of a plurality of selected frames of the shot, each set of local alignment values including one local alignment value for the images of each pair of adjacent image positions of the selected frame;

determining a single set of global alignment values for all frames of the shot, one global alignment value for the same pair of adjacent image positions of each frame; and

applying each global alignment value to at least one of the images of the corresponding pair of adjacent image positions of each frame.

28. A method comprising:

receiving, with a processing system, a sequence of p frames, the p frames together forming a shot, each of the p frames comprising a sequence of n images at n image positions acquired from a corresponding image capturing device for each image position, wherein the images at each of n−

1 pairs of adjacent image positions have an overlap region;

determining, with a processor of the imaging system, p sets of local alignment values, one for each of the p frames, each of the p sets of local alignment values including n−

1 local alignment values, one local alignment value for the images at each of the n−

1 pairs of adjacent image positions, wherein each of the n−

1 local alignment values is based on minimizing a difference of a selected parameter in the overlap region between the images at the corresponding pair of adjacent image positions;

determining, with the processor, one set of global alignment values for the shot, the set of global alignment values including n−

1 global alignment values, one global alignment value for each of the n−

1 pairs of adjacent image positions of all p frames of the shot, wherein each of the n−

1 global alignment values is based on the p local alignment values of corresponding pair of adjacent image positions of all sets of local alignment values; and

applying, with the processing system, each of the n−

1 global alignment values to one of the images at the corresponding pair of adjacent image positions of each of the p frames to adjust a position of the one image relative to the other image.

29. The method of claim 28, wherein n and p are integer values greater than one.

30. The method of claim 28, wherein the selected parameter comprises values associated with one or more color channels of pixels in the overlap region.