Metadata based alignment of distorted images

• US 8,830,347 B2
• Filed: 02/01/2013
• Issued: 09/09/2014
• Est. Priority Date: 08/29/2008
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Jin, Hailin
• Primary Examiner(s): Khan, Usman
• Application Number:
  US13/757,518
• Publication Number:
  US 20130142431A1
• Time in Patent Office: 585 Days
• Field of Search: 348/345, 348/222.1, 382/190, 382/215, 382/254, 382/288
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

A method for aligning and unwarping distorted images in which lens profiles for a variety of lens and camera combinations are precomputed. Metadata stored with images is used to automatically determine if a set of component images include an excessive amount of distortion, and if so the metadata is used to determine an appropriate lens profile and initial unwarping function. The initial unwarping function is applied to the coordinates of feature points of the component images to generate substantially rectilinear feature points, which are used to estimate focal lengths, centers, and relative rotations for pairs of the images. A global nonlinear optimization is applied to the initial unwarping function(s) and the relative rotations to generate optimized unwarping functions and rotations for the component images. The optimized unwarping functions and rotations may be used to render a panoramic image.

Claims

1. A computer-implemented method, comprising:

automatically selecting a particular profile for a set of distorted component images from among a plurality of profiles according to metadata corresponding to the set of distorted component images, wherein each of the plurality of profiles stores information for a particular camera/lens combination including corresponding pre-calculated unwarping functions each operable to detect feature points in individual images associated with a corresponding camera/lens combination, match detected feature points one to another to establish feature correspondences between the individual images, and generate substantially rectilinear feature point coordinates for unwarping according to the matching;

determining a pre-calculated unwarping function for the set of distorted component images that is stored in the particular profile; and

applying the pre-calculated unwarping function to generate a composite image from the set of distorted component images by joining the distorted component images one to another using the rectilinear feature point coordinates generated using the pre-calculated unwarping function.

2. The computer-implemented method as recited in claim 1, further comprising generating a set of feature points for the set of distorted component images used to join the distorted component images one to another.

3. The computer-implemented method as recited in claim 2, wherein generating the set of feature points comprises:

recognizing feature points within individual images of the set of distorted component images; and

establishing feature correspondences between feature points recognized in the set of distorted component images.

4. The computer-implemented method as recited in claim 2, wherein the feature points comprises substantially rectilinear features.

5. The computer-implemented method as recited in claim 1, wherein the plurality of profiles is stored according to a markup language format.

6. The computer-implemented method as recited in claim 5, wherein the markup language is extensible markup language (XML).

7. The computer-implemented method as recited in claim 1, wherein the pre-calculated unwarping function is configured to align and unwarp the distorted component images to form the composite image.

8. The computer-implemented method as recited in claim 1, wherein the composite image is a panoramic image.

9. The computer-implemented method as recited in claim 1, further comprising determining that a first image of the set of distorted component images was captured with a different camera/lens combination than a second image of the set of distorted component images and selecting different profiles from the plurality of profiles for use with the first image and the second image.

10. The computer-implemented method as recited in claim 1, further comprising determining that at least one image of the set of distorted component images does not match one of the plurality of profiles and providing an option to enable a user to select a profile from among the plurality of profiles to use for the at least one image.

11. A computing system comprising:

one or more processing devices;

memory storing program instructions executable via the one or more processing devices to perform operations comprising;

automatically selecting a particular profile for a set of distorted component images from among a plurality of profiles according to metadata corresponding to the set of distorted component images, wherein each of the plurality of profiles stores information for a particular camera/lens combination and contains corresponding pre-calculated unwarping functions;

obtaining a pre-calculated unwarping function for the set of distorted component images that is contained in the particular profile, the pre-calculated unwarping function configured to detect feature points in individual images of the distorted component images, match detected feature points one to another to establish feature correspondences between the set of the distorted component images, and generate substantially rectilinear feature point coordinates for unwarping according to the matching; and

joining the distorted component images one to another to generate a composite image by applying the pre-calculated unwarping function to the set of distorted component images.

12. The computing system as recited in claim 11, wherein joining the distorted component images comprises generating a set of feature points for the set of component images used to join the distorted component images one to another.

13. The computing system as recited in claim 12, wherein generating the set of feature points comprises:

recognizing feature points within individual images of the set of distorted component images; and

establishing feature correspondences between feature points recognized in the set of distorted component images.

14. The computing system as recited in claim 12, wherein the feature points comprises substantially rectilinear features.

15. The computing system as recited in claim 12, wherein the pre-calculated unwarping function is applied to align and unwarp the distorted component images to form the composite image based upon the set of feature points.

16. One or more computer-readable storage media other than a signal per se storing program instructions executable via a computing device to perform operations comprising:

automatically selecting a particular profile for a set of distorted component images from among a plurality of profiles according to metadata corresponding to the set of distorted component images, wherein each of the plurality of profiles stores a pre-calculated unwarping function for a particular camera/lens combination;

obtaining the pre-calculated unwarping function for the set of distorted component images from the particular profile; and

generating a set of substantially rectilinear feature points for the set of distorted component images used to align the distorted component images one to another; and

applying the pre-calculated unwarping function to the set of distorted component images to generate a composite image, the pre-calculated unwarping function configured to unwarp the distorted component images and align the distorted component images based upon the set of substantially rectilinear feature points.

17. One or more computer-readable storage media as recited in claim 16, wherein generating the set of substantially rectilinear feature points comprises:

recognizing feature points within individual images of the set of distorted component images; and

establishing feature correspondences between feature points recognized in the set of distorted component images.

18. One or more computer-readable storage media as recited in claim 17, wherein applying the pre-calculated unwarping function comprises performing pair-wise processing to generate optimized rotations and optimized unwarping functions based on the substantially rectilinear feature points, the optimized rotations and optimized unwarping functions used to align, unwarp, and stitch the distorted component images.

19. One or more computer-readable storage media as recited in claim 16, wherein the composite image is a panoramic image.

20. One or more computer-readable storage media as recited in claim 16, wherein the operations further comprise:

determining that a first image of the set of distorted component images was captured with a different camera/lens combination than a second image of the set of distorted component images and selecting different profiles from the plurality of profiles for use with the first image and the second image.