Metadata-driven method and apparatus for multi-image processing

• US 8,724,007 B2
• Filed: 10/14/2008
• Issued: 05/13/2014
• Est. Priority Date: 08/29/2008
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Chen, Simon, Chien, Jen-Chan, Jin, Hailin
• Primary Examiner(s): Khan, Usman
• Application Number:
  US12/251,253
• Publication Number:
  US 20130124471A1
• Time in Patent Office: 2,037 Days
• Field of Search: 348/345
• US Class Current: 348/345
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Methods and apparatus for processing collections of images are described in which metadata from a set of images may be used in directing a multi-image processing workflow. One or more output images may be rendered from a set of input images, with each output image being some combination of two or more of the input images. To render an output image, a workflow including one or more processing steps may be applied to the images. Metadata may be used in directing and performing the workflow. For example, metadata may be used in determining a particular workflow for a set of images. As another example, metadata may be used to sort a collection of images into multiple categories for automated workflow processing. As yet another example, metadata may be used to retrieve information stored in a profile database that may be used in processing the images.

Claims

1. A computer-implemented method, comprising:

performing, by one or more computing devices;

examining metadata corresponding to each image in a collection of input images to determine information indicating how the respective input image was captured;

estimating an image center for the respective input image based at least in part on unwarping functions that utilize the determined information and coordinates of matching feature points in two or more of the input images;

directing workflow processing of the collection of input images according to the determined information for each of the input images including the respective estimated image center; and

generating one or more output images from the collection of input images according to the workflow processing.

2. The computer-implemented method as recited in claim 1, wherein the information indicating how a given input image was captured includes camera make, camera model, and one or more lens characteristics, wherein the one or more lens characteristics include one or more of focal length, F number, or lens type.

3. The computer-implemented method as recited in claim 1, wherein said workflow processing further comprises modifying one or more of the input images according to one or more image processing techniques to generate one or more modified images, wherein each of the one or more modified images is generated from a respective single input image.

4. The computer-implemented method as recited in claim 1, wherein the one or more image processing techniques include one or more of a vignette removal process to reduce or remove vignetting in an image, an unwarping process to reduce or remove lens distortion in an image, a chromatic aberration removal process to reduce or remove chromatic aberration in an image, or a sensor noise removal process to reduce or remove sensor noise in an image.

5. The computer-implemented method as recited in claim 1, wherein the information indicating how a given input image was captured includes indications of one or more conditions under which the given input image was captured, wherein the one or more conditions under which the given input image was captured include one or more particular camera and lens settings used when capturing the given input image.

6. The computer-implemented method as recited in claim 1, wherein the information indicating how a given input image was captured includes one or more characteristics of a camera, a lens, or a camera/lens combination used to capture the given input image.

7. The computer-implemented method as recited in claim 1, wherein the information indicating how a given input image was captured includes one or more of geospatial information or camera orientation information.

8. The computer-implemented method as recited in claim 3, wherein the one or more output images include at least one of the one or more modified images.

9. The computer-implemented method as recited in claim 3, wherein said combining image data from two or more of the input images to generate a single output image comprises combining two or more of the modified images to generate a respective output image.

10. The computer-implemented method as recited in claim 1, wherein said workflow processing comprises two or more different workflows, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images comprises:

determining, from the information for each of the input images, an appropriate one of the two or more workflows for at least a portion of the collection of input images; and

processing the at least a portion of the collection of input images according to the determined workflow.

11. The computer-implemented method as recited in claim 1, wherein said workflow processing comprises two or more different workflows, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images comprises:

determining, from the information for each of the input images, a recommended workflow from the two or more workflows for at least a portion of the collection of input images;

displaying the recommended workflow for the at least a portion of the collection of input images;

receiving user input accepting or rejecting the recommended workflow; and

if the recommended workflow is accepted, processing the at least a portion of the collection of input images according to the recommended workflow.

12. The computer-implemented method as recited in claim 1, wherein said workflow processing comprises two or more different workflows, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images comprises:

classifying each image in the collection of input images into one of two or more categories according to the information for each of the input images, wherein each category corresponds to a particular one of the two or more workflows; and

processing the input images in each category according to the corresponding workflow.

13. The computer-implemented method as recited in claim 1, wherein said workflow processing comprises two or more different workflows, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images comprises:

classifying the collection of input images into one or more Image subsets according to the information for each of the input images;

processing the input images in each image subset according to a first workflow to generate a set of intermediate images;

classifying the set of intermediate images into one or more intermediate image subsets according to the information for each of the input images; and

processing the intermediate images in each intermediate image subset according to a second workflow.

14. The computer-implemented method as recited in claim 1, wherein the collection of input images includes subsets of images captured using different camera/lens combinations.

15. The computer-implemented method as recited in claim 1, wherein the collection of input images includes subsets of images captured using different camera and lens settings.

16. The computer-implemented method as recited in claim 1, wherein the collection of input images includes at least one subset of images captured using a high dynamic range (HDR) image capture technique, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images comprises:

detecting the at least one subset of images captured using the HDR image capture technique according to the information for each of the input images; and

applying HDR processing to the at least one subset of images captured using the HDR image capture technique in response to said detecting.

17. The computer-implemented method as recited in claim 1, wherein the collection of input images includes at least one subset of images captured using a time-lapse photography technique, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images comprises:

detecting the at least one subset of images captured using the time-lapse photography technique according to the information for each of the input images; and

applying time-lapse processing to the at least one subset of images captured using the time-lapse image capture technique in response to said detecting.

18. The computer-implemented method as recited in claim 1, wherein the collection of input images includes at least one subset of images captured using a panoramic image capture technique, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images comprises:

detecting the at least one subset of images captured using the panoramic image capture technique according to the information for each of the input images; and

applying panoramic image stitching to the at least one subset of images captured using the panoramic image capture technique in response to said detecting.

19. The computer-implemented method as recited in claim 1, wherein the collection of input images includes at least one subset of images captured using a combination of two or more of a high dynamic range (HDR) image capture technique, a time-lapse photography technique, and a panoramic image capture technique, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images comprises:

detecting the at least one subset of images captured using the combination of two or more of said techniques according to the information for each of the input images; and

applying two or more of HDR processing, time-lapse processing, and panoramic image stitching to the at least one subset of images captured using the combination of two or more of said techniques in response to said detecting.

20. The computer-implemented method as recited in claim 19, wherein said generating one or more output images from the collection of input images according to the workflow processing comprises combining results from said applying two or more of HDR processing, time-lapse processing, and panoramic image stitching to generate the one or more output images.

21. The computer-implemented method as recited in claim 1, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images is performed automatically without user interaction.

22. The computer-implemented method as recited in claim 1, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images comprises:

displaying one or more workflow processing recommendations via a user interface; and

receiving user input accepting or rejecting at least one of the one or more workflow processing recommendations.

23. The computer-implemented method as recited in claim 1, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images comprises:

retrieving additional information about a particular camera/lens combination from a profile database, wherein at least a portion of the information determined from the metadata is used to locate the particular camera/lens combination in the profile database; and

applying the additional information during the workflow processing of at least a subset of the collection of input images captured with the particular camera/lens combination.

24. A system, comprising:

at least one processor; and

a memory comprising program instructions, the program instructions executable by the at least one processor to;

examine metadata corresponding to each image in a collection of input images to determine information indicating how the respective input image was captured;

estimate an image center for the respective input image in a width direction normalized by an image width and based at least in part on unwarping functions that utilize the determined information and coordinates of matching feature points in two or more of the input images;

estimate an image center for the respective image in a height direction normalized by an image height and based at least in part on unwarping functions that utilize the determined information and coordinates of matching feature points in two or more of the input images;

direct workflow processing of the collection of input images according to the determined information for each of the input images including the respective estimated image center in the width direction and the respective estimated image center in the height direction; and

generate one or more output images from the collection of input images according to the workflow processing.

25. The system as recited in claim 24, wherein the information indicating how a given input image was captured includes camera make, camera model, and one or more lens characteristics, wherein the one or more lens characteristics include focal length, F number, and lens type.

26. The system as recited in claim 24, wherein said workflow processing further comprises modifying one or more of the input images according to one or more image processing techniques to generate one or more modified images, wherein each of the one or more modified images is generated from a respective single input image, wherein the one or more image processing techniques include one or more of a vignette removal process to reduce or remove vignetting in an image, an unwarping process to reduce or remove lens distortion in an image, a chromatic aberration removal process to reduce or remove chromatic aberration in an image, or a sensor noise removal process to reduce or remove sensor noise in an image.

27. The system as recited in claim 24, wherein the information indicating how a given input image was captured includes indications of one or more conditions under which the given input image was captured, wherein the one or more conditions under which the given input image was captured include one or more particular camera and lens settings used when capturing the given input image.

28. The system as recited in claim 24, wherein the information indicating how a given input image was captured includes one or more characteristics of a camera, a lens, or a camera/lens combination used to capture the given input image.

29. The system as recited in claim 24, wherein the information indicating how a given input image was captured includes one or more of geospatial information or camera orientation information.

30. The system as recited in claim 26, wherein the one or more output images include at least one of the one or more modified images.

31. The system as recited in claim 24, wherein said workflow processing comprises two or more different workflows, wherein, to direct workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are executable by the at least one processor to:

determine, from the information for each of the input images, an appropriate one of the two or more workflows for at least a portion of the collection of input images; and

process the at least a portion of the collection of input images according to the determined workflow.

32. The system as recited in claim 24, wherein said workflow processing comprises two or more different workflows, wherein, to direct workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are executable by the at least one processor to:

determine, from the information for each of the input images, a recommended workflow from the two or more workflows for at least a portion of the collection of input images;

display the recommended workflow for the at least a portion of the collection of input images;

receive user input accepting or rejecting the recommended workflow; and

if the recommended workflow is accepted, process the at least a portion of the collection of input images according to the recommended workflow.

33. The system as recited in claim 24, wherein said workflow processing comprises two or more different workflows, wherein, to direct workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are executable by the at least one processor to:

classify each image in the collection of input images into one of two or more categories according to the information for each of the input images, wherein each category corresponds to a particular one of the two or more workflows; and

process the input images in each category according to the corresponding workflow.

34. The system as recited in claim 24, wherein said workflow processing comprises two or more different workflows, wherein, to direct workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are executable by the at least one processor to:

classify the collection of input images into one or more image subsets according to the information for each of the input images;

process the input images in each image subset according to a first workflow to generate a set of intermediate images;

classify the set of intermediate images into one or more intermediate image subsets according to the information for each of the input images; and

process the intermediate images in each intermediate image subset according to a second workflow.

35. The system as recited in claim 24, wherein the collection of input images includes subsets of images captured using different camera/lens combinations and subsets of images captured using different camera and lens settings.

36. The system as recited in claim 24, wherein the collection of input images includes at least one subset of images captured using a particular image capture technique, wherein, to direct workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are executable by the at least one processor to:

detect the at least one subset of images captured using the particular image capture technique according to the information for each of the input images; and

apply particular processing specific to images captured using the particular image capture technique to the at least one subset of images captured using the particular image capture technique in response to said detecting.

37. The system as recited in claim 24, wherein the collection of input images includes at least one subset of images captured using a combination of two or more of a high dynamic range (HDR) image capture technique, a time-lapse photography technique, and a panoramic image capture technique, wherein, to direct workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are executable by the at least one processor to:

detect the at least one subset of images captured using the combination of two or more of said techniques according to the information for each of the input images; and

apply two or more of HDR processing, time-lapse processing, and panoramic image stitching to the at least one subset of images captured using the combination of two or more of said techniques in response to said detecting.

38. The system as recited in claim 37, wherein, to generate one or more output images from the collection of input images according to the workflow processing, the program instructions are executable by the at least one processor to combine results from said applying two or more of HDR processing, time-lapse processing, and panoramic image stitching to generate the one or more output images.

39. The system as recited in claim 24, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images is performed automatically without user interaction.

40. The system as recited in claim 24, wherein, to direct workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are executable by the at least one processor to:

retrieve additional information about a particular camera/lens combination from a profile database, wherein at least a portion of the information determined from the metadata is used to locate the particular camera/lens combination in the profile database; and

apply the additional information during the workflow processing of at least a subset of the collection of input images captured with the particular camera/lens combination.

41. A computer-readable storage medium, excluding signals per se, storing program instructions, the program instructions executable to implement:

examining metadata corresponding to each image in a collection of input images to determine information indicating how the respective input image was captured;

estimating an image center for the respective input image based at least in part on unwarping functions that utilize the determined information and coordinates of matching feature points in two or more of the input images;

directing workflow processing of the collection of input images according to the determined information for each of the input images including the respective estimated image center; and

generating one or more output images from the collection of input Images according to the workflow processing.

42. The computer-readable storage medium as recited in claim 41, wherein the information indicating how a given input image was captured includes camera make, camera model, and one or more lens characteristics, wherein the one or more lens characteristics include focal length, F number, and lens type.

43. The computer-readable storage medium as recited in claim 41, wherein said workflow processing further comprises modifying one or more of the input images according to one or more image processing techniques to generate one or more modified images, wherein each of the one or more modified images is generated from a respective single input image, wherein the one or more image processing techniques include one or more of a vignette removal process to reduce or remove vignetting in an image, an unwarping process to reduce or remove lens distortion in an image, a chromatic aberration removal process to reduce or remove chromatic aberration in an image, or a sensor noise removal process to reduce or remove sensor noise in an image.

44. The computer-readable storage medium as recited in claim 41, wherein the information indicating how a given input image was captured includes indications of one or more conditions under which the given input image was captured, wherein the in one or more conditions under which the given input image was captured include one or more particular camera and lens settings used when capturing the given input image.

45. The computer-readable storage medium as recited in claim 41, wherein the information indicating how a given input image was captured includes one or more characteristics of a camera, a lens, or a camera/lens combination used to capture the given input image.

46. The computer-readable storage medium as recited in claim 41, wherein the information indicating how a given input image was captured includes one or more of geospatial information or camera orientation information.

47. The computer-readable storage medium as recited in claim 43, wherein the one or more output images include at least one of the one or more modified images.

48. The computer-readable storage medium as recited in claim 41, wherein said workflow processing comprises two or more different workflows, wherein, in said directing workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are computer-executable to implement:

determining, from the information for each of the input images, an appropriate one of the two or more workflows for at least a portion of the collection of input images; and

processing the at least a portion of the collection of input images according to the determined workflow.

49. The computer-readable storage medium as recited in claim 41, wherein said workflow processing comprises two or more different workflows, wherein, in said directing workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are computer-executable to implement:

determining, from the information for each of the input images, a recommended workflow from the two or more workflows for at least a portion of the collection of input images;

displaying the recommended workflow for the at least a portion of the collection of input images;

receiving user input accepting or rejecting the recommended workflow; and

if the recommended workflow is accepted, processing the at least a portion of the collection of input images according to the recommended workflow.

50. The computer-readable storage medium as recited in claim 41, wherein said workflow processing comprises two or more different workflows, wherein, in said directing workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are computer-executable to implement:

classifying each image in the collection of input images into one of two or more categories according to the information for each of the input images, wherein each category corresponds to a particular one of the two or more workflows; and

processing the input images in each category according to the corresponding workflow.

51. The computer-readable storage medium as recited in claim 41, wherein said workflow processing comprises two or more different workflows, wherein, in said directing workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are computer-executable to implement:

classifying the collection of input images into one or more Image subsets according to the information for each of the input images;

processing the input images in each image subset according to a first workflow to generate a set of intermediate images;

classifying the set of intermediate images into one or more intermediate image subsets according to the information for each of the input images; and

processing the intermediate images in each intermediate image subset according to a second workflow.

52. The computer-readable storage medium as recited in claim 41, wherein the collection of input images includes subsets of images captured using different camera/lens combinations or subsets of images captured using different camera and lens settings.

53. The computer-readable storage medium as recited in claim 41, wherein the collection of input images includes at least one subset of images captured using a particular image capture technique, wherein, in said directing workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are computer-executable to implement:

detecting the at least one subset of images captured using the particular image capture technique according to the information for each of the input images; and

applying particular processing specific to images captured using the particular image capture technique to the at least one subset of images captured using the particular image capture technique in response to said detecting.

54. The computer-readable storage medium as recited in claim 41, wherein the collection of input images includes at least one subset of images captured using a combination of two or more of a high dynamic range (HDR) image capture technique, a time-lapse photography technique, and a panoramic image capture technique, wherein, in said directing workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are computer-executable to implement:

detecting the at least one subset of images captured using the combination of two or more of said techniques according to the information for each of the input images; and

applying two or more of HDR processing, time-lapse processing, and panoramic image stitching to the at least one subset of images captured using the combination of two or more of said techniques in response to said detecting.

55. The computer-readable storage medium as recited in claim 54, wherein, in said generating one or more output images from the collection of input images according to the workflow processing, the program instructions are computer-executable to implement combining results from said applying two or more of HDR processing, time-lapse processing, and panoramic image stitching to generate the one or more output images.

56. The computer-readable storage medium as recited in claim 41, wherein said directing workflow processing of the collection of input images according to the determined information for each of the input images is performed automatically without user interaction.

57. The computer-readable storage medium as recited in claim 41, wherein, in said directing workflow processing of the collection of input images according to the determined information for each of the input images, the program instructions are computer-executable to implement:

retrieving additional information about a particular camera/lens combination from a profile database, wherein at least a portion of the information determined from the metadata is used to locate the particular camera/lens combination in the profile database; and

applying the additional information during the workflow processing of at least a subset of the collection of input images captured with the particular camera/lens combination.