Content aware slideshows

• US 8,811,771 B2
• Filed: 11/26/2008
• Issued: 08/19/2014
• Est. Priority Date: 08/22/2008
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Shechtman, Eli, Bagon, Shai, Agarwala, Aseem O.
• Primary Examiner(s): BEKELE, MEKONEN T
• Application Number:
  US12/324,345
• Publication Number:
  US 20130124951A1
• Time in Patent Office: 2,092 Days
• Field of Search: 382/165, 382/170, 382/236, 382/243, 382/268, 382/266, 382/235, 382/173, 382/286, 382/284, 382/118, 382/293, 348/700, 348/699, 348/402.1, 348/407.1, 348/413.1, 348/416, 375/240.25, 375/240.16
• US Class Current: 382/293
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 3/10 Selection of transformation methods according to the characteristics of the input images
• CPC Class Codes: G06T 7/38 Registration of image sequences
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49

Abstract

A method, system, and computer-readable storage medium for performing content based transitions between images. Image content within each image of a set of images are analyzed to determine at least one respective characteristic metric for each image. A respective transition score for each pair of at least a subset of the images is determined with respect to each transition effect of a plurality of transition effects based on the at least one respective characteristic metric for each image. Transition effects implementing transitions between successive images for a sequence of the images are determined based on the transition scores. An indication of the determined transition effects is stored. The determined transition effects are useable to present the images in a slideshow or other image sequence presentation.

Claims

1. A computer-readable memory device storing program instructions that are computer-executable to perform operations comprising:

analyzing image content within each image of a set of images to determine at least one respective characteristic metric for each image, said analyzing including detecting whether an image includes edges or lines that are longer than a threshold length;

determining a respective transition effect for each pair of a plurality of pairs of the images based on the at least one respective characteristic metric for each image the respective transition effect useable to implement a transition between the pair of images, said determining the respective transition effect including determining a region in each image of the pair to which the respective transition effect will be applied and including, when the edges or lines longer than the threshold length are detected in the pair, determining whether the respective transition effect to be applied involves the edges or lines longer than the threshold length; and

storing an indication of the determined transition effects.

2. The computer-readable memory device of claim 1, wherein the operations further comprise:

determining a sequence of images, comprising at least a subset of the set of images and the determined transition effects; and

storing an indication of the sequence of images.

3. The computer-readable memory device of claim 2, wherein the operations further comprise:

displaying the sequence of images on a display, including transitions between the images using the determined transition effects.

4. The computer-readable memory device of claim 1, wherein said analyzing the image content comprises analyzing content metadata from each image, wherein the at least one respective characteristic metric for each image is determined at least in part based on the content metadata, and wherein the content metadata comprise one or more of:

a depth and geometric context map;

image segmentation;

an edge map;

objects;

faces;

color distribution;

feature points;

a global descriptor;

or

a saliency map.

5. The computer-readable memory device of claim 4, wherein the operations further comprise:

automatically extracting the content metadata from each image of at least a subset of the images.

6. The computer-readable memory device of claim 1, wherein the region for at least one image comprises the entire image.

7. The computer-readable memory device of claim 1, wherein said determining the respective transition effect for each pair of the plurality of pairs of the images comprises:

determining a transition score for each pair of the plurality of pairs of the images with respect to each transition effect of a plurality of transition effects based on the at least one respective characteristic metric for each image, wherein the transition score represents suitability of the transition effect to transition between the pair of images; and

determining transition effects between successive images for a sequence of the images based on the transition scores.

8. The computer-readable memory device of claim 7, wherein said determining the transition effects between successive images for the sequence of the images based on the transition scores comprises:

solving an optimization problem to determine a solution that substantially maximizes similarity over the sequence of images.

9. The computer-readable memory device of claim 8, wherein said solving the optimization problem is performed subject to one or more constraints, wherein the one or more constraints are specified with respect to one or more of:

priorities or preferences with respect to image attributes;

repetition or frequency of image attributes;

priorities or preferences regarding available transition effects;

repetition or frequency of transition effects;

or

distribution of image content.

10. The computer-readable memory device of claim 1, wherein the transition effects comprise one or more of:

an object based transition effect;

a face based transition effect;

a large segment based transition effect;

a long edge based transition effect that involves the edges or lines longer than the threshold length;

a depth discontinuity based transition effect;

a panoramic stitch based transition effect;

a local similar regions based transition effect, wherein the local similar regions comprise similar backgrounds, similar objects, or similar logos;

or

a global image based transition effect.

11. The computer-readable memory device of claim 1, wherein said determining the respective transition for each pair of the plurality of pairs of the images comprises determining one or more image operations for each of the determined transition effects to implement the transition, wherein said determining the one or more image operations comprises determining one or more of:

camera motion path;

zoom;

rotation;

morph;

uniform cross-fade;

non-uniform cross-fade;

or

synthesis of a boundary or transition region.

12. The computer-readable memory device of claim 1, wherein said determining the respective transition effect comprises determining an optimal camera motion path based on one or more of:

a saliency map for each image of the pair of images;

locations of similar regions or objects in the pair of images;

depth information for the pair of images;

or

one or more constraints regarding path smoothness and/or camera velocity at one or more points in the pair of images.

13. The computer-readable memory device of claim 1, wherein said analyzing the image content is performed as a pre-process in which said analyzing is performed for all of the images of the set of images prior to performing said determining the respective transition effect for any of the plurality of pairs of the images.

14. The computer-readable memory device of claim 1, wherein at least a portion of said analyzing the image content or said determining the respective transition is performed by a first one or more execution threads, and wherein at least another portion of said analyzing the image content or said determining the respective transition is performed by a second one or more execution threads different from the first one or more execution threads.

15. A system, comprising:

at least one processor; and

a memory coupled to the at least one processor, wherein the memory stores program instructions executable by the at least one processor to;

analyze image content within each image of a set of images to determine at least one respective characteristic metric for each image, the image content analyzed to detect whether an image includes edges or lines that are longer than a threshold length;

determine a respective transition effect for each pair of a plurality of pairs of the images based on the at least one respective characteristic metric for each image, the respective transition effect useable to implement a transition between the pair of images, to determine the respective transition effect the program instructions are executable to determine a region in each image of the pair to which the respective transition effect will be applied and, when the edges or lines longer than the threshold length are detected, determine whether the respective transition effect to be applied is to involve the edges or lines that are longer than the threshold length; and

store an indication of the determined transition effects.

16. The system of claim 15, wherein the program instructions are further executable to:

determine a sequence of images, comprising at least a subset of the set of images and the determined transition effects; and

store an indication of the sequence of images.

17. The system of claim 16, wherein the program instructions are further computer-executable to:

cause the sequence of images to be displayed on a display, including transitions between the images using the determined transition effects.

18. The system of claim 15, wherein to analyze the image content, the program instructions are executable to analyze content metadata from each image, wherein the at least one respective characteristic metric for each image is determined at least in part based on the content metadata, and wherein the content metadata comprises one or more of:

a depth and geometric context map;

image segmentation;

an edge map;

objects;

faces;

color distribution;

feature points;

a global descriptor;

or

a saliency map.

19. The system of claim 18, wherein the program instructions are further executable to automatically extract the content metadata from each image of at least a subset of the images.

20. The system of claim 15, wherein the region for at least one image comprises the entire image.

21. The system of claim 15, wherein to determine the respective transition effect for each pair of the plurality of pairs of the images the program instructions are executable to:

determine a transition score for each pair of the plurality of pairs of the images with respect to each transition effect of a plurality of transition effects based on the at least one respective characteristic metric for each image, wherein the transition score represents suitability of the transition effect to transition between the pair of images; and

determine transition effects between successive images for a sequence of the images based on the transition scores.

22. The system of claim 21, wherein to determine the transition effects between successive images for the sequence of the images based on the transition scores the program instructions are executable to:

solve an optimization problem to determine a solution that substantially maximizes similarity over the sequence of images.

23. The system of claim 22, wherein the optimization problem is solved subject to one or more constraints, wherein the one or more constraints are specified with respect to one or more of:

priorities or preferences with respect to image attributes;

repetition or frequency of image attributes;

priorities or preferences regarding available transition effects;

repetition or frequency of transition effects;

or

distribution of image content.

24. The system of claim 15, wherein the transition effects comprise one or more of:

an object based transition effect;

a face based transition effect;

a large segment based transition effect;

a long edge based transition effect that involves the edges or lines longer than the threshold length;

a depth discontinuity based transition effect; and

a panoramic stitch based transition effect;

a local similar regions based transition effect, wherein the local similar regions comprise similar backgrounds, similar objects, or similar logos;

or

a global image based transition effect.

25. The system of claim 15, wherein to determine the respective transition for each pair of the plurality of pairs of the images the program instructions are executable to determine one or more image operations for each of the determined transition effects to implement the transition, and wherein to determine the one or more image operations the program instructions are executable to determine one or more of:

camera motion path;

zoom;

rotation;

morph;

uniform cross-fade;

non-uniform cross-fade;

or

synthesis of a boundary or transition region.

26. The system of claim 15, wherein to determine the respective transition effect, the program instructions are executable to determine an optimal camera motion path based on one or more of:

a saliency map for each image of the pair of images;

locations of similar regions or objects in the pair of images;

depth information for the pair of images;

or

one or more constraints regarding path smoothness and/or camera velocity at one or more points in the pair of images.

27. The system of claim 15, wherein the image content is analyzed as a pre-process in which analysis is performed for all of the images of the set of images before the respective transition effect is determined for any of the plurality of pairs of the images.

28. The system of claim 15, wherein at least a portion of the image content is analyzed or the respective transition is determined by a first one or more execution threads, and wherein at least another portion of the image content is analyzed or the respective transition is determined by a second one or more execution threads different from the first one or more execution threads.

29. A computer-implemented method, comprising:

executing instructions on a specific apparatus so that binary digital electronic signals representing content within each image of a set of images is analyzed to determine at least one respective characteristic metric for each image, including to detect whether an image includes edges or lines that are longer than a threshold length;

executing instructions on said specific apparatus so that binary digital electronic signals representing a respective transition effect is determined for each pair of a plurality of pairs of the images based on the at least one respective characteristic metric for each image, the binary digital electronic signals representing the respective transition effect useable to implement a transition between the pair of images, to determine the respective transition effect the instructions are executed on said specific apparatus so that binary digital signals representing a region is determined in each image of the pair to which the respective transition effect will be applied and, when the edges or lines longer than the threshold length are detected, whether the respective transition effect to be applied involves the edges or lines that are longer than the threshold length is determined; and

storing an indication of the determined transition effects in a memory location of said specific apparatus for later use.

30. A computer-implemented method comprising:

analyzing image content within each image of a set of images to determine at least one respective characteristic metric for each image, said analyzing including detecting whether an image includes edges or lines that are longer than a threshold length;

determining a respective transition effect for each pair of a plurality of pairs of the images based on the at least one respective characteristic metric for each image, the respective transition effect useable to implement a transition between the pair of images, said determining the respective transition effect including determining a region in each image of the pair to which the respective transition effect will be applied and including, when the edges or lines longer than the threshold length are detected in the pair, determining whether the respective transition effect to be applied involves the edges or lines longer than the threshold length; and

storing an indication of the determined transition effects.

31. The computer-implemented method of claim 30, wherein at least one of the determined transition effects comprises:

zooming in to the region in a first image of the pair of images to which the respective transition effect will be applied;

cross fading between the region in the first image of the pair of images and the region in a second image of the pair of images to which the respective transition effect will be applied; and

zooming out from the region in the second image of the pair of images to which the respective transition effect will be applied to reveal the second image.

32. The computer-implemented method of claim 31, wherein the at least one determined transition effect further comprises panning to the region in the first image of the pair of images to which the respective transition effect will be applied.

33. The computer-implemented method of claim 30, wherein at least one of the determined transition effects that involves the edges or lines longer than the threshold length comprises:

zooming in and panning to a portion of a first image of the pair of images that includes the edges or lines longer than the threshold length;

cross fading between the portion of the first image of the pair of images that includes the edges or lines longer than the threshold length and a portion of the second image of the pair of images that includes the edges or lines longer than the threshold length; and

zooming out from the portion of the second image of the pair of images that includes the edges or lines longer than the threshold length to reveal the second image.

34. The computer-implemented method of claim 30, further comprising:

determining a sequence of images, comprising at least a subset of the set of images and the determined transition effects; and

storing an indication of the sequence of images.

35. The computer-implemented method of claim 34, further comprising displaying the sequence of images on a display, including transitions between the images using the determined transition effects.

36. The computer-implemented method of claim 30, wherein said analyzing the image content comprises analyzing content metadata from each image, wherein the at least one respective characteristic metric for each image is determined at least in part based on the content metadata, and wherein the content metadata comprise one or more of:

a depth and geometric context map;

image segmentation;

an edge map;

objects;

faces;

color distribution;

feature points;

a global descriptor;

or

a saliency map.

37. The computer-implemented method of claim 36, further comprising automatically extracting the content metadata from each image of at least a subset of the images.

38. The computer-implemented method of claim 30, wherein the region for at least one image comprises the entire image.

39. The computer-implemented method of claim 30, wherein said determining the respective transition effect for each pair of the plurality of pairs of the images comprises:

determining a transition score for each pair of the plurality of pairs of the images with respect to each transition effect of a plurality of transition effects based on the at least one respective characteristic metric for each image, wherein the transition score represents suitability of the transition effect to transition between the pair of images; and

determining transition effects between successive images for a sequence of the images based on the transition scores.

40. The computer-implemented method of claim 39, wherein said determining the transition effects between successive images for the sequence of the images based on the transition scores comprises solving an optimization problem to determine a solution that substantially maximizes similarity over the sequence of images.

41. The computer-implemented method of claim 40, wherein said solving the optimization problem is performed subject to one or more constraints, wherein the one or more constraints are specified with respect to one or more of:

priorities or preferences with respect to image attributes;

repetition or frequency of image attributes;

priorities or preferences regarding available transition effects;

repetition or frequency of transition effects;

or

distribution of image content.

42. The computer-implemented method of claim 30, wherein the transition effects comprise one or more of:

an object based transition effect;

a face based transition effect;

a large segment based transition effect;

a long edge based transition effect that involves the edges or lines longer than the threshold length;

a depth discontinuity based transition effect;

a panoramic stitch based transition effect;

a local similar regions based transition effect, wherein the local similar regions comprise similar backgrounds, similar objects, or similar logos;

or

a global image based transition effect.

43. The computer-implemented method of claim 30, wherein said determining the respective transition for each pair of the plurality of pairs of the images comprises determining one or more image operations for each of the determined transition effects to implement the transition, wherein said determining the one or more image operations comprises determining one or more of:

camera motion path;

zoom;

rotation;

morph;

uniform cross-fade;

non-uniform cross-fade;

or

synthesis of a boundary or transition region.