Content-based dynamic photo-to-video methods and apparatuses

• US 20040264939A1
• Filed: 06/30/2003
• Published: 12/30/2004
• Est. Priority Date: 06/30/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hua, Xian-Sheng, Zhang, Hong-Jiang, Lu, Lie
• Granted Patent: US 7,904,815 B2
• Time in Patent Office: Days
• US Class Current: 386/280
• CPC Class Codes: H04N 1/32101 Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
• CPC Class Codes: H04N 2201/3247 Data linking a set of images to one another, e.g. sequence, burst or continuous capture mode
• CPC Class Codes: H04N 2201/3264 of sound signals

Abstract

Methods and apparatuses are provided for automatically generating video data based on still image data. Certain aspects of the video may also be configured to correspond to audio features identified within associated audio data.

Claims

1. A method comprising:

accessing at least one initial digital image;

determining at least one photo motion clip that includes information that can be used to render a plurality of digital images based on at least one portion of said initial digital image; and

generating output data based on at least said photo motion clip.

2. The method as recited in claim 1, further comprising:

determining at least one audio clip, and

wherein generating said output data based on at least said photo motion clip also includes generating said output data based on at least said photo motion clip and said at least one audio clip.

3. The method as recited in claim 2, wherein said audio clip includes a sub-music clip and determining said at least one audio clip includes selectively dividing music data into a plurality of sub-music clips.

4. The method as recited in claim 3, further comprising determining at least one boundary of at least one of said sub-music clips based on an audio signal energy level.

5. The method as recited in claim 4, further comprising detecting said audio signal energy level associated with said music data.

6. The method as recited in claim 3, further comprising determining at least one boundary of at least one of said sub-music clips based on at least one strong beat.

7. The method as recited in claim 6, further comprising detecting said at least one strong beat in an audio signal associated with said music data.

8. The method as recited in claim 3, further comprising determining at least one boundary of at least one of said sub-music clips based on at least one onset.

9. The method as recited in claim 8, further comprising detecting at least one onset in an audio signal associated with said music data.

10. The method as recited in claim 3, further comprising determining at least one boundary of at least one of said sub-music clips based on a defined period of time.

11. The method as recited in claim 2, wherein said audio clip includes music data.

12. The method as recited in claim 2, wherein said audio clip includes non-music data.

13. The method as recited in claim 2, further comprising associating said photo motion clip with at least one sub-music clip.

14. The method as recited in claim 1, wherein said initial digital image includes captured digital image data associated with at least one photo subject.

15. The method as recited in claim 1, wherein said initial digital image includes non-captured digital image generated by a computing device.

16. The method as recited in claim 1, wherein said at least one portion of said initial digital image includes at least one focus area.

17. The method as recited in claim 16, further comprising detecting said at least one focus area.

18. The method as recited in claim 16, wherein said focus area is associated with facial image data.

19. The method as recited in claim 18, further comprising detecting said facial image data.

20. The method as recited in claim 16, wherein said focus area is associated with salience image data.

21. The method as recited in claim 20, further comprising detecting said salience image data.

22. The method as recited in claim 16, further comprising determining a confidence value associated with said focus area.

23. The method as recited in claim 1, wherein said photo motion clip includes at least two portions of a digital image.

24. The method as recited in claim 23, wherein each of said portions of said digital image includes at least one unique focus area.

25. The method as recited in claim 24, further comprising sorting each of said portions based on said unique focus areas.

26. The method as recited in claim 25, wherein said portions are sorted based on a confidence level associated with each of said unique focus areas.

27. The method as recited in claim 25, wherein said portions are sorted based on a type of focus area for each of said unique focus areas.

28. The method as recited in claim 1, wherein said portion is rectangular in shape.

29. The method as recited in claim 1, wherein said portion is non-rectangular in shape.

30. The method as recited in claim 2, further comprising rendering video data based on said audio clip and said photo motion clip.

31. The method as recited in claim 30, wherein said video data includes at least two different photo motion clips.

32. The method as recited in claim 31, wherein each of said photo motion clips is associated with a different sub-music clip within said audio clip.

33. The method as recited in claim 31, further comprising applying at least one transition effect when transitioning from a first photo motion clip to a second photo motion clip.

34. The method as recited in claim 33, wherein said transition effect is selected from a group of transition effects comprising a fade transition effect and a wipe transition effect.

35. The method as recited in claim 30, further comprising applying at least one style effect to at least a stylized portion of image data within said photo motion clip when rendering said video.

36. The method as recited in claim 35, wherein said style effect is selected from a group of style effects comprising a grayscale effect, a color tone effect, a sepia tone effect, a contrast effect, a watercolor effect, and a brightness effect.

37. The method as recited in claim 30, wherein said video data includes a plurality of frames associated with said photo motion clip.

38. The method as recited in claim 2, further comprising generating script data based on said audio clip and said photo motion clip.

39. The method as recited in claim 38, further comprising rendering video data based on said script data.

40. The method as recited in claim 1, further comprising determining at least one motion pattern to be applied in said photo motion clip.

41. The method as recited in claim 40, wherein said motion pattern is associated with at least one sub-music clip.

42. The method as recited in claim 40, further comprising applying said determined motion pattern to said motion clip.

43. The method as recited in claim 40, wherein said motion pattern is selected from a group of motion patterns comprising still motion, light zooming-in motion, light zooming-out motion, panning motion, zooming-in motion, zooming-out motion, panning with light zooming-in motion, panning with light zooming-out motion, panning with zooming-in motion, and panning with zooming-out motion.

44. The method as recited in claim 40, wherein said motion pattern includes at least a starting point and an ending point within said digital image.

45. The method as recited in claim 40, wherein said motion pattern includes trajectory information.

46. The method as recited in claim 45, wherein said trajectory information is associated with simulated virtual camera motion with regard to said initial digital image.

47. The method as recited in claim 45, wherein said trajectory information is associated with at least a starting point within said initial digital image.

48. The method as recited in claim 47, wherein said trajectory information is associated with an ending point within said initial digital image.

49. The method as recited in claim 47, wherein said trajectory information is associated with a time parameter.

50. The method as recited in claim 45, wherein said trajectory information is associated with at least one speed control function.

51. The method as recited in claim 50, wherein said speed control function varies over time.

52. The method as recited in claim 50, wherein said speed control function is linearly tapered.

53. The method as recited in claim 50, wherein said speed control function is non-linearly tapered.

54. The method as recited in claim 50, wherein said speed control function is selected from a group of speed control functions comprising a zooming speed control function and a panning speed control function.

55. The method as recited in claim 45, wherein said trajectory information is based on at least one arc length parameter.

56. The method as recited in claim 45, wherein said trajectory information is based on at least one time parameter.

57. The method as recited in claim 1, further comprising generating at least one storyboard based on a plurality of photo motion clips.

58. A computer-readable medium having computer-implementable instructions for causing at least one processing unit to perform acts comprising:

accessing at least one initial digital image;

configuring at least one photo motion clip that includes information that can be used to render a plurality of digital images based on at least one portion of said initial digital image; and

creating output data based on at least said photo motion clip.

59. The computer-readable medium as recited in claim 58, further comprising:

analyzing at least one audio clip, and

wherein creating said output data based on at least said photo motion clip also includes generating said output data based on at least said photo motion clip and said at least one audio clip.

60. The computer-readable medium as recited in claim 59, wherein said audio clip includes a sub-music clip and analyzing said at least one audio clip includes selectively dividing music data into a plurality of sub-music clips.

61. The computer-readable medium as recited in claim 60, further comprising:

detecting audio signal energy level associated with said music data; and

=p1 determining at least one boundary of at least one of said sub-music clips based on said audio signal energy level.

62. The computer-readable medium as recited in claim 60, further comprising:

detecting at least one strong beat in an audio signal associated with said music data; and

determining at least one boundary of at least one of said sub-music clips based on at least one strong beat.

63. The computer-readable medium as recited in claim 60, further comprising:

detecting at least one onset in an audio signal associated with said music data; and

determining at least one boundary of at least one of said sub-music clips based on said at least one onset.

64. The computer-readable medium as recited in claim 60, further comprising determining at least one boundary of at least one of said sub-music clips based on a defined period of time.

65. The computer-readable medium as recited in claim 59, further comprising associating said photo motion clip with at least one sub-music clip.

66. The computer-readable medium as recited in claim 58, further comprising detecting said at least one focus area within said portion of said initial digital image.

67. The computer-readable medium as recited in claim 66, wherein said focus area is associated data selected from a group of data comprising facial image data and salience image data.

68. The computer-readable medium as recited in claim 66, further comprising determining a confidence value associated with said focus area.

69. The computer-readable medium as recited in claim 59, further comprising rendering video data based on said audio clip and said photo motion clip, and wherein said video data includes at least two different photo motion clips and each of said photo motion clips is associated with a different sub-music clip within said audio clip.

70. The computer-readable medium as recited in claim 69, further comprising applying at least one transition effect when transitioning from a first photo motion clip to a second photo motion clip.

71. The computer-readable medium as recited in claim 59, further comprising generating script data based on said audio clip and said photo motion clip.

72. The computer-readable medium as recited in claim 59, further comprising applying motion pattern to said motion clip.

73. The computer-readable medium as recited in claim 72, wherein said motion pattern is associated with at least one sub-music clip.

74. The computer-readable medium as recited in claim 72, wherein said motion pattern is selected from a group of motion patterns comprising still motion, light zooming-in motion, light zooming-out motion, panning motion, zooming-in motion, zooming-out motion, panning with light zooming-in motion, panning with light zooming-out motion, panning with zooming-in motion, and panning with zooming-out motion.

75. The computer-readable medium as recited in claim 72, wherein said motion pattern includes trajectory information associated with simulated virtual camera motion with regard to said digital image.

76. The computer-readable medium as recited in claim 75, wherein said trajectory information is associated with at least one speed control function.

77. The computer-readable medium as recited in claim 76, wherein said speed control function varies over time.

78. The computer-readable medium as recited in claim 75, wherein said trajectory information is based on at least one arc length parameter.

79. The computer-readable medium as recited in claim 75, wherein said trajectory information is based on at least one time parameter.

80. An apparatus comprising logic configured to access at least on initial digital image, generate at least one photo motion clip that includes information that can be used to render a plurality of digital images based on at least one portion of said initial digital image, and produce output data based on at least said photo motion clip.

81. The apparatus as recited in claim 80, wherein said logic is further configured to process at least one audio clip and produce said output data based on at least said photo motion clip and said at least one audio clip.

82. The apparatus as recited in claim 81, wherein said audio clip includes a sub-music clip and said logic is configured to selectively divide music data into a plurality of sub-music clips.

83. The apparatus as recited in claim 82, wherein said logic is further configured to detect an audio signal energy level associated with said music data and determine at least one boundary of at least one of said sub-music clips based on said audio signal energy level.

84. The apparatus as recited in claim 82, wherein said logic is further configured to detect at least one strong beat in an audio signal associated with said music data, and determine at least one boundary of at least one of said sub-music clips based on at least one strong beat.

85. The apparatus as recited in claim 82, wherein said logic is further configured to detect at least one onset in an audio signal associated with said music data, and determine at least one boundary of at least one of said sub-music clips based on said at least one onset.

86. The apparatus as recited in claim 82, wherein said logic is further configured to determine at least one boundary of at least one of said sub-music clips based on a defined period of time.

87. The apparatus as recited in claim 81, wherein said logic is further configured to associate said photo motion clip with at least one sub-music clip.

88. The apparatus as recited in claim 80, wherein said logic is further configured to detect said at least one focus area within said portion of said digital image.

89. The apparatus as recited in claim 88, wherein said focus area is associated data selected from a group of data comprising facial image data and salience image data.

90. The apparatus as recited in claim 88, wherein said logic is further configured to determine a confidence value associated with said focus area.

91. The apparatus as recited in claim 81, wherein said logic is further configured to render video data based on said audio clip and said photo motion clip, and wherein said video data includes at least two different photo motion clips and each of said photo motion clips is associated with a different sub-music clip within said audio clip.

92. The apparatus as recited in claim 80, wherein said logic is further configured to apply at least one transition effect when transitioning from a first photo motion clip to a second photo motion clip.

93. The apparatus as recited in claim 81, wherein said logic is further configured to generate script data based on said audio clip and said photo motion clip.

94. The apparatus as recited in claim 81, wherein said logic is further configured to apply motion pattern to said motion clip.

95. The apparatus as recited in claim 94, wherein said motion pattern is associated with at least one sub-music clip.

96. The apparatus as recited in claim 94, wherein said motion pattern is selected from a group of motion patterns comprising still motion, light zooming-in motion, light zooming-out motion, panning motion, zooming-in motion, zooming-out motion, panning with light zooming-in motion, panning with light zooming-out motion, panning with zooming-in motion, and panning with zooming-out motion.

97. The apparatus as recited in claim 94, wherein said motion pattern includes trajectory information associated with simulated virtual camera motion with regard to said digital image.

98. The apparatus as recited in claim 97, wherein said trajectory information is associated with at least one speed control function.

99. The apparatus as recited in claim 98, wherein said speed control function varies over time.

100. The apparatus as recited in claim 97, wherein said trajectory information is based on at least one arc length parameter.

101. The apparatus as recited in claim 97, wherein said trajectory information is based on at least one time parameter.