Automatically generating audiovisual works

• US 8,347,213 B2
• Filed: 03/02/2007
• Issued: 01/01/2013
• Est. Priority Date: 03/02/2007
• Status: Active Grant

• Current Assignee: Redbrick Technologies, Inc.
• Original Assignee: Animoto, Inc. (Redbrick Technologies, Inc.)
• Inventors: Clifton, Stephen J., Hsiao, Jason R., Jefferson, Bradley C., Clifton, Thomas M.
• Primary Examiner(s): Abdul-Ali, Omar
• Application Number:
  US11/681,679
• Publication Number:
  US 20080215979A1
• Time in Patent Office: 2,132 Days
• Field of Search: 715/716, 715/721, 715/723, 715730-732
• US Class Current: 715/730
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: H04N 2201/3215 of a time or duration
• CPC Class Codes: H04N 2201/3264 of sound signals
• CPC Class Codes: H04N 2201/3266 of text or character information, e.g. text accompanying an image H04N2201/3226 takes precedence
• CPC Class Codes: H04N 2201/3267 of motion picture signals, e.g. video clip

Abstract

A method comprises receiving one or more digital visual media items; receiving one or more digital audio media items; automatically selecting and arranging one or more design modules based upon analysis of information obtained from the digital visual media items and the digital audio media items; and automatically creating an audiovisual work by arranging the selected and arranged design modules. Audiovisual works can be automatically created based upon inferred and implicit metadata including music genre, image captions, song structure, image focal points, as well as user-supplied data such as text tags, emphasis flags, groupings, and preferred video style.

Claims

1. A computer system, comprising:

logic encoded in one or more storage media for execution and when executed operable to;

receive one or more digital visual media items;

receive one or more digital audio media items;

inferentially select which of a plurality of design animation modules to use to create an overall audiovisual work, based at least in part upon all of;

(a) one or more detected visual characteristics of at least one of the digital visual media items, (b) one or more detected audio characteristics of at least one of the digital audio media items, and (c) one or more metadata values of one or more attributes obtained from at least one of the plurality of design animation modules;

assign the digital visual media items to one or more digital visual media item placeholders of one or more of the selected design animation module(s);

arrange the selected design animation module(s) to the digital audio media items;

create the audiovisual work using the selected and arranged design animation module(s);

wherein each design animation module of the plurality of design animation modules is an independent interchangeable unit that can be combined with other design animation modules of the plurality of design animation modules to form different overall audiovisual works.

2. The computer system of claim 1, further comprising the plurality of design animation modules, wherein each design animation module of the plurality of design animation modules comprises beat timing information, and one or more metadata values of one or more attributes of that design animation module.

3. The computer system of claim 1,

wherein the logic when executed is operable to inferentially select which of the plurality of design animation modules to use to create the audiovisual work, based at least in part upon all of:

(a), (b), (c), a comparison of one or more of the detected visual characteristics to one or more of the metadata values, and a comparison of one or more of the detected audio characteristics to one or more of the metadata values.

4. The computer system of claim 1, wherein the logic when executed is operable to:

obtain first data representing the one or more detected visual characteristics, based at least in part upon analysis of the at least one digital visual media item;

obtain second data representing the one or more detected audio characteristics, based at least in part upon analysis of the at least one digital audio item;

wherein the logic operable to obtain the first data does not obtain the first data based upon user input;

wherein the logic operable to obtain the second data does not obtain the second data based upon user input; and

wherein the logic operable to inferentially select which of the plurality of design animation modules does not inferentially select based upon user input.

5. The computer system of claim 1, wherein the logic when executed is operable to automatically re-time one or more of the selected design animation module(s) to one or more of the digital audio media items.

6. The computer system of claim 1, wherein one of the digital visual media items is a digital video file.

7. The computer system of claim 1, wherein the one or more detected audio characteristics includes one or more beat points of the at least one digital audio media item.

8. The computer system of claim 1, wherein the one or more detected audio characteristics includes any of a ramp-up period, climactic point, verse, chorus, bridge, genre, or style of the at least one digital audio media item.

9. The computer system of claim 1, wherein the one or more detected visual characteristics includes one or more focal points of the at least one digital visual media item.

10. The computer system of claim 1, wherein the one or more detected visual characteristics includes one or more of a face, subject, hue, saturation, orientation, brightness, contrast, title, or caption of the at least one digital visual media item.

11. The computer system of claim 1, wherein the logic when executed is operable to obtain information indicating that one or more specified digital visual media items should be grouped in the audiovisual work, and to create the audiovisual work comprising modules having groupings of the digital visual media items or digital visual media items aligned with structure attributes in the digital audio media items.

12. The computer system of claim 1, wherein the logic when executed is operable to obtain one or more text values associated with one or more of the digital visual media items, and to inferentially select which of the plurality of design animation modules to use to create the audiovisual work based at least in part upon all of:

(a), (b), (c), and a similarity of the one or more text values to one or more of the metadata values.

13. The computer system of claim 1, wherein the logic when executed is operable to obtain a preferred style of the audiovisual work, and to inferentially select which of the plurality of design animation modules to use to create the audiovisual work based at least in part upon all of:

(a), (b), (c), and a similarity of the preferred style to one or more of the metadata values.

14. The computer system of claim 1, wherein the logic when executed is operable to determine a genre of one or more of the digital audio media items, to obtain a preferred style of the audiovisual work, and to inferentially select which of the plurality of design animation modules to use to create the audiovisual work based at least in part upon all of:

(a), (b), (c), and a similarity of one or more of the metadata values to all of the genre and the preferred style.

15. The computer system of claim 1, wherein the logic when executed is operable to inferentially select a plurality of the design animation modules and to create the audiovisual work wherein at least two of the selected design animation modules are layered.

16. The computer system of claim 1, wherein the logic when executed is operable to determine a starting point in the digital audio media items, and to arrange the selected design animation module(s) to the digital audio media items beginning at the starting point.

17. The computer system of claim 1, wherein the logic operable to inferentially select comprises logic operable to inferentially select one or more three-dimensional design animation modules, based at least in part upon all of:

(a), (b), and (c);

wherein the logic to create the audiovisual work comprises logic operable to create the audiovisual work using the selected three-dimensional design animation module(s).

18. The computer system of claim 1, wherein at least one of the selected design animation module(s) is created using a design animation program.

19. The computer system of claim 1,

wherein one or more of the selected design animation module(s) comprise the one or more digital visual media item placeholders;

wherein each placeholder of the one or more digital visual media item placeholders accepts a range of digital visual media items;

wherein the selected design animation module(s) that comprise the digital visual media item placeholders comprise one or more particular metadata values, each particular metadata value corresponding to one digital visual media item placeholder of the one or more digital visual media item placeholders;

wherein each particular metadata value of the one or more particular metadata values indicates a range of digital visual media items that the corresponding digital visual media item placeholder accepts;

wherein the logic when executed is further operable to;

obtain the one or more particular metadata values from the design animation module(s) that comprise the digital visual media item placeholders; and

inferentially select, from among the plurality of design animation modules, the design animation module(s) that comprise the digital visual media item placeholders to use to create the overall audiovisual work, based at least in part upon a comparison, for each of the received digital visual media items, between a detected visual characteristic of the each received digital visual media item and one of the particular metadata values.

20. The computer system of claim 19,

wherein each of the digital visual media items is a digital image;

wherein each particular metadata value of the one or more particular metadata values indicates an image orientation of digital visual media items that the corresponding digital visual media item placeholder accepts;

wherein the logic when executed is further operable to inferentially select, from among the plurality of design animation, the design animation module(s) that comprise the digital visual media item placeholders to use to create the overall audiovisual work, based at least in part upon a comparison, for each of the received digital visual media items, between a detected image orientation of the each received digital visual media item and one of the particular metadata values.

21. The computer system of claim 18, wherein the at least one selected design animation module created using the design animation program comprises a specification of an animation scene created using the design animation program.

22. The computer system of claim 21, wherein the at least one selected design animation module created using the design animation program comprises one or more digital visual media item placeholders that, according to the specification, are animated in the animation scene.

23. The computer system of claim 22, wherein the logic when executed is further operable to assign one or more of the received digital visual media items to the one or more digital visual media item placeholders of the at least one selected design animation module created using the design animation program including modifying the specification of the animation scene such that, according to the modified specification, the digital visual media item(s) assigned to the placeholder(s) of the at least one selected design animation module created using the design animation program are animated in the animation scene.

24. A method, comprising:

receiving one or more digital visual media items;

receiving one or more digital audio media items;

one or more computing devices inferentially selecting which of a plurality of design animation modules to use to create an overall audiovisual work, based at least in part upon all of;

(a) one or more detected visual characteristics of at least one of the digital visual media items, (b) one or more detected audio characteristics of at least one of the digital audio media items, and (c) one or more metadata values of one or more attributes obtained from at least one of the plurality of design animation modules;

one or more computing devices assigning the digital visual media items to one or more digital visual media item placeholders of one or more of the selected design animation module(s);

one or more computing devices arranging the selected design animation module(s) to the digital audio media items;

one or more computing devices creating the audiovisual work using the selected and arranged design animation module(s);

wherein each design animation module of the plurality of design animation modules is an independent interchangeable unit that can be combined with other design animation modules of the plurality of design animation modules to form different overall audiovisual works.

25. The method of claim 24, further comprising accessing the plurality of design animation modules, wherein each design animation module of the plurality of design animation modules comprises beat timing information and one or more metadata values of one or more attributes of that design animation module.

26. The method of claim 24, wherein the inferentially selecting includes inferentially selecting which of the plurality of design animation to use to create the audiovisual work, based at least in part upon all of:

(a), (b), (c), and a comparison of one or more of the detected visual characteristics to one or more of the metadata values, and a comparison of one or more of the detected audio characteristics to one or more of the metadata values.

27. The method of claim 24, further comprising:

obtaining first data representing the one or more detected visual characteristics, based at least in part upon analysis of the at least one digital visual media item;

obtaining second data representing the one or more detected audio characteristics, based at least in part upon analysis of the at least one digital audio item;

wherein obtaining the first data is not based upon user input; and

wherein the inferentially selecting is not based upon user input.

28. The method of claim 24, comprising automatically re-timing one or more of the selected design animation module(s) to one or more of the digital audio media items.

29. The method of claim 24, wherein one of the digital visual media items is a digital video file.

30. The method of claim 24, wherein the one or more detected audio characteristics includes one or more beat points of the at least one digital audio media item.

31. The method of claim 24, wherein the one or more detected audio characteristics includes any of a ramp-up period, climactic point, verse, chorus, bridge, genre, or style of the at least one digital audio media item.

32. The method of claim 24, wherein the one or more detected visual characteristics includes one or more focal points of the at least one digital visual media item.

33. The method of claim 24, wherein the one or more detected visual characteristics includes one or more of a face, subject, hue, saturation, orientation, brightness, contrast, title, or captions of the at least one digital visual media item.

34. The method of claim 24, comprising obtaining information indicating that one or more specified digital visual media items should be emphasized or grouped in the audiovisual work, and creating the audiovisual work comprising modules that would allow the digital visual media items or digital visual media item groupings to align with structure attributes in the digital audio media items.

35. The method of claim 24, comprising obtaining one or more text values associated with one or more of the digital visual media items, and inferentially selecting which of the plurality of design animation modules to use to create the audiovisual work based at least in part upon all of:

(a), (b), (c), and a similarity of the one or more text values to one or more of the metadata values.

36. The method of claim 24, comprising obtaining a preferred style of the audiovisual work, and inferentially selecting which of the plurality of design animation modules to use to create the audiovisual work based at least in part upon all of:

(a), (b), (c), and a similarity of the preferred style to one or more of the metadata values.

37. The method of claim 24, comprising determining a genre of one or more of the digital audio media items, to obtain a preferred style of the audiovisual work, and inferentially selecting which of the plurality of design animation modules to use to create the audiovisual work based at least in part upon all of:

(a), (b), (c), and a similarity of one or more of the metadata values to all of the genre and the preferred style.

38. The method of claim 24, comprising selecting at least a plurality of the design animation modules and generating a plurality of layers in the audiovisual work based upon the selected design animation modules.

39. The method of claim 24, comprising determining a starting point in the digital audio media items, and arranging the selected design animation module(s) to the digital audio media items beginning at the starting point.

40. A non-transitory computer-readable medium carrying one or more sequences of instructions, which instructions, when executed by one or more processors, cause the one or more processors to perform:

receiving one or more digital visual media items;

receiving one or more digital audio media items;

inferentially selecting which of a plurality of design animation modules to use to create an overall audiovisual work, based at least in part upon all of;

(a) one or more detected visual characteristics of at least one of the digital visual media items, (b) one or more detected audio characteristics of at least one of the digital audio media items, and (c) one or more metadata values of one or more attributes obtained from at least one of the plurality of design animation modules;

assigning the digital visual media items to one or more digital visual media item placeholders of one or more of the selected design animation module(s);

arranging the selected design animation module(s) to the digital audio media items;

creating the audiovisual work using the selected and arranged design animation module(s);

wherein each design animation module of the plurality of design animation modules is an independent interchangeable unit that can be combined with other design animation modules of the plurality of design animation modules to form different overall audiovisual works.

41. A method, comprising:

providing one or more digital visual media items to an online server system comprising one more computing devices;

providing one or more digital audio media items to the online server system;

requesting the online server system to generate an audiovisual work based on the digital visual media items and the digital audio media items;

receiving the audiovisual work;

wherein the audiovisual work has been generated based on;

the online server system inferentially selecting which of a plurality of design animation modules to use to create the audiovisual work, based at least in part upon all of;

(a) one or more detected visual characteristics of at least one of the digital visual media items, (b) one or more detected audio characteristics of at least one of the digital audio media items, and (c) one or more metadata values of one or more attributes obtained from at least one of the plurality of design animation modules;

the online server system assigning the digital visual media items to one or more digital visual media item placeholders of one or more of the selected design animation module(s);

the online server system arranging the selected design animation module(s) to the digital audio media items; and

the online server system creating the audiovisual work using the selected and arranged design animation module(s);

wherein each design animation module of the plurality of design animation modules is an independent interchangeable unit that can be combined with other design animation modules of the plurality of design animation modules to form different overall audiovisual works.

42. The method of claim 41, comprising arranging the digital visual media items in a desired order before the requesting.

43. The method of claim 41, comprising specifying a start point in the digital audio media items at which the audiovisual work should begin.

44. The method of claim 41, comprising obtaining information indicating that one or more specified digital visual media items should be emphasized or grouped in the audiovisual work, and creating the audiovisual work comprising modules that would allow the digital visual media items or digital visual media item groupings to align with structure attributes in the digital audio media items.

45. The method of claim 41, comprising providing one or more text values relating to the digital visual media items, and wherein one or more of the design animation modules are inferentially selected based in part upon a similarity of the one or more text values to one or more of the metadata values.

46. The method of claim 41, comprising providing a preferred style of the audiovisual work, and wherein one or more of the design animation modules are inferentially selected based in part upon the preferred style.

47. The method of claim 41, comprising providing the one or more digital visual media items to the online server system from an Internet location.

48. The method of claim 41, comprising providing the one or more digital visual media items by uploading the digital visual media items to the online server system from any of a client computer, a mobile phone, or a handheld computer.

49. A method, comprising:

in response to receiving one or more digital visual media items and one or more digital audio media items,

one or more computing devices inferentially selecting which of a plurality of design animation modules to use to create an overall audiovisual work, based at least in part upon all of;

(a) one or more detected visual characteristics of at least one of the digital visual media items, (b) one or more detected audio characteristics of at least one of the digital audio media items, and (c) one or more metadata values of one or more attributes obtained from at least one of the plurality of design animation modules;

one or more computing devices assigning the digital visual media items to one or more digital visual media item placeholders of one or more of the selected design animation module(s);

one or more computing devices arranging the selected design animation module(s) to the digital audio media items;

one or more computing devices creating the audiovisual work using the selected and arranged design animation module(s);

wherein each design animation module of the plurality of design animation modules is an independent interchangeable unit that can be combined with other design animation modules of the plurality of design animation modules to form different overall audiovisual works.

50. The method of claim 49, wherein one of the digital visual media items is a digital video file.

51. The method of claim 49, comprising analyzing the digital audio media items to identify one or more beat points and song structure attributes of the digital audio media items, and obtaining metadata values representing one or more of the detected audio characteristics at least in part based upon any of the beat points or the song structure attributes.

52. The method of claim 49, comprising analyzing the digital audio media items to identify any of a ramp-up period, climactic point, verse, chorus, bridge, genre, or style attributes of the digital audio media items, and obtaining metadata values representing one or more of the detected audio characteristics at least in part based upon any of the attributes.

53. The method of claim 49, comprising analyzing the digital visual media items to identify one or more focal points, image characteristics, or image metadata values, and obtaining metadata values representing one or more of the detected visual characteristics at least in part based upon any of the focal points, image characteristics or image metadata values.

54. The method of claim 49, comprising analyzing the digital visual media items to identify one or more of a face, subject, hue, saturation, orientation, brightness, contrast, title, or captions and obtaining metadata values representing one or more of the detected visual characteristics at least in part based upon any of the face, subject, hue, saturation, orientation brightness, contrast, title, or captions.

55. The method of claim 49, comprising obtaining information indicating that one or more specified digital visual media items should be emphasized or grouped in the audiovisual work, and creating the audiovisual work comprising modules that would allow the digital visual media items or digital visual media item groupings to align with structure attributes in the digital audio media items.

56. The method of claim 49, comprising obtaining a preferred style of the audiovisual work, and inferentially selecting one or more of the design animation modules based in part upon the preferred style.

57. The method of claim 49, comprising determining a genre of the digital audio media items, to obtain a preferred style of the audiovisual work, and inferentially selecting one or more of the design animation modules based in part upon a similarity of one or more of the metadata values to the genre and preferred style.

58. The method of claim 49, further comprising accessing the plurality of design animation modules, wherein each design animation module comprises beat timing information and one or more metadata values of one or more attributes of that design animation module.

59. A computer system, comprising:

a storage device comprising a plurality of design animation modules for portions of an audiovisual work, wherein each design animation module comprises beat timing information, and one or more second metadata values of one or more attributes of that design animation module, wherein each design animation module of the plurality of design animation modules is an independent interchangeable unit that can be combined with other design animation modules of the plurality of design animation modules to form different overall audiovisual works;

an audiovisual creation system coupled to the storage device and comprising logic encoded in one or more storage media for execution and when executed operable to;

receive and store one or more digital visual media items;

receive and store one or more digital audio media items;

generate one or more first metadata values relating to the digital visual media items and the digital audio media items, and store the first metadata values;

analyze the digital audio media items to identify one or more beat points and song structure attributes of the digital audio media items, and to determine the first metadata values at least in part based upon any of the beat points or the song structure attributes;

analyze the digital visual media items to identify one or more focal points, image characteristics, or image metadata values, and to determine the first metadata values at least in part based upon any of the focal points, image characteristics or image metadata values;

determine an image orientation of the received digital visual media items, determine a genre of the digital audio media items, obtain a preferred style of the audiovisual work, and inferentially select which of the plurality of design animation modules to use to create an overall audiovisual work based upon a similarity of the second metadata values to all of the image orientation, genre, and the first metadata values;

arrange one or more of the selected design animation module(s) based on the first metadata values and the second metadata values;

assign the digital visual media items to one or more digital visual media item placeholders of one or more of the selected design animation module(s);

create and store the audiovisual work using the selected and arranged design animation module(s).

60. A method, comprising:

receiving one or more digital visual media items;

receiving one or more digital audio media items;

one or more computing devices inferentially selecting which of a plurality of design animation modules to use to create an overall audiovisual work, based at least in part upon all of;

(a) one or more detected visual characteristics of at least one of the digital visual media items, (b) one or more detected audio characteristics of at least one of the digital audio media items, and (c) one or more metadata values of one or more attributes obtained from at least one of the plurality of design animation modules;

one or more computing devices assigning the digital visual media items to one or more digital visual media item placeholders of one or more of the selected design animation module(s);

one or more computing devices arranging the selected design animation module(s);

one or more computing devices automatically creating the audiovisual work by using the selected and arranged design animation module(s);

wherein each design animation module of the plurality of design animation modules is an independent interchangeable unit that can be combined with other design animation modules of the plurality of design animation modules to form different overall audiovisual works.