System and method of automatically creating an emotional controlled soundtrack

• US 20060122842A1
• Filed: 12/01/2005
• Published: 06/08/2006
• Est. Priority Date: 12/03/2004
• Status: Active Grant

• Current Assignee: MAGIX Software GmbH
• Original Assignee: Bellevue Investments GmbH & Co. KGaA
• Inventors: Herberger, Tilman, Tost, Titus
• Granted Patent: US 7,754,959 B2
• Time in Patent Office: Days
• US Class Current: 704/278
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/125 Medley, i.e. linking parts of different musical pieces in one single piece, e.g. sound collage, DJ mix
• CPC Class Codes: G10H 2240/085 Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
• CPC Class Codes: G10H 2240/135 Library retrieval index, i.e. using an indexing scheme to efficiently retrieve a music piece
• CPC Class Codes: G10H 2250/641 Waveform sampler, i.e. music samplers; Sampled music loop processing, wherein a loop is a sample of a performance that has been edited to repeat seamlessly without clicks or artifacts
• CPC Class Codes: G11B 2220/2545 CDs
• CPC Class Codes: G11B 2220/2562 DVDs [digital versatile discs]; Digital video discs; MMCDs; HDCDs
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/24 by sensing features on the record carrier other than the transducing track ; sensing signals or marks recorded by another method than the main recording
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

There is provided herein a system and method for enabling a user of digital video editing software to automatically create an emotionally controlled soundtrack that is matched in overall emotion or mood to the scenes in the underlying video work. In the preferred arrangement, the user will be able to control the generation of the soundtrack by positioning emotion tags in the video work that correspond to the general mood of each scene. The subsequent soundtrack generation step will utilize these tags to prepare a musical accompaniment to the video work that generally matches its on-screen activities.

Claims

1. A method of automatically creating a soundtrack to accompany a video work, said video work having a time duration associated therewith, wherein is provided a plurality of music loops, each of said music loops having at least one musical style associated therewith, comprising the steps of:

a. selecting a plurality of emotion tags, each of said emotion tags being associated with at least one mood and said selected plurality of emotion tags comprising at least two different moods;

b. assigning each of said selected plurality of emotion tags to at least one time point within said time duration of said video work, said assignment of said selected plurality of emotion tags being made to at least approximately match a mood of the video work at each time point so assigned;

c. for each of said selected emotion tags, selecting at least one of said music loops and said at least one music styles associated therewith, wherein at least one of said selected music loop'"'"'s at least one music styles is consistent with said selected emotion tag mood;

d. automatically generating a soundtrack to accompany said video work according to said assigned times points, said selected emotion tags, and said selected music loops; and

,

e. performing at least a portion of said selected video work and said generated soundtrack together.

2. A method of automatically creating a soundtrack according to claim 1, wherein step (b) comprises the step of:

(b1) assigning each of said selected plurality of emotion tags to only one time point within said time duration of said video work, said assignment of said selected plurality of emotion tags being made to at least approximately match a mood of the video work at each time point so assigned.

3. A method of automatically creating a soundtrack according to claim 1, wherein step (c) comprises the step of:

(c1) for each of said selected emotion tags, selecting exactly one of said music loops and said at least one music styles associated therewith, wherein each of said selected music loops has at least one style associated therewith that is consistent with said associated emotion tag.

4. A method of automatically creating a soundtrack according to claim 1, wherein said moods associated with said emotion tags are selected from a group consisting of happy, sad, romantic, quite, excited, scary, tense, frantic, contemplative, angry, nervous, and ecstatic.

5. A method of automatically creating a soundtrack according to claim 1, wherein said music styles associated with said plurality of music loops are selected from a group consisting of rock, swing, jazz, waltz, disco, Latin, country, gospel, ragtime, calypso, reggae, oriental, rhythm and blues, salsa, hip hop, rap, samba, zydeco, blues, and classical.

6. A method of automatically creating a soundtrack according to claim 1, wherein at least one of said selected emotion tags is a restriction tag, and wherein step (c) comprises the step of:

(c1) for each of said non-restriction selected emotion tags, selecting at least one of said music loops and said at least one music styles associated therewith, wherein at least one of said selected music loop'"'"'s at least one musical styles is consistent with said selected emotion tag musical mood,

and, wherein step (d) comprises the step of

(d1) automatically generating a soundtrack to accompany said video work according to said assigned times, said selected emotion tags, and said selected music loops, wherein at times within said generated soundtrack proximate to said assigned restrictions tags, said generated soundtrack is required to be at a reduced volume level.

7. A method of automatically creating a soundtrack according to claim 6, wherein step (d1) comprises the step of:

(i) automatically generating a soundtrack to accompany said video work according to said assigned times, said selected emotion tags, and said selected music loops, wherein at times within said generated soundtrack proximate to said assigned restrictions tags, said generated soundtrack is silent.

8. A method of automatically creating a soundtrack to accompany a video work, said video work having a predetermined playing time, wherein is provided a plurality of audio clips, each audio clip having an associated music style, comprising the steps of:

a. selecting a plurality of emotion tags, each of said emotion tags being associated with at least one of a plurality of different music styles, such that at least two different music styles are included among said selected emotion tags;

b. assigning each of said selected emotion tags to at least one different point in time within said video work playing time such that a mood of said video work at said assigned time at least approximately matches one of said assigned emotion tag music styles;

c. for each of said points in times at which one of said emotion tags has been assigned,

(c1) choosing one of said points in time and said assigned emotion tag associated therewith, thereby choosing an emotion tag,

(c2) determining a musical style associated with said chosen emotion tag,

(c3) selecting an audio clip having an associated music style consistent with said determined musical style;

(c4) using at least said chosen point in time to determine a starting time and an ending time within said video work during which said selected audio clip is played;

d. automatically generating said soundtrack to accompany said video work according to said assigned points in time, said selected audio clips, and said selected audio clip starting and ending times; and

,

e. simultaneously performing said soundtrack and said video work together on a display device.

9. A method of automatically creating a soundtrack to accompany a video work according to claim 8, wherein step (e) comprises the steps of:

(e1) simultaneously performing said soundtrack and said video work together on a display device, and,

(e2) storing a representation of said soundtrack and said video work together on a computer readable medium.

10. A method of automatically creating a soundtrack to accompany a video work according to claim 8, wherein said display device comprises

(i) at least one audio speaker, and,

(ii) a visual display medium selected from a group consisting of a video monitor, an LCD display, a plasma display, a television screen, a rear projection television screen, and a projection screen.

11. A method of automatically creating a soundtrack to accompany a video work according to claim 9, further comprising the steps of:

(e3) reading at least a portion of said stored representations of said soundtrack and said video work, and,

(e4) simultaneously performing at least a portion of said read video work and said read soundtrack.

12. A method of automatically creating a soundtrack to accompany a video work, said video work having a time duration associated therewith, comprising the steps of:

a. selecting a plurality of emotion tags, each of said selected emotion tags having a mood associated therewith, said plurality of emotion tags taken together comprising at least two different moods;

b. assigning each of said selected emotion tags to at least one time point within said time duration of video work;

c. for each of said assigned emotion tags, choosing at least one music loop associated therewith, each of said chosen music loops having a style consistent with said mood of said associated emotion tag;

d. automatically generating a soundtrack to accompany said video work at least according to said assigned time points, and said chosen music loops; and

,

e. simultaneously performing at least a portion of said selected video work and said generated soundtrack together.

13. A method of automatically creating a soundtrack to accompany a video work according to claim 12, wherein step (e) comprises the steps of:

(e1) simultaneously performing said soundtrack and said video work together on a display device, and,

(e2) storing a representation of said soundtrack and said video work together on a computer readable medium.

14. A method of automatically creating a soundtrack to accompany a video work according to claim 13, wherein said display device comprises

(i) at least one audio speaker, and,

(ii) a visual display medium selected from a group consisting of a video monitor, an LCD display, a plasma display, a television screen, a rear projection television screen, and a projection screen.

15. A method of automatically creating a soundtrack according to claim 12, wherein step (b) comprises the step of:

(b1) assigning each of said selected plurality of emotion tags to one or more different time points within said time duration of said video work;

16. A method of automatically creating a soundtrack according to claim 12, wherein step (b) comprises the step of:

(b1) assigning each of said selected emotion tags to only one time point within said time duration of video work.

17. A method of automatically creating a soundtrack according to claim 12, wherein said moods of said emotion tags are selected from a group consisting of happy, sad, romantic, quite, excited, scary, tense, frantic, contemplative, and ecstatic.

18. A method of automatically creating a soundtrack according to claim 12, wherein said musical styles are selected from a group consisting of rock, swing, jazz, waltz, disco, Latin, country, gospel, ragtime, calypso, reggae, oriental, rhythm and blues, salsa, hip hop, rap, samba, zydeco, blues, and classical.

19. A method of automatically creating a soundtrack according to claim 12, wherein at least one of said selected emotion tags is a restrictions tag, and wherein step (c) comprises the step of:

(c1) for each of said selected non-restrictions emotion tags, selecting at least one music loop associated therewith, each of said selected music loops having a style consistent with said mood of said associated emotion tag;

wherein step (d) comprises the step of

(d1) automatically generating a soundtrack to accompany said video work according to said assigned time points, and said chosen music loops, wherein at times within said generated soundtrack proximate to said assigned restrictions tags, said generated soundtrack is at a reduced volume level.

20. A method of automatically creating a soundtrack according to claim 19, step (d1) comprises the step of:

(i) automatically generating a soundtrack to accompany said video work according to said assigned time points and said chosen music loops, wherein at times within said generated soundtrack proximate to said assigned restrictions tags, said generated soundtrack is silent.