Method for automatically producing music videos

• US 7,027,124 B2
• Filed: 02/28/2002
• Issued: 04/11/2006
• Est. Priority Date: 02/28/2002
• Status: Expired due to Term

• Current Assignee: Fuji Xerox Company Limited (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Fujifilm Holdings Corporation)
• Inventors: Girgensohn, Andreas, Foote, Jonathan, Uchihashi, Shingo, Cooper, Matthew
• Primary Examiner(s): FULLER, RODNEY EVAN
• Application Number:
  US10/086,795
• Publication Number:
  US 20030160944A1
• Time in Patent Office: 1,503 Days
• Field of Search: 352/1, 348/722, 348/738, 348/515, 382/170, 382/173, 386/52, 386/95, 386/96
• US Class Current: 352/1
• CPC Class Codes: G03B 31/00 Associated working of cameras or projectors with sound-recording or sound-reproducing means
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: G11B 27/36 Monitoring, i.e. supervising the progress of recording or reproducing for digital recording G11B20/00 and s.gr.; for monitoring, testing or measuring of TV recorders of the type covered by H04N5/76 and subgroups, see H04N17/06

Abstract

Music videos are automatically produced from source audio and video signals. The music video contains edited portions of the video signal synchronized with the audio signal. An embodiment detects transition points in the audio signal and the video signal. The transition points are used to align in time the video and audio signals. The video signal is edited according to its alignment with the audio signal. The resulting edited video signal is merged with the audio signal to form a music video.

Claims

1. A method for automatically producing a music video, comprising:

receiving an audio signal in which an audio framework is present;

receiving a video signal;

detecting transition points in the audio signal and the video signal;

aligning in time the video signal with the audio signal, wherein the video signal is aligned with the audio signal based on the transition points and audio signal framework;

editing and truncating the aligned video signal; and

merging the aligned video signal with the audio signal to form a music video signal.

2. The method of claim 1, wherein the step of aligning comprises:

forming a video segment comprising a portion of the video signal based upon the transition points in the video signal; and

aligning the video segment with a transition point in the audio signal.

3. The method of claim 2, wherein:

a boundary of the video segment is defined by a transition point in the video signal; and

the boundary of the video segment is aligned with the transition point in the audio signal.

4. The method of claim 3, wherein:

the boundary is located at beginning of video segment.

5. The method of claim 2, wherein:

a first and second transition in the audio signal define an audio segment having a length; and

the video segment has a video segment length, wherein the video segment length is reduced to equal the length of the audio segment.

6. The method of claim 5, wherein:

the video segment is truncated in order to equal the length of the audio segment.

7. The method of claim 5, wherein:

the video segment is evaluated using a suitability measurement in order to define a portion of the video segment equal to the length of the audio segment.

8. The method of claim 1, wherein the step of detecting comprises:

parameterizing the audio and video signals in order to form corresponding sets of feature vectors; and

evaluating the sets of feature vectors in order to determine the location of transitions in the corresponding signals.

9. The method of claim 8, wherein:

the parameterization includes a histogram.

10. The method of claim 8, wherein:

the parameterization includes a frequency domain transform.

11. The method of claim 8, wherein the step of evaluating comprises:

constructing a self-similarity matrix based upon a distance metric for a set of feature vectors; and

extracting structure from the self-similarity matrix via a kernel correlation.

12. The method of claim 11, wherein:

the distance metric is the Euclidean distance between feature vectors.

13. The method of claim 11, wherein:

the distance metric is the cosine of the angle between feature vectors.

14. The method of claim 11, wherein:

the distance metric is the Karhunen-Loeve distance.

15. The method of claim 8, wherein:

the evaluation of sets of feature vectors includes determining a difference between adjacent feature vectors.

16. The method of claim 8, wherein:

the evaluation of sets of feature vectors includes determining an accumulated difference between a plurality of feature vectors.

17. The method of claim 8, wherein:

the evaluation of sets of feature vectors includes comparing the time period between a previous transition and a feature vector in order to determine the location of a transition.

18. The method of claim 1, wherein the step of aligning in time comprises:

aligning the detected transitions in the audio signal with the detected transitions in the video signal using dynamic programming.

19. The method of claim 1, wherein the step of aligning in time comprises:

aligning the detected transitions in the audio signal with the detected transitions in the video signal using the tempo of the audio signal.

20. The method of claim 1, wherein the step of aligning in time comprises:

adjusting a transition detection threshold to produce a specific number of transitions corresponding to the desired number of video cuts; and

aligning the transitions of the audio signal with the transitions of the video signal.

21. The method of claim 11, wherein:

the width of the kernel determines the duration of transitions detected.

22. The method of claim 1, wherein the step of merging comprises:

combining the audio signal with a second audio signal associated with the video signal, wherein the audio signal is reduced in volume during periods of speech on the second audio signal.

23. The method of claim 1, wherein the step of aligning comprises:

receiving a user selection designating a video segment for inclusion in the music video signal.