Method and apparatus for summarizing a music video using content analysis

• US 7,599,554 B2
• Filed: 04/02/2004
• Issued: 10/06/2009
• Est. Priority Date: 04/14/2003
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Kender, John, Dimitrova, Nevenka, Agnihotri, Lalitha
• Primary Examiner(s): Couso; Jose L
• Application Number:
  US10/552,829
• Publication Number:
  US 20060210157A1
• Time in Patent Office: 2,013 Days
• Field of Search: 382/173, 382/175, 382/176, 382180-181, 382190-191, 382224-228, 382/236, 382/254, 382/282, 382305-306, 386/46, 386/52, 386/83, 386/96, 704/278, 348/402.1, 348/407.1, 3484131-4141, 3484161-4181, 348/422.1, 3484301-4311, 375/240.16, 707/104.1, 715/716
• US Class Current: 382/173
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04H 60/58 of audio determination or detection of speech characteristics in general G10L25/00; speech recognition in general G10L15/00

Abstract

A method and apparatus are provided for segmenting and summarizing a music video (507) in a multimedia stream (505) using content analysis. A music video (507) is segmented in a multimedia stream (505) by evaluating a plurality of content features that are related to the multimedia stream. The plurality of content features includes at least two of a face presence feature; a videotext presence feature; a color histogram feature; an audio feature, a camera cut feature; and an analysis of key words obtained from a transcript of the at least one music video. The plurality of content features are processed using a pattern recognition engine (1000), such as a Bayesian Belief Network, or one or more video segmentation rules (1115) to identify the music video (507) in the multimedia stream (505). A chorus is detected in at least one music video (507) using a transcript (T) of the music video (507) based upon a repetition of words in the transcript. The extracted chorus may be employed for the automatic generation of a summary of the music video (507).

Claims

1. A method for segmenting a music video in a multimedia stream, said method comprising the steps of:

receiving a multimedia stream including at least one music video;

using a processor for segmenting said at least one music video from said multimedia by evaluating a plurality of content features related to said multimedia stream; and

identifying said at least one music video, wherein said plurality of content features includes a face presence feature to evaluate patterns in the presentation of faces in said multimedia stream.

2. The method as claimed in claim 1, wherein said method further comprises the step of:

generating a summary of said at least one music video.

3. The method as claimed in claim 1, wherein said summary of said at least one music video is presented to a user based on personalized preferences.

4. The method as claimed in claim 1, wherein said at least one music video may be retrieved by a user based on personalized preferences.

5. The method as claimed in claim 1, wherein said plurality of content features are processed using a pattern recognition engine to identify said at least one music video.

6. The method as claimed in claim 1, wherein said plurality of content features are processed using a Bayesian Belief Network to identify said at least one music video.

7. The method as claimed in claim 1, wherein said plurality of content features are processed using one or more video segmentation rules to identify said at least one music video.

8. The method as claimed in claim 1, wherein said plurality of content features further includes a videotext presence feature that determines when videotext appears in said multimedia stream.

9. The method as claimed in claim 1, wherein said plurality of content features further includes a color histogram feature to evaluate patterns in the color content of said multimedia stream.

10. The method as claimed in claim 1, wherein said plurality of content features further includes a camera cut feature to evaluate patterns in the camera cuts and movements in said multimedia stream.

12. The method as claimed in claim 1, wherein said plurality of content features further includes an analysis of low level features derived directly from said multimedia stream.

13. The method as claimed in claim 12, wherein said low level features include one or more of a number of edges or shapes or local or global motion.

14. The method as claimed in claim 1, wherein said plurality of content features further includes an audio feature.

15. The method as claimed in claim 14, wherein said audio feature evaluates a volume of said multimedia stream.

16. The method as claimed in claim 14, wherein said audio feature evaluates one or more of a mel frequency cepstral coefficient (MFCC), linear predictive coefficient (LPC), or variations in pitch bandwidth, volume or tone.

17. The method as claimed in claim 1, wherein said method further comprises the step of:

obtaining identifying information for said at least one music video from an external source.

11. A method for segmenting a music video in a multimedia stream, said method comprising the steps of:

receiving a multimedia stream including at least one music video;

using a processor for segmenting said at least one music video from said multimedia stream by evaluating a plurality of content features related to said multimedia stream; and

identifying said at least one music video, wherein said plurality of content features includes an analysis of key words obtained from a transcript of said at least one music video.

18. A method for detecting a chorus in at least one music video, said method comprising the steps of:

receiving a multimedia stream including said at least one music video;

accessing a transcript associated with said at least one music video; and

detecting said chorus based upon a repetition of words in said transcript.

19. The method as claimed in claim 18, wherein said transcript is obtained from closed caption information.

20. The method as claimed in claim 18, wherein said chorus is employed for an automatic generation of a summary of said at least one music video.

21. The method as claimed in claim 18, wherein said method further comprising the steps of:

detecting said repeated words; and

clustering said repeated words.

22. The method as claimed in claim 18, wherein said detecting step is further based upon additional content features related to said multimedia stream.

23. The method as claimed in claim 18, wherein said method further comprising the step of:

obtaining identifying information for said at least one music video from an external source.

24. An apparatus for segmenting a music video in a multimedia stream, said apparatus comprising:

a memory; and

at least one controller (270), coupled to the memory (280), operative to;

receive a multimedia stream including at least one music video;

apply a plurality of content features related to said multimedia stream to a pattern recognition engine to segment said at least one music video from said multimedia stream; and

identify said at least one music video, wherein said plurality of content features includes a face presence feature and at least one of;

a videotext presence feature;

a color histogram feature;

a camera cut feature; and

an analysis of key words obtained from a transcript of said at least one music video.

25. The apparatus as claimed in claim 24, wherein said pattern recognition engine is a Bayesian Belief Network.

26. The apparatus as claimed in claim 24, wherein said pattern recognition engine is a neural network.

27. The apparatus as claimed in claim 24, wherein said pattern recognition engine employs an Auto Regressive Moving Average technique.

28. An apparatus for segmenting a music video in a multimedia stream, said apparatus comprising:

a memory; and

at least one controller, coupled to the memory, operative to;

receive a multimedia stream including at least one music video;

apply a plurality of content features related to said multimedia stream to one or more video segmentation rules to segment said at least one music video from said multimedia stream; and

identify said at least one music video, wherein said plurality of content features includes a face presence feature and at least one of;

a videotext presence feature;

a color histogram feature;

a camera cut feature; and

an analysis of key words obtained from a transcript of said at least one music video.

29. The apparatus as claimed in claim 28, wherein said one or more video segmentation rules define a threshold for said plurality of content features to determine when a video segment has occurred.