Automatic detection and segmentation of music videos in an audio/video stream

• US 7,336,890 B2
• Filed: 02/19/2003
• Issued: 02/26/2008
• Est. Priority Date: 02/19/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Mingjing, Hua, Xian-Sheng, Zhang, Hong-Jiang, Lu, Lie, Sun, Yan-Feng
• Primary Examiner(s): Tran; Thai Q.
• Assistant Examiner(s): Hasan; Syed Y.
• Application Number:
  US10/370,314
• Publication Number:
  US 20040170392A1
• Time in Patent Office: 1,833 Days
• Field of Search: 386/46, 386/52, 386/83, 386/95, 386/96, 386/98, 386/121, 386/125, 358/908, 382/176, 382/190, 704/278, 707/5, 707/104.1, 709/224, 715/719, 715/721
• US Class Current: 386/239
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G11B 2220/218 Write-once discs
• CPC Class Codes: G11B 2220/2545 CDs
• CPC Class Codes: G11B 2220/2562 DVDs [digital versatile discs]; Digital video discs; MMCDs; HDCDs
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/58 of audio determination or detection of speech characteristics in general G10L25/00; speech recognition in general G10L15/00
• CPC Class Codes: H04H 60/59 of video recognising characters or patterns in general G06F18/00, G06V20/00
• CPC Class Codes: H04H 60/74 using programme related information, e.g. title, composer or interpreter
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/4332 by placing content in organized collections, e.g. local EPG data repository interfaces, Database management systems or updating for information retrieval G06F16/23; details of retrieval of video data and associated meta data in video database G06F16/739
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44204 Monitoring of content usage, e.g. the number of times a movie has been viewed, copied or the amount which has been watched monitoring of user activities for profile generation for accessing a video database G06F16/739; protecting generic digital content where the protection is independent of the precise nature of the content G06F21/10; arrangements for monitoring the use made of the broadcast services in broadcast systems H04H60/31
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 5/147 Scene change detection
• CPC Class Codes: H04N 5/602 for digital sound signals

Abstract

A “music video parser” automatically detects and segments music videos in a combined audio-video media stream. Automatic detection and segmentation is achieved by integrating shot boundary detection, video text detection and audio analysis to automatically detect temporal boundaries of each music video in the media stream. In one embodiment, song identification information, such as, for example, a song name, artist name, album name, etc., is automatically extracted from the media stream using video optical character recognition (OCR). This information is then used in alternate embodiments for cataloging, indexing and selecting particular music videos, and in maintaining statistics such as the times particular music videos were played, and the number of times each music video was played.

Claims

1. A system for automatically detecting and segmenting music videos in an audio-video media stream, comprising:

preprocessing a media stream using a plurality of boundary detection methods to locate a plurality of potential music video boundaries within the media stream;

jointly evaluating the potential music video boundaries to identify one or more of the potential music video boundaries that are not likely to be a part of an actual music video;

eliminating from further consideration the identified potential music video boundaries that are not likely to be a part of an actual music video; and

analyzing the content of segments of the media stream between any remaining potential music video boundaries to determine whether the content between any two or more potential music video boundaries represents an actual music video.

2. The system of claim 1 further comprising performing audio discrimination of the media stream to eliminate one or more of the remaining potential music video boundaries where the eliminated potential boundaries are determined to represent portions of the media stream that are not likely to be a part of a music video prior to analyzing the content of segments of the media stream.

3. The system of claim 1 further comprising identifying start and end points for each music video in the media stream from the remaining potential music video boundaries.

4. The system of claim 3 further comprising segmenting each music video from the media stream and storing the music videos on a computer readable medium.

5. The system of claim 1 wherein the boundary detection methods include any of shot detection, black screen detection, audio boundary detection, and video text detection.

6. The system of claim 1 wherein determining whether the content between any two or more potential music video boundaries represents a music video comprises identifying the content as a music video when there is text information in the beginning and the end of the content and the length of the content is on the order of an average song length.

7. The system of claim 1 wherein determining whether the content between any two or more potential music video boundaries represents a music video comprises identifying the content as a music video when:

there is text information in the beginning of a first section of content between two potential music video boundaries and there is text information in the end of a subsequent adjacent section of content between two potential music video boundaries;

a total length of the adjacent sections of content when combined is on the order of an average song length; and

a combined music ratio of these adjacent sections is greater than a predetermined threshold.

8. The system of claim 1 wherein determining whether the content between any two or more potential music video boundaries represents a music video comprises identifying the content as a music video when:

there is text information in either the beginning of a first section of content between two potential music video boundaries or the end of an adjacent second section of content between two potential music video boundaries;

a total length of the adjacent sections of content when combined is on the order of an average song length; and

a combined music ratio of these adjacent sections is greater than a predetermined threshold.

9. The system of claim 1 wherein determining whether the content between any two or more potential music video boundaries represents a music video comprises identifying the content as a music video when:

there is no text information in a section of content between two potential music video boundaries;

a total length of the section of content is on the order of an average song length; and

a music ratio of the section of content is greater than a predetermined threshold.

10. The system of claim 1 wherein determining whether the content between any two or more potential music video boundaries represents a music video comprises identifying the content as a music video when:

there is no text information in a section of content between two potential music video boundaries;

a total length of the section of content is approximately an average song length; and

a music ratio of the section of content is not less than a predetermined minimum threshold.

11. The system of claim 1 further comprising performing video optical character recognition on text within each music video to extract information related to each music video.

12. A computer-implemented process for automatically parsing music videos in an audio-video media stream, comprising:

preprocessing a media stream using a plurality of boundary detection methods including one or more of shot detection, black screen detection, audio boundary detection, and video text detection to locate a plurality of potential music video boundaries within the media stream;

jointly evaluating the potential music video boundaries to identify one or more of the potential music video boundaries that are likely to be a part of an actual music video;

eliminating any potential music video boundaries from further consideration that are not identified as being likely to be a part of an actual music video; and

analyzing the content of segments of the media stream between any two or more remaining potential music video boundaries to determine whether the content between any two or more potential music video boundaries represents an actual music video.

13. The computer-implemented process of claim 12 further comprising performing audio discrimination of the media stream to eliminate one or more of the remaining potential music video boundaries where the eliminated potential boundaries are determined to represent portions of the media stream that are not likely to be a part of a music video prior to analyzing the content of segments of the media stream.

14. The computer-implemented process of claim 12 further comprising identifying start and end points for each music video in the media stream from the remaining potential music video boundaries.

15. The computer-implemented process of claim 14 further comprising segmenting each music video from the media stream and storing the music videos on a computer readable medium.

16. The computer-implemented process of claim 12 further comprising performing video optical character recognition on text within each music video to extract information related to each music video.

17. A physical computer storage media having computer executable instructions for automatically extracting endpoint information for music videos embedded in an audio-video media stream, comprising:

analyzing a media stream using one or more boundary detection methods including shot detection, black screen detection, audio boundary detection, and video text detection to locate a plurality of potential music video boundaries within the media stream;

jointly evaluating the potential music video boundaries to identify one or more of the potential music video boundaries that are likely to be a part of an actual music video and eliminating any potential music video boundaries from further consideration that are identified as not being likely to be a part of an actual music video;

performing audio discrimination of the media stream to eliminate one or more of the remaining potential music video boundaries where the eliminated potential boundaries are determined to represent portions of the media stream that are not likely to be a part of an actual music video; and

analyzing the content of segments of the media stream between any two or more remaining potential music video boundaries to determine whether a segment of the media stream between any two or more potential music video boundaries represents an actual music video.

18. The physical computer storage media of claim 17 further comprising segmenting each music video from the media stream and storing the music videos on a computer readable medium.

19. The physical computer storage media of claim 17 further comprising performing video optical character recognition on text within each music video to extract information related to each music video.

20. The physical computer storage media of claim 17 wherein determining whether the content between any two or more potential music video boundaries represents a music video comprises identifying the segment as a music video when there is text information in the beginning and the end of the segment and the length of the segment is on the order of about two to six minutes.

21. A system for automatically parsing music videos in a composite audio-video media stream, comprising:

preprocessing a composite audio-video media stream using a plurality of boundary detection methods to locate a plurality of potential music video boundaries within the media stream;

jointly evaluating the potential music video boundaries to identify one or more of the potential music video boundaries that are likely to be a part of an actual music video;

eliminating any potential music video boundaries from further consideration that are not identified as being likely to be a part of an actual music video; and

analyzing the content of segments of the media stream between any remaining potential music video boundaries to determine whether the content between any two or more of the remaining potential music video boundaries represents a complete music video.

22. The system of claim 21 wherein the boundary detection methods include at least one of:

shot detection;

black screen detection;

audio boundary detection;

audio discrimination analysis; and

video text detection.

23. The system of claim 21 further comprising identifying start and end points for each music video in the media stream from the potential music video boundaries.

24. The system of claim 23 further comprising segmenting each music video from the media stream and storing the music videos on a computer readable medium.