Combined-media scene tracking for audio-video summarization

• US 20030218696A1
• Filed: 05/21/2002
• Published: 11/27/2003
• Est. Priority Date: 05/21/2002
• Status: Active Grant

• Current Assignee: Avaya Incorporated
• Original Assignee: Avaya Incorporated
• Inventors: Hu, Jianying, Bagga, Amit, Zhong, Jialin
• Granted Patent: US 8,872,979 B2
• Time in Patent Office: Days
• US Class Current: 348/700
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 3/018 Input/output arrangements for oriental characters
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 21/4143 embedded in a Personal Computer [PC]
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/440236 by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 5/073 for mutually locking plural sources of synchronising signals, e.g. studios or relay stations
• CPC Class Codes: H04N 5/144 : Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/145 : Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/147 : Scene change detection
• CPC Class Codes: H04N 5/44504 : Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits

Abstract

? Techniques are presented for analyzing audio-video segments, usually from multiple sources. A combined similarity measure is determined from text similarities and video similarities. The text and video similarities measure similarity between audio-video scenes for text and video, respectively. The combined similarity measure is then used to determine similar scenes in the audio-video segments. When the audio-video segments are from multiple audio-video sources, the similar scenes are common scenes in the audio-video segments. Similarities may be converted to or measured by distance. Distance matrices may be determined by using the similarity matrices. The text and video distance matrices are normalized before the combined similarity matrix is determined. Clustering is performed using distance values determined from the combined similarity matrix. Resulting clusters are examined and a cluster is considered to represent a common scene between two or more different audio-video segments when scenes in the cluster are similar.

Claims

1. A method, comprising:

determining two or more combined similarity measures by using two or more text similarities of two or more audio-video scenes and two or more video similarities of the two or more audio-video scenes; and

determining whether there are similar audio-video scenes in the two or more audio-video scenes by using the two or more combined similarity measures.

2. The method of claim 1, wherein there are a plurality of audio-video scenes, and the plurality of audio-video scenes originate from a single audio-video source.

3. The method of claim 1, wherein there are a plurality of audio-video scenes, and the plurality of audio-video scenes originate from a plurality of separate audio-video sources.

4. The method of claim 1, further comprising the step of determining the two or more text similarities by using text corresponding to the two or more audio-video scenes.

5. The method of claim 4, wherein the audio-video scenes originate from a plurality of audio-video sources, and wherein the step of determining the two or more text similarities further comprises the steps of:

(i) comparing text corresponding to an audio-video scene of one of the audio-video sources with text corresponding to another audio-video scene of one of the audio-video sources, the comparison producing a value of one of the text similarities; and

(ii) performing step (i) until text from each audio-video scenes for each of the audio-video sources has been compared with text for every other audio-video scenes from each of the audio-video sources.

6. The method of claim 5, wherein the step of determining the two or more text similarities further comprises the steps of:

(iii) setting a text similarity corresponding to a single audio-video scene to a predetermined value;

(iv) performing step (iii) until each text similarity for every single audio-video scene for each of the audio-video sources has been set to a predetermined value.

7. The method of claim 4, further comprising the step of obtaining the text through speech-to-text conversion.

8. The method of claim 4, further comprising the step of obtaining the text by accessing closed captioning corresponding to the audio-video scenes.

9. The method of claim 1, further comprising the step of determining the two or more video similarities by using video corresponding to the two or more audio-video scenes.

10. The method of claim 9, further comprising the steps of:

detecting scene changes; and

setting each audio-video scene to be audio-video information between two of the scene changes.

11. The method of claim 10, further comprising the step of determining text corresponding to each audio-video scene.

12. The method of claim 9, wherein the audio-video scenes originate from a plurality of audio-video sources, and wherein the step of determining the two or more video similarities further comprises the steps of:

(i) comparing video corresponding to an audio-video scene of one of the audio-video sources with video corresponding to another audio-video scene from one of the audio-video sources, the comparison producing a value of one of the video similarities; and

(ii) performing step (i) until video from each audio-video scene for each of the audio-video sources has been compared with video for every other audio-video scene from each of the audio-video sources.

13. The method of claim 12, wherein the step of determining the two or more video similarities further comprises the steps of:

(iii) setting a video similarity corresponding to a single audio-video scene to a predetermined value;

(iv) performing step (iii) until each video similarity for every single audio-video scene for each of the audio-video sources has been set to a predetermined value.

14. The method of claim 9, farther comprising the step of determining, for each video corresponding to an audio-video scene, a single image as a key frame for each video that corresponds to an audio-video scene.

15. The method of claim 9, wherein:

there is one combined similarity measure corresponding to one of the text similarities and one of the video similarities;

each text similarity is converted to a distance and each video similarity is converted to a distance, whereby each combined similarity measure is represented as a combined distance measure; and

the method further comprises the steps of normalizing the text distances and normalizing the video distances.

16. The method of claim 15, wherein each text distance is stored in a matrix, each video distance is stored in a matrix, and each combined distance measure is stored in a matrix.

17. The method of claim 15, wherein the step of determining one or more combined similarity measures further comprises the step of linearly combining each text distance and each video distance to create one of the combined distance measures.

18. The method of claim 15, wherein the step of determining one or more combined similarity measures further comprises the step of combining each text distance and each video distance through a non-linear equation to create one of the combined distance measures.

19. The method of claim 9, wherein the two or more audio-video scenes come from two or more audio-video segments, and wherein the step of determining whether there are similar audio-video scenes further comprises the step of determining whether there are similar audio-video scenes in the two or more audio-video segments by using the two or more combined similarity measures.

20. The method of claim 19, wherein the step of determining whether there are similar audio-video scenes in the two or more audio-video segments by using the two or more combined similarity measures further comprises the step of:

clustering the audio-video scenes, wherein the step of clustering uses the two or more combined similarity measures; and

determining clusters having audio-video scenes corresponding to more than one audio-video segment, whereby scenes from multiple audio-video segments corresponding to a single cluster are considered similar portions of the multiple audio-video segments.

21. The method of claim 19, wherein the step of clustering comprises the steps of:

assigning each audio-video scene to a unique cluster;

determining, by using the distance measures, a minimum inter-cluster distance from the clusters, the minimum inter-cluster distance occurring between two clusters;

when the inter-cluster distance is not greater than a predefined distance, merging the two clusters and performing the steps of assigning and determining a minimum inter-cluster distance; and

when the inter-cluster distance is greater than a predefined distance, stopping clustering.

22. A method for determining common audio-video segments, each audio-video segment comprised of a plurality of audio-video scenes, comprising:

combining a first metric and a second metric to create a third metric, wherein the first metric is based on comparisons of text segments from at least two audio-video scenes, and wherein the second metric is based on comparisons of video segments from the at least two audio-video scenes; and

selecting common audio-video segments by using the at least two audio-video scenes and the third metric.

23. A method for comparing and, if desired, summarizing audio-video segments, comprising:

determining a plurality of text similarities by comparing text portions from the audio-video scenes, each text portion corresponding to an audio-video scene;

determining a plurality of video similarities by comparing video portions from the audio-video scene, each video portion corresponding to an audio-video scene;

normalizing the text similarities;

normalizing the video similarities;

determining a plurality of combined similarities by using the normalized text and video similarities;

clustering the audio-video scenes into a plurality of clusters by using the combined similarities; and

determining which clusters, if any, have audio-video scenes from multiple audio-video segments.

24. The method of claim 23, wherein the step of determining which clusters, if any, have audio-video scenes from multiple audio-video segments determines that there is at least one cluster having multiple audio-video scenes from multiple audio-video segments and the method further comprises the step of outputting the multiple audio-video scenes corresponding to the at least one cluster.

25. The method of claim 23, wherein the step of determining which clusters, if any, have audio-video scenes from multiple audio-video segments determines that there are no clusters having multiple audio-video scenes from multiple audio-video segments.

26. The method of claim 23, wherein each audio-video segment is selected from one or more of the following:

a different audio-video source and a different program of a single audio-video source.

27. The method of claim 23, wherein the step of determining a plurality of combined similarities by using the normalized text and video similarities further comprises the step of linearly combining the normalized text and video similarities to create the combined similarities.

28. The method of claim 23, wherein the step of determining a plurality of combined similarities by using the normalized text and video similarities further comprises the step of non-linearly combining the normalized text and video similarities to create the combined similarities.

29. The method of claim 23, wherein the step of clustering comprises the steps of:

assigning each audio-video scene to a unique cluster;

determining, by using the combined similarities, a minimum inter-cluster distance from the clusters, the minimum inter-cluster distance occurring between two clusters; and

when the inter-cluster distance is not greater than a predefined distance, merging the two clusters and performing the steps of assigning and determining a minimum inter-cluster distance.

30. An article of manufacture comprising:

a computer-readable medium having computer-readable code means embodied thereon, said computer-readable program code means comprising;

a step to determine two or more combined similarity measures by using two or more text similarities of two or more audio-video scenes and two or more video similarities of the two or more audio-video scenes; and

a step to determine whether there are similar audio-video scenes in the two or more audio-video scenes by using the two or more combined similarity measures.

31. An apparatus comprising:

a memory that stores computer-readable code; and

a processor operatively coupled to the memory, said processor configured to implement the computer-readable code, said computer-readable code configured to;

determine two or more combined similarity measures by using two or more text similarities of two or more audio-video scenes and two or more video similarities of the two or more audio-video scenes; and

determine whether there are similar audio-video scenes in the two or more audio-video scenes by using the two or more combined similarity measures.