Character-based automated media summarization

• US 8,392,183 B2
• Filed: 10/27/2008
• Issued: 03/05/2013
• Est. Priority Date: 04/25/2006
• Status: Active Grant

• Current Assignee: Nutshell Media, Inc.
• Original Assignee: Nutshell Media, Inc.
• Inventors: Weber, Frank Elmo
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): Sharma, Neeraj
• Application Number:
  US12/298,709
• Publication Number:
  US 20090100454A1
• Time in Patent Office: 1,590 Days
• Field of Search: 704/9, 704/246, 704/268
• US Class Current: 704/231
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

Methods, devices, systems and tools are presented that allow the summarization of text, audio, and audiovisual presentations, such as movies, into less lengthy forms. High-content media files are shortened in a manner that preserves important details, by splitting the files into segments, rating the segments, and reassembling preferred segments into a final abridged piece. Summarization of media can be customized by user selection of criteria, and opens new possibilities for delivering entertainment, news, and information in the form of dense, information-rich content that can be viewed by means of broadcast or cable distribution, “on-demand” distribution, internet and cell phone digital video streaming, or can be downloaded onto an iPod™ and other portable video playback devices.

Claims

1. A method for summarizing an audiovisual presentation, comprising:

a) providing an audiovisual presentation having a first temporal length, the audiovisual presentation comprising;

i) a plurality of dialogue segments, each dialogue segment comprising a segment component;

ii) a plurality of secondary segments selected from the group comprising a non-dialogue segment, a non-dialogue segment with music segment, a shot segment, and an audience reaction segment;

b) automatically determining a dialogue segment significance value for each dialogue segment, the method comprising;

determining at least one word significance value for at least one segment component characteristic including determining whether the segment component is a noun, pronoun, reference to a story character, a frequency of occurrence of the segment component, a match of the segment component to a predetermined database word; and

combining the at least one word significance value to determine the dialogue segment significance value;

c) optionally determining the secondary significance value of one or more of the secondary segments wherein the secondary significance value weights the dialogue segment significance value of the dialogue segment;

d) receiving a user generated time significance value threshold;

e) using the significance value threshold to select the dialogue segment if the dialogue segment significance value exceeds the significance value threshold, and delete the dialogue segment if the dialogue segment significance value is below the significance value threshold;

f) combining each dialogue segment with the dialogue segment significance value exceeding the significance value threshold into a summarized version of the audiovisual presentation; and

g) providing the summarized version of the audiovisual presentation, the audiovisual presentation having a second temporal length.

2. The method of claim 1, wherein the word significance values are determined by an identity of the character referenced.

3. The method of claim 2, wherein the word is the reference to a character and the word significance value is determined by a frequency of use of the character'"'"'s name in the audiovisual presentation and a frequency of use of a pronoun referring to the character in the audiovisual presentation.

4. The method of claim 3, wherein the pronoun is a first person pronoun matched to a speaking character, the speaking character determined by a method selected from the group consisting of a voice imprint recognition analysis, a face recognition image analysis, an embedded subtitle analysis, and a manual analysis for the purpose of antecedent identification of the first person pronoun.

5. The method of claim 3, wherein the pronoun is a second person pronoun matched to a non-speaking character when only two characters are present in a scene containing a dialogue segment, wherein the non-speaking character is determined by a method selected from the group consisting of a face recognition image analysis and a manual analysis for the purpose of antecedent identification of the second person pronoun.

6. The method of claim 3, wherein an unmatched pronoun is assigned a pronoun significance value determined by a method selected from the group consisting of averaging the word significance values of all characters referred to, averaging the word significance value of characters referred to within a predetermined number of adjacent dialogue segments, and averaging the word significance value of a predetermined number of most frequently referred to characters.

7. The method of claim 3, wherein the pronoun is identified by comparing the pronoun used in the dialogue segment with a database of pronoun type sets, and wherein the pronoun is matched to an antecedent noun based on proximity of the pronoun to the antecedent noun.

8. The method of claim 3, wherein a relative pronoun is ignored.

9. The method of claim 1 further comprising comparing the segment components with a verb set and an adjective set to determine negativity of the dialogue segment to weight the word significance value based on a predetermined set of values.

10. The method of claim 1, further comprising:

a) analyzing a group of dialogue segments, wherein the group of dialogue segments comprises a first cluster of rated dialogue segments having a first mode, followed by at least one countervailing rated dialogue segment having a countervailing mode opposite that of the first mode, followed by a second cluster of rated dialogue segments having a second mode that is the same as the first mode;

b) determining a percent of composition of the countervailing rated dialogue segments to the total number of dialogue segments in the group of dialogue segments; and

c) converting the countervailing mode to that of the first and second mode if the percent composition is less than a predetermined value.

11. The method of claim 10, wherein the converted countervailing mode is reversed if the absolute value of the significance value deviance of the countervailing rated dialogue segment is greater than the average of the absolute values of the significance value deviances of the two dialogue segments immediately adjacent to the countervailing rated dialogue segment.

12. The method of claim 1, wherein the dialogue segment significance value is weighted by the audience reaction, and determining a secondary significance value of the audience reaction further comprises:

a) separating the audience reaction into reaction segments;

b) characterizing the reaction segments in terms of at least one reaction parameter selected from the group consisting of reaction type, reaction length, and reaction intensity; and

c) rating the characterized audience reaction based on the combined rating of the reaction segments for the audience reaction to weight the dialogue segment significance value preceding the audience reaction.

13. The method of claim 1, wherein the weighting of the dialogue segment significance value comprises plotting the dialogue segment significance value of the dialogue segment and at least one secondary significance value as a significance curve overlay on a two- axis graph representing the significance value and the at least one secondary significance value as a function of time in order to weight the significance value for each dialogue segment.

14. The method of claim 1, wherein the dialogue segment significance value is weighted by a perceived presence of stress and the perceived presence of stress is determined by assessing a character communication parameter selected from the group consisting of a volume of speech, a tone of speech, a rate of speech, and a facial expression.

15. The method of claim 1 further comprising smoothing the summarized version of the audiovisual presentation using contour interpolation, comprising:

a) preparing a multiple axis plot of final segment attribute values over time;

b) connecting the plotted segment values creating a graph that resembles a cross-section of mountains and valleys;

c) tracing the plotted line;

d) resisting a predetermined maximum angle deviance of a predetermined input; and

e) effectively removing sharp peaks and crevices of said cross-sections, thereby minimizing content adjacency artifacts.

16. The method of claim 1 further comprising:

a) receiving a playback time request within which to play the summarized audiovisual presentation; and

b) automatically calculating the significance value threshold so that the summarized audiovisual presentation fits within the playback time request.

17. The method of claim 1, wherein a sound track of the audiovisual presentation is decoded into a sound industry standard multi-track format, the multi-track format comprising peripheral and a center track, wherein the center track data is comprised in the majority, of dialogue audio, and in the minority, background interference as compared to the peripheral tracks wherein the rating of each dialogue segment includes an analysis of the center track.

18. The method of claim 1, wherein the audiovisual presentation further comprises an original non-dialogue segment with music, wherein a predetermined first portion of the original non-dialogue segment with music and a predetermined last portion of the original non-dialogue segment with music are selected to include in the summarized version of the audiovisual presentation and a predetermined middle portion of the original non-dialogue segment with music is excluded from the summarized version of the audiovisual presentation, wherein the ratio of the combined length of the included portions to the length of the original non-dialogue segment with music is equal to the ratio of the second temporal length to the first temporal length.

19. The method of claim 1, wherein each dialogue segment is a sentence and the segment component is a word having a word significance value, wherein the dialogue segment significance value of each dialogue segment is determined by a combined rating of the word significance value of each segment component in the sentence.

20. The method of claim 19, wherein the word significance value is determined by comparing the word to at least one database of words selected from the group consisting of a user-defined database, a field of expertise database, and a noun database to determine whether the word segment matches any word in the at least one database, wherein a match increases the word significance value.

21. The method of claim 20, wherein the word is a reference to a character having a noun significance value, wherein the noun significance value is determined by a frequency of references to the character in the audiovisual work.

22. The method of claim 21 further comprising:

identifying a pronoun in the sentence by comparing the noun in the sentence to a database of pronoun type sets; and

matching the pronoun to an antecedent noun based on proximity of the pronoun and the antecedent noun in the sentence to increase the noun significance value of the noun.

23. The method of claim 1, wherein the sentence segment significance threshold is determined automatically when a user determines a percentage of truncation of the audiovisual presentation.

24. The method of claim 20, further comprising:

a) analyzing a group of sentences, wherein the group of sentences comprises a first cluster of rated sentences having a first mode, followed by at least one countervailing rated sentence having a countervailing mode opposite that of the first mode, followed by a second cluster of rated sentences having a second mode that is the same as the first mode;

b) determining a percent of composition of the countervailing rated sentence to the total number of sentences in the group of sentences; and

c) converting the countervailing mode to the first and second mode if the percent composition is less than a predetermined value.

25. The method of claim 24, wherein the converted countervailing mode is reversed if the absolute value of the significance value deviance of the countervailing rated sentence is greater than the average of the absolute values of the significance value deviances of the two sentences immediately adjacent to the countervailing rated sentence.

26. The method of claim 19 further comprising smoothing the summarized version of the audiovisual presentation using contour interpolation, comprising:

a) preparing a multiple axis plot of final segment attribute values over time;

b) connecting the plotted segment values creating a graph that resembles a cross-section of mountains and valleys;

c) tracing the plotted line;

d) resisting a maximum angle deviance of user input; and

e) effectively removing sharp peaks and crevices of said cross-sections, thereby minimizing content adjacency artifacts.

27. The method of claim 19, wherein the word significance value is determined by at least one word characteristic selected from the group consisting of whether the word is a noun or a pronoun, whether the word is a references to a character, a frequency of occurrence of the word in the audiovisual work, an identity of the character referenced, and a match of the word to a predetermined database word.