Method and system for indexing and searching timed media information based upon relevance intervals

• US 7,490,092 B2
• Filed: 02/12/2003
• Issued: 02/10/2009
• Est. Priority Date: 07/06/2000
• Status: Expired due to Term

• Current Assignee: Comcast Cable Communications Management LLC (Comcast Corporation)
• Original Assignee: StreamSage Inc (Comcast Corporation)
• Inventors: Sibley, Tim V., Aveni-Deforge, Kyle, Davis, Anthony Ruiz, Unger, Noam Carl, Morton, Michael Scott, Rubinoff, Robert
• Primary Examiner(s): LEWIS, CHERYL RENEA
• Application Number:
  US10/364,408
• Publication Number:
  US 20050216443A1
• Time in Patent Office: 2,190 Days
• Field of Search: 707/1, 707/10, 707/102, 707/104.1, 707 2- 5, 707/100, 707/200, 704/1, 704/8, 704/231, 704/235, 704/246, 704/251, 725/53, 725/115, 715719-728
• US Class Current: 1/1
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/31 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/447 Temporal browsing, e.g. timeline
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: Y10S 707/913 Multimedia
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A method and system for indexing, searching, and retrieving information from timed media files based upon relevance intervals. The method and system for indexing, searching, and retrieving this information is based upon relevance intervals so that a portion of a timed media file is returned, which is selected specifically to be relevant to the given information representations, thereby eliminating the need for a manual determination of the relevance and avoiding missing relevant portions. The timed media includes streaming audio, streaming video, timed HTML, animations such as vector-based graphics, slide shows, other timed media, and combinations thereof.

Claims

1. A method of indexing and searching timed media files comprising the steps of:

extracting data from a timed media file, said extracted data comprising at least one information representation, wherein each of said at least one information representation is an indicator of material relevant to a search; and

calculating relevance intervals for each of said at least one information representation from said extracted data, wherein each relevance interval is a continuous section of the timed media file in which a corresponding one of said at least one information representation occurs in the timed media file and which is deemed relevant to the corresponding one of said at least one information representation based on said extracted data, said calculation of relevance intervals for each of said at least one information representation including a calculation of relevance interval start and end times that is dependent upon the information representation such that each continuous section of the timed media file has start and end times.

2. A method of indexing and searching timed media files, as recited in claim 1, wherein said extracted data comprises multiple information representations.

3. A method of indexing and searching timed media files, as recited in claim 2, wherein said calculation of relevance intervals for each of said information representations results in relevance intervals for different information representations that strictly overlap.

4. A method of indexing and searching timed media files, as recited in claim 1, further comprising the step of creating a search index containing said relevance intervals.

5. A method of indexing and searching timed media files, as recited in claim 4, further comprising the steps of

calculating a magnitude of relevance for each said relevance interval;

and

storing said magnitude of relevance for each said relevance interval in the said search index.

6. A method of indexing and searching timed media files, as recited in claim 1, wherein said data extraction comprises the step of performing speech recognition of data on said timed media file.

7. A method of indexing and searching timed media files, as recited in claim 6, wherein said data extraction includes an indication of the certainty of a correct word by said speech recognition.

8. A method of indexing and searching timed media files, as recited in claim 6, wherein said data extraction includes extracting an identification of a speaker of said at least one information representation.

9. A method of indexing and searching timed media files, as recited in claim 8, wherein said extracting an identification of a speaker includes extracting information describing said speaker of said at least one information representation.

10. A method of indexing and searching timed media files, as recited in claim 6, further comprising the step of separating the output of said speech recognition into sentences.

11. A method of indexing and searching timed media files, as recited in claim 10, wherein said separation of the output of said speech recognition into sentences includes the analysis of words and phrases contained within the output.

12. A method of indexing and searching timed media files, as recited in claim 10, wherein said separation of the output of said speech recognition into sentences includes the analysis of grammatical information derived from the words and phrases contained within the output.

13. A method of indexing and searching timed media files, as recited in claim 10, wherein said separation of the output of said speech recognition into sentences includes the analysis of prosodic information contained within the output.

14. A method of indexing and searching timed media files, as recited in claim 1, wherein said data extraction includes extraction time-code data indicating a time for each occurrence of said at least one information representation.

15. A method of indexing and searching timed media files, as recited in claim 1, wherein said data extraction comprises the step of performing optical character recognition on data in said timed media file.

16. A method of indexing and searching timed media files, as recited in claim 15, wherein said data extraction includes extracting an indication of the certainty of a correct word by said optical character recognition.

17. A method of indexing and searching timed media files, as recited in claim 15, wherein said data extraction includes extracting the time-code corresponding to the time at which the optical characters are visible.

18. A method of indexing and searching timed media files, as recited in claim 1, wherein said data extraction includes extracting meta-data about text visible on-screen within the timed media file.

19. A method of indexing and searching timed media files, as recited in claim 18, wherein said meta-data includes the display position of said text visible on-screen within the timed media file.

20. A method of indexing and searching timed media files, as recited in claim 18, wherein said meta-data includes the orientation of said text visible on-screen within the timed media file.

21. A method of indexing and searching timed media files, as recited in claim 18, wherein said meta-data includes font characteristics of said text visible on-screen within the timed media file.

22. A method of indexing and searching timed media files, as recited in claim 1, wherein inputs include a transcript of the text spoken within the timed media file.

23. A method of indexing and searching timed media files, as recited in claim 22, further comprising the step of using speech recognition to synchronize the transcript with the time-code of the timed media file.

24. A method of indexing and searching timed media files, as recited in claim 22, further comprising the step of dividing the transcript into sentences based upon punctuation, capitalized words, and other formatting.

25. A method of indexing and searching timed media files, as recited in claim 1, wherein said calculation includes the processing of language contained within the timed media file.

26. A method of indexing and searching timed media files, as recited in claim 25, wherein said processed language includes text that is visible within the timed media file.

27. A method of indexing and searching timed media files, as recited in claim 26, wherein said processing of visible text includes dividing simultaneously visible text into logical elements based upon meta-data of said visible text.

28. A method of indexing and searching timed media files, as recited in claim 27, wherein said processing of visible text further comprises the step of determining the logical hierarchy of each set of said logical elements that are simultaneously visible based upon meta-data of said visible text.

29. A method of indexing and searching timed media files, as recited in claim 28, wherein said processing of visible text further comprises the step of combining said hierarchies by identifying logical elements that are similar but not simultaneously visible.

30. A method of indexing and searching timed media files, as recited in claim 27, wherein

said processing of language further includes processing language that is spoken within the timed media file;

and

said processing of language that is spoken within the timed media file includes calculating what language was spoken in reference to at least one particular visible logical object.

31. A method of indexing and searching timed media files, as recited in claim 30, wherein said calculation of what language was spoken in reference to at least one particular visible logical object includes statistical analysis of the occurrences within said spoken language of n-grams that occur within the said particular visible logical object.

32. A method of indexing and searching timed media files, as recited in claim 30, wherein a time interval is associated with at least one visible information representation based upon the time interval during which language is spoken in reference to the logical object that contains said visible information representation.

33. A method of indexing and searching timed media files, as recited in claim 26, wherein a time interval is associated with at least one visible information representation based upon the time interval over which the logical object is visible.

34. A method of indexing and searching timed media files, as recited in claim 33, wherein

said processing of language includes the association of a time interval with at least one information representation that is contained in visible text;

and

said calculation of relevance intervals includes considering the initial relevance intervals for each information representation that is contained in visible text to be the time interval associated with said information representation.

35. A method of indexing and searching timed media files, as recited in claim 25, wherein said processed language includes language that is spoken within the timed media file.

36. A method of indexing and searching timed media files, as recited in claim 35, wherein said processing of language that is spoken within the timed media file includes analyzing the logical structure of the language.

37. A method of indexing and searching timed media files, as recited in claim 36, wherein

language contained within the timed media file is segmented into sentences;

and

said analysis of the logical structure of the language includes using a rule-based system to identify sentences that indicate logical structure.

38. A method of indexing and searching timed media files, as recited in claim 37, wherein said rule-based system identifies sentences that indicate logical structure based upon the inclusion within said sentences of combinations of words, phrases, parts of speech, and grammatical structures that indicate logical structure.

39. A method of indexing and searching timed media files, as recited in claim 36, wherein said analysis of the logical structure of the includes identifying indications of topic shift

40. A method of indexing and searching timed media files, as recited in claim 36, wherein said analysis of the logical structure of the language includes identifying indications of semantic dependence, including intersentential anaphora.

41. A method of indexing and searching timed media files, as recited in claim 36, wherein said analysis of the logical structure of the language includes identifying indications of topical lists.

42. A method of indexing and searching timed media files, as recited in claim 36, wherein said analysis of the logical structure of the language includes identifying indications of continuation of topic.

43. A method of indexing and searching timed media files, as recited in claim 42, wherein said calculation of at least one relevance interval for a given information representation includes the expansion of relevance intervals based on logical structure cues.

44. A method of indexing and searching timed media files, as recited in claim 43, wherein said expansion of relevance intervals based on logical structure cues includes expansion to include information that is required to understand the context of the said information representation.

45. A method of indexing and searching timed media files, as recited in claim 43, wherein said expansion of relevance intervals based on logical structure cues includes expansion to include information that is referring to the said information representation.

46. A method of indexing and searching timed media files, as recited in claim 36, wherein said analysis of the logical structure of the language includes identifying a specific logical relationship with the preceding or following language.

47. A method of indexing and searching timed media files, as recited in claim 36, wherein said analysis of the logical structure of the language includes identifying indications of information concerning a previous, current, or subsequent speaker.

48. A method of indexing and searching timed media files, as recited in claim 36, wherein said analysis of the logical structure of the language includes identifying indications of summary or conclusion content.

49. A method of indexing and searching timed media files, as recited in claim 36, wherein said analysis of the logical structure of the language includes identifying indications of introductory content.

50. A method of indexing and searching timed media files, as recited in claim 36, wherein said analysis of the logical structure of the language includes identifying indications of explicit mention of topics discussed within the media.

51. A method of indexing and searching timed media files, as recited in claim 36, wherein said analysis of the logical structure of the language includes identifying indications of sentences that do not contain substantive information.

52. A method of indexing and searching timed media files, as recited in claim 36, wherein said rule-based system for identifying sentences that contain indications of logical structure further comprises the step of labeling sentences according to the logical structure indications.

53. A method of indexing and searching timed media files, as recited in claim 25, wherein a part of speech is determined for at least one information representation.

54. A method of indexing and searching timed media files, as recited in claim 25, wherein said processing of language includes grammatical parsing of at least a portion of said language.

55. A method of indexing and searching timed media files, as recited in claim 25, wherein a lemmatized form is determined for at least one word in said language.

56. A method of indexing and searching timed media files, as recited in claim 25, further comprising the step of calculating a centrality number for at least one occurrence of at least one information representation contained within the timed media file.

57. A method of indexing and searching timed media files, as recited in claim 56, wherein

said processing of language contained within the timed media file includes segmenting said language into sentences;

and

said calculation of a centrality number of at least one occurrence of at least one information representation is based on the position of said occurrence within the grammatical structure of the sentence that contains the occurrence.

58. A method of indexing and searching timed media files, as recited in claim 56, wherein said calculation of a centrality number includes the identification of phrases within sentences.

59. A method of indexing and searching timed media files, as recited in claim 58, further comprising the step of assigning a hierarchical structure to the phrases.

60. A method of indexing and searching timed media files, as recited in claim 56, wherein said calculation of a centrality number involves analysis of the verb argument structure within the sentence containing said at least one occurrence of at least one information representation.

61. A method of indexing and searching timed media files, as recited in claim 25, wherein said processing of language includes the filtering of language to exclude at least one word that does not carry semantic information.

62. A method of indexing and searching timed media files, as recited in claim 25, wherein said processing of language includes the calculation of one or more major topic shifts.

63. A method of indexing and searching timed media files, as recited in claim 62, wherein said calculation of major topic shifts includes maximizing a lexical cohesiveness function.

64. A method of indexing and searching timed media files, as recited in claim 63, wherein

said calculation of major topic shifts includes chunking language contained within the timed media file into short segments or sentences;

and

said lexical cohesiveness function includes the pair-wise comparison of segments to determine a relatedness value between each pair of segments.

65. A method of indexing and searching timed media files, as recited in claim 64, wherein said pair-wise comparison of segments is calculated for two segments A and B by performing a calculation that includes quantitatively comparing one or more information representations in segment A with each of one or more information representations in segment B to obtain a numerical value of relatedness between each of the information representations being compared.

66. A method of indexing and searching timed media files, as recited in claim 65, wherein said numerical value of relatedness between each of the information representations being compared takes into account whether the information representations being compared are equivalent, as in the case of identical phrases or synonyms.

67. A method of indexing and searching timed media files, as recited in claim 65, wherein said numerical value of relatedness between each of the information representations being compared takes into account the mutual information between the information representations being compared.

68. A method of indexing and searching timed media files, as recited in claim 65, wherein a subset of the numerical relatedness values between information representations in segment A and information representations in segment B are chosen such that the subset does not include more than one numerical value for any one occurrence of a particular information representation.

69. A method of indexing and searching timed media files, as recited in claim 68, wherein said pair-wise relatedness for two segments A and B includes the sum of the said subset of numerical relatedness values between information representations in segment A and information representations in segment B.

70. A method of indexing and searching timed media files, as recited in claim 64, wherein

a relatedness rank is calculated from each said relatedness value;

and

the relatedness rank for two segments A and B indicates the relative relatedness as compared to the relatedness of segments that are near A and B.

71. A method of indexing and searching timed media files, as recited in claim 70, wherein said lexical cohesive function is the inside density of the relatedness rank of the subset of segments.

72. A method of indexing and searching timed media files, as recited in claim 64, wherein said topic shifts are calculated via an iterative process including

calculating a sequence of lexical cohesiveness values, where each of said lexical cohesive values is calculated as if there were a topic shift at a particular potential topic shift location;

and

inserting a topic boundary at the potential topic shift location with the maximum calculated lexical cohesiveness value.

73. A method of indexing and searching timed media files, as recited in claim 62, wherein each topic shift that is identified is rated by a boundary sharpness measure that includes

a measurement of the lexical cohesiveness of nearby segments on either side of the topic shift;

and

a measurement of the lexical difference between nearby segments on opposite sides of the topic shift.

74. A method of indexing and searching timed media files, as recited in claim 62, wherein

each topic shift that is identified is rated by a boundary sharpness measure;

said boundary sharpness measure is calculated for all potential topic shift locations; and

local maxima of the boundary sharpness measure are identified.

75. A method of indexing and searching timed media files, as recited in claim 74, wherein identified topic shifts that are not sufficiently near a local maximum of the boundary sharpness measure are treated as though they are not topic shifts.

76. A method of indexing and searching timed media files, as recited in claim 74, wherein identified topic shifts that are not at a local maximum of the boundary sharpness measure but are sufficiently near a local maximum of the boundary sharpness measure are treated as though they are located at the nearby local maximum of the boundary sharpness measure.

77. A method of indexing and searching timed media files, as recited in claim 74, wherein

topic shifts are identified in an iterative process that is terminated based upon an analysis of the boundary sharpness measure of topic shifts identified by said iterative process;

and

said iterative process is terminated if more than a maximum number of consecutive identified topic shifts are determined to be not sufficiently near a local maximum of the boundary sharpness measure.

78. A method of indexing and searching timed media files, as recited in claim 74, wherein

topic shifts are identified in an iterative process that is terminated based upon an analysis of the boundary sharpness measure of topic shifts identified by said iterative process;

and

said iterative process is terminated if more than a maximum percentage of the identified topic shifts after any given number of iterations are determined to be not sufficiently near a local maximum of the boundary sharpness measure.

79. A method of indexing and searching timed media files, as recited in claim 62, wherein

each topic shift that is identified is rated by a boundary sharpness measure;

and

topic shifts are identified in an iterative process that is terminated based upon an analysis of the boundary sharpness measure of topic shifts identified by said iterative process.

80. A method of indexing and searching timed media files, as recited in claim 79, wherein said iterative process is not terminated before a minimum number of topic shifts have been identified that are sufficiently near a local maximum of the boundary sharpness measure.

81. A method of indexing and searching timed media files, as recited in claim 62, wherein said calculation of at least one relevance interval for a given information representation includes the determination of whether to expand relevance intervals to begin or end at topic shifts.

82. A method of indexing and searching timed media files, as recited in claim 62, wherein said calculation of at least one relevance interval for a given information representation includes the determination of whether to expand relevance intervals to entire topic segments.

83. A method of indexing and searching timed media files, as recited in claim 25, wherein said processing of language includes the identification of named entities within the language.

84. A method of indexing and searching timed media files, as recited in claim 83, wherein said processing of language further comprises the step of classification of named entities by type.

85. A method of indexing and searching timed media files, as recited in claim 84, wherein said classification of named entities by type includes the use of rules based upon semantic information related to the words contained in the identified named entity.

86. A method of indexing and searching timed media files, as recited in claim 83, wherein said processing of language further comprises the step of identifying multiple information representations that are referring to the same named entity.

87. A method of indexing and searching timed media files, as recited in claim 86, wherein said identification of multiple information representations that are referring to the same named entity includes the use of rules concerning common manipulations of specific types of named entities.

88. A method of indexing and searching timed media files, as recited in claim 86, wherein said identification of multiple information representations that are referring to the same named entity includes the use of lists of words that are synonyms when used as part of named entities.

89. A method of indexing and searching timed media files, as recited in claim 86, wherein

said processing of language includes the calculation of one or more major topic shifts;

and

said identification of multiple information representations that are referring to the same named entity includes the use of rules concerning topic shifts.

90. A method of indexing and searching timed media files, as recited in claim 86, wherein said processing of language further comprises the steps of

classification of named entities by type;

and

creating a co-reference table of named entities within the timed media file that includes the classification by type of each occurrence of each named entity and indicates multiple occurrences that refer to the same named entity.

91. A method of indexing and searching timed media files, as recited in claim 86, wherein

said calculation of at least one relevance interval for a given information representation includes the consideration of the temporal distribution of occurrences of said information representation;

and

said occurrences of the information representation include occurrences of one or more named entities that refer to the information representation.

92. A method of indexing and searching timed media files, as recited in claim 83, wherein

a part of speech is determined for at least one information representation;

and

said identification of named entities includes the use of rules for identifying named entities based upon parts of speech.

93. A method of indexing and searching timed media files, as recited in claim 83, wherein

said processing of language includes grammatical parsing of at least a portion of said language;

and

said identification of named entities includes the use of rules for identifying named entities based upon grammatical parsing information.

94. A method of indexing and searching timed media files, as recited in claim 83, wherein said identification of named entities includes checking language against lists of named entities.

95. A method of indexing and searching timed media files, as recited in claim 83, wherein said identification of named entities includes checking language against lists of words that are sometimes a part of named entities.

96. A method of indexing and searching timed media files, as recited in claim 95, wherein said lists of words that are sometime a part of named entities include an indication of the likelihood that each word is a part of a named entity.

97. A method of indexing and searching timed media files, as recited in claim 96, wherein said lists of words that are sometimes a part of named entities include multiple indications of the likelihood that indicate the likelihood that each word is a part of a named entity in each of multiple registers.

98. A method of indexing and searching timed media files, as recited in claim 83, wherein said identification of named entities includes checking language against lists of suffixes and prefixes to named entities.

99. A method of indexing and searching timed media files, as recited in claim 25, wherein said processing of languages includes the identification of anaphora within the language.

100. A method of indexing and searching timed media files, as recited in claim 99, wherein said identification of anaphora within the language includes identifying pronouns.

101. A method of indexing and searching timed media files, as recited in claim 100, wherein said processing of language further comprises the step of identifying and excluding from further analysis non-referential occurrences of pronouns.

102. A method of indexing and searching timed media files, as recited in claim 99, wherein said identification of anaphora within the language includes identifying definite references.

103. A method of indexing and searching timed media files, as recited in claim 99, wherein said identification of anaphora within the language includes identifying indirect anaphora or implicit references.

104. A method of indexing and searching timed media files, as recited in claim 99, wherein said processing of language further comprises the step of determining the antecedents of identified anaphora.

105. A method of indexing and searching timed media files, as recited in claim 104, wherein

said processing of language includes the calculation of one or more major topic shifts;

and

said determination of the antecedents of identified anaphora includes the use of major topic shifts.

106. A method of indexing and searching timed media files, as recited in claim 104, wherein

said processing of language includes the identification of named entities within the language;

and

said determination of the antecedents of identified anaphora includes the use of named entities that have been identified within the language.

107. A method of indexing and searching timed media files, as recited in claim 106, wherein

said processing of language further comprises the step of classification of named entities by type;

and

said determination of the antecedents of identified anaphora includes the use of the classification of named entities by type.

108. A method of indexing and searching timed media files, as recited in claim 104, wherein said determination of the antecedents of identified anaphora includes the use of ontological information.

109. A method of indexing and searching timed media files, as recited in claim 108, wherein said determination of the antecedents of identified anaphora includes the filtering of potential antecedents of personal pronouns using an ontology to determine if each potential antecedent could represent a human.

110. A method of indexing and searching timed media files, as recited in claim 108, wherein

said identification of anaphora within the language includes identifying definite references;

and

said determination of the antecedents of identified definite references includes the use of an ontology to filter the potential antecedents of each definite reference according to whether the potential antecedent is an example of the definite reference.

111. A method of indexing and searching timed media files, as recited in claim 108, wherein

said identification of anaphora within the language includes identifying indirect references;

and

said determination of the antecedents of identified definite references includes the use of an ontology to filter the potential antecedents of each definite reference according to whether the potential antecedent implies the existence of the indirect referent.

112. A method of indexing and searching timed media files, as recited in claim 104, wherein

a part of speech is determined for at least one information representation;

and

said determination of the antecedents of identified anaphora includes the use of part of speech information.

113. A method of indexing and searching timed media files, as recited in claim 10, wherein said determination of the antecedents of identified anaphora includes the filtering of potential antecedents of personal pronouns using an ontology or lexicon to determine the gender of the potential antecedents.

114. A method of indexing and searching timed media files, as recited in claim 10, wherein said determination of the antecedents of identified anaphora includes the use of an ontology or lexicon to determine whether potential antecedents take a singular or plural anaphor.

115. A method of indexing and searching timed media files, as recited in claim 10, wherein

said processing of language includes grammatical parsing of at least a portion of said language;

said determination of the antecedents of identified anaphora includes the identification of one or more grammatical constraints on one or more identified anaphora; and

said determination of the antecedents of the antecedents of identified anaphora further comprises the step of filtering potential antecedents according to the grammatical constraints.

116. A method of indexing and searching timed media files, as recited in claim 10, wherein

said processing of language includes grammatical parsing of at least a portion of said language;

and

said determination of the antecedents of identified anaphora includes the comparison of the grammatical role of an anaphora and the grammatical role of its potential antecedents.

117. A method of indexing and searching timed media files, as recited in claim 10, wherein said determination of the antecedents of identified anaphora includes

the identification of one or more semantic constraints on one or more identified anaphora;

and

filtering potential antecedents according to the semantic constraints.

118. A method of indexing and searching timed media files, as recited in claim 104, wherein said determination of the antecedents of identified anaphora includes the consideration of the distance between an anaphora and its potential antecedents.

119. A method of indexing and searching timed media files, as recited in claim 104, wherein

said processing of language further comprises the step of calculating a centrality number for at least one occurrence of at least one information representation contained within the timed media file;

and

said determination of the antecedents of identified anaphora includes the consideration of the centrality of the anaphor and the centrality of the potential antecedents.

120. A method of indexing and searching timed media files, as recited in claim 104, wherein

said calculation of at least one relevance interval for a given information representation includes the consideration of the temporal distribution of occurrences of said information representation;

and

said occurrences of the information representation include occurrences of one or more anaphora that have been determined to refer to the information representation.

121. A method of indexing and searching timed media files, as recited in claim 25, wherein said processing of language includes the disambiguation of at least one word or phrase.

122. A method of indexing and searching timed media files, as recited in claim 1, comprising the further step of creating a raw data index containing data used in and produced by said calculation of at least one relevance interval.

123. A method of indexing and searching timed media files, as recited in claim 122, further comprising storing said extracted data in said raw data index.

124. A method of indexing and searching timed media files, as recited in claim 122, wherein said calculation includes the processing of language contained within the timed media file, further comprising the step of storing results of said processing in said raw data index.

125. A method of indexing and searching timed media files, as recited in claim 1, further comprising the use of a quantitative model of semantic relatedness between at least one pair of information representations.

126. A method of indexing and searching timed media files, as recited in claim 125, wherein said quantitative model of semantic relatedness is built through the analysis of a corpus of spoken and/or written language.

127. A method of indexing and searching timed media files, as recited in claim 126, wherein said analysis includes calculating the frequency of co-occurrence of at least one pair of information presentations.

128. A method of indexing and searching timed media files, as recited in claim 127, wherein said model of semantic relatedness includes a measure of the mutual information, or probability of co-occurrence of the pair of information representations relative to the probability of random co-occurrence of such a pair of information representations, between the pair of information representations.

129. A method of indexing and searching timed media files, as recited in claim 128, wherein frequency of co-occurrence is measured as the number of occurrences of both information representations within a certain textual or temporal distance.

130. A method of indexing and searching timed media files, as recited in claim 129, wherein

said analysis of a corpus includes filtering to exclude at least one word that does not carry semantic information;

and

frequency of co-occurrence is measured as the number of occurrences of both information representations within a distance of a constant number of words that remain after filtering to exclude at least one word that does not carry semantic information.

131. A method of indexing and searching timed media files, as recited in claim 128, wherein if a pair of information representations co-occur so infrequently that the uncertainty in the calculated mutual information relative to the calculated mutual information is above an acceptable threshold, then the frequency of co-occurrence of the pair of information representations is considered to be the frequency that would occur in a true corpus if the information representations co-occur randomly given their individual frequency of occurrence.

132. A method of indexing and searching timed media files, as recited in claim 128, wherein if either or both of a pair of information representations occurs so infrequently that the uncertainty in the calculated mutual information is above an acceptable threshold, then the frequency of co-occurrence of the pair of information representations is considered to be the frequency that would occur in a true corpus if the information representations co-occur randomly given their individual frequency of occurrence.

133. A method of indexing and searching timed media files, as recited in claim 127, wherein

said analysis of a corpus includes the calculation of a centrality score for the occurrences of at least one information representation within the corpus;

and

said frequency of co-occurrence is calculated such that each occurrence of each information representation being considered is weighted according to its centrality score.

134. A method of indexing and searching timed media files, as recited in claim 127, wherein if a pair of information representations that do not co-occur within the corpus of language, then the frequency of co-occurrence of the pair of information representations is considered to be the frequency that would occur in a true corpus if the information representations co-occur randomly given their individual frequency of occurrence.

135. A method of indexing and searching timed media files, as recited in claim 126, wherein said analysis of a corpus includes the calculation of a centrality score for the occurrences of at least one information representation within the corpus.

136. A method of indexing and searching timed media files, as recited in claim 126, wherein said analysis of a corpus includes the lemmatization of occurrences of at least one information representation within the corpus.

137. A method of indexing and searching timed media files, as recited in claim 126, wherein said analysis of a corpus includes filtering to exclude at least one word that does not carry semantic information.

138. A method of indexing and searching timed media files, as recited in claim 125, wherein said calculation of at least one relevance interval for a given information representation includes the expansion of relevance intervals to include content that contains information representations that have a sufficiently high degree of semantic relatedness with the said information representation.

139. A method of indexing and searching timed media files, as recited in claim 1, wherein said calculation of at least one relevance interval for a given information representation includes the consideration of the temporal distribution of occurrences of said information representation.

140. A method of indexing and searching timed media files, as recited in claim 139, wherein said occurrences of the information representation include occurrences of synonyms and other information representations that have similar meanings.

141. A method of indexing and searching timed media files, as recited in claim 139, wherein

said processing of language includes the determination of sentences within the language;

and

said calculation of relevance intervals includes considering the initial relevance intervals for each spoken information representation to be the sentences that contain occurrences of said spoken information representation.

142. A method of indexing and searching timed media files, as recited in claim 1, wherein said calculation of at least one relevance interval for a given information representation includes the combining of any adjacent or overlapping relevance interval.

143. A method of indexing and searching timed media files, as recited in claim 1, further comprising the step of calculating a magnitude of relevance for at least one relevance interval for an indexing information representation.

144. A method of indexing and searching timed media files, as recited in claim 143, wherein said calculation of a magnitude of relevance for a relevance interval includes calculating the number of occurrences of the indexing information representation.

145. A method of indexing and searching timed media files, as recited in claim 143, wherein said calculation of a magnitude of relevance for a relevance interval includes calculating the number of anaphora that refer to the indexing information representation.

146. A method of indexing and searching timed media files, as recited in claim 143, wherein said calculation of a magnitude of relevance for a relevance interval includes calculating the length of the relevance interval.

147. A method of indexing and searching timed media files, as recited in claim 143, wherein said calculation of a magnitude of relevance for a relevance interval includes the use of a degree of semantic relatedness between the indexing information representation and other information representations occurring within the relevance interval.

148. A method of indexing and searching timed media files, as recited in claim 143, wherein said magnitudes of relevance are measures of a degree to which said relevance interval is relevant to said at least one information representation.

149. A method of indexing and searching timed media files, as recited in claim 143, wherein a ranking of said relevance intervals according to said magnitudes of relevance is displayed.

150. A method of indexing and searching timed media files, as recited in claim 1, further comprising the step of creating at least one virtual document comprising a collection of at least one of said relevance intervals.

151. A method of indexing and searching timed media files, as recited in claim 150, wherein said collection of at least one of said relevance intervals contains relevance intervals from multiple timed media files.

152. A method of indexing and searching timed media files, as recited in claim 150, further comprising the step of creating a search index containing said at least one virtual document.

153. A method of indexing and searching timed media files, as recited in claim 150, further comprising the step of calculating a magnitude of relevance for each said at least one virtual document.

154. A method of indexing and searching timed media files, as recited in claim 153, further comprising the steps of

calculating a magnitude of relevance for each said virtual document;

and

storing said magnitude of relevance for each said virtual document in said search index.

155. A method of indexing and searching timed media files, as recited in claim 153, wherein said magnitudes of relevance are measures of a degree to which said virtual document is relevant to said at least one information representation.

156. A method of indexing and searching timed media files, as recited in claim 153, wherein a ranking of said virtual documents according to said magnitudes of relevance is displayed.

157. A method of indexing and searching timed media files, as recited in claim 150, wherein said virtual documents are displayed.

158. A method of indexing and searching timed media files, as recited in claim 157, wherein said display of said virtual documents includes playing the said collection of relevance intervals back-to-back.

159. A method of indexing and searching timed media files, as recited in claim 157, wherein the viewer can expand the display of said virtual documents to include playing additional portions of the timed media file.

160. A method of indexing and searching timed media files, as recited in claim 1, further comprising the step of receiving at least one query information representation.

161. A method of indexing and searching timed media files, as recited in claim 160, further comprising the step of determining if at least two query information representations have been input.

162. A method of indexing and searching timed media files, recited in claim 161, wherein when at least two query information representations have been input, further comprises the step of determining if a user has requested a degree of accuracy higher than the default.

163. A method of indexing and searching timed media files, as recited in claim 162, wherein when said user has not requested a search with a degree of accuracy higher than the default, further comprises the steps of

creating a search index containing at least one relevance interval or virtual document;

and

comparing said at least two query information representations to said search index to perform step of creating at least one virtual document.

164. A method of indexing and searching timed media files, as recited in claim 162, wherein when said user has requested a search with a degree of accuracy higher than the default, further comprises the steps of

creating a raw data index containing data used in and produced by said calculation of at least one relevance interval

comparing said at least two query information representations to said raw data index to perform step of creating at least one virtual document.

165. A method of indexing and searching timed media files, as recited in claim 161, wherein when at least two query information representations have been input, further comprises the step of calculating a merged virtual document that is relevant to all of the said at least two query information representations.

166. A method of indexing and searching timed media files, as recited in claim 165, wherein said calculation of a merged virtual document includes the calculation of a relevant virtual document and a highly relevant virtual document.

167. A method of indexing and searching timed media files, as recited in claim 165, wherein said calculation of a merged virtual document includes the calculation of the intersection of the virtual documents associated with each of the query information representations for a given timed media file.

168. A method of indexing and searching timed media files, as recited in claim 167, wherein said calculation of a merged virtual document includes the calculation of the minimal expansion of the intersection of the virtual documents associated with each of the query information representations for a given timed media file such that the expanded intersection includes at least one occurrence of each query information representation.

169. A method of indexing and searching timed media files, as recited in claim 165, wherein said calculation of a merged virtual document includes the calculation of the union of the virtual documents associated with each of the query information representations for a given timed media file.

170. A method of indexing and searching timed media files, as recited in claim 165, further comprising he step of calculating a relevance magnitude for the said merged virtual document.

171. A method of indexing and searching timed media files, as recited in claim 1, further comprising the step of displaying at least one relevance interval to the user.