Method and system for querying multimedia data where adjusting the conversion of the current portion of the multimedia data signal based on the comparing at least one set of confidence values to the threshold

• US 7,487,072 B2
• Filed: 08/04/2004
• Issued: 02/03/2009
• Est. Priority Date: 08/04/2004
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Semple, Bruce P., Stein, Frank L.
• Primary Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US10/912,616
• Publication Number:
  US 20060031216A1
• Time in Patent Office: 1,644 Days
• Field of Search: 707/204, 707 1- 3, 707/5, 707/7, 707/10, 707/100, 707/104.1, 707/200, 707/102, 715/201, 715/205, 715/255, 709/203, 709223-224, 345/440
• US Class Current: 703/3
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

Methods and systems are provided that coordinate the operation of multiple search engines through portions of a multimedia object based on the anticipated accuracy of each search engine. One feature of the present invention provides a content management system that efficiently archives and retrieves multimedia content, for example, based on a confidence level or index. As multimedia content is generated, it is analyzed and cataloged into a set of indexes based on the analysis and confidence level of the analysis. The multimedia content is then stored into one or more databases. When one or more items are requested by a query, the query is analyzed and broken down into components or subqueries. These components or subqueries can then be routed to one or more appropriate search engine as a function of the confidence index. The results from the search engines are then gathered and compiled for presentation to a user.

Claims

1. A method of storing and searching multimedia data, the method comprising:

receiving a multimedia data signal having content that progresses based on a timing;

converting the multimedia data signal into a searchable format;

identifying a set of features in the searchable format and outputting the set of features as content index data;

determining a set of confidence values for the set of features in the searchable format that estimates the accuracy of the set of features in the searchable format as a function of the timing of the multimedia data signal and outputting the confidence values as confidence index data;

linking the content index data and the confidence index data with the multimedia data signal as a function of the timing of the multimedia data signal;

storing the content index data and the confidence index data into a database that is indexed based on the set of features and the confidence values;

receiving, from a requestor, a query that specifies a set of temporal parameters and content parameters;

parsing the query into a set of sub-queries;

selecting respective search engines for each of the sub-queries, wherein the set of the sub-queries comprises at least two sub-queries, wherein the at least two sub-queries are transmitted to at least two different search engines based on confidence index data among the stored confidence index data which corresponds to the specified set of at least the content parameters;

receiving, in response to the transmitted at least two sub-queries, results from the at least two different search engines; and

transmitting the received results to the requestor,

wherein the converting the multimedia data signal into the searchable format comprises;

receiving at least one of the set of confidence values for a previous portion of the multimedia data signal; and

adjusting the conversion of a current portion of the multimedia data signal based on the received at least one set of confidence values,

wherein the adjusting the conversion of the current portion of the multimedia data signal comprises;

comparing the at least one set of confidence values to a threshold; and

adjusting the conversion of the current portion of the multimedia data signal based on the comparison of the at least one set of confidence values to the threshold.

2. The method of claim 1, wherein converting the multimedia data signal comprises:

identifying an event in the multimedia data signal; and

selectively converting the multimedia data signal based on the event.

3. The method of claim 2, wherein identifying the event in the multimedia data signal comprises identifying a time code in the multimedia data signal.

4. The method of claim 2, wherein identifying the event in the multimedia data signal comprises identifying a closed caption control sequence in the multimedia data signal.

5. The method of claim 2, wherein identifying the event in the multimedia data signal comprises identifying a scene change in the multimedia data signal.

6. The method of claim 2, wherein identifying the event in the multimedia data signal comprises detecting a threshold level of audio in the multimedia data signal.

7. The method of claim 2, wherein identifying the event in the multimedia data signal comprises identifying a slate in a video stream of the multimedia data signal.

8. The method of claim 1, wherein the receiving the results further comprises:

gathering a first set of results from each of the search engines that satisfy their respective sub-queries; and

combining the first set of results into a second set of results based on satisfying the temporal parameters and content parameters of the query.

9. The method of claim 8, wherein selecting the respective search engines for each of the subqueries further comprises selecting respective search engines based on a set of location parameters in addition to the temporal parameters and the content parameters.

10. The method of claim 8, wherein the combining the first set of results into a second set of results comprises identifying results from different multimedia data items that occur within a period of time that is specified in the query.

11. The method of claim 8, wherein receiving the query comprises receiving a query that specifies temporal constraints across a plurality of media types.

12. The method of claim 8, wherein receiving the query comprises receiving a query that specifies an offset of time within a multimedia data item.

13. The method of claim 8, wherein the parsing the query into the set of the sub-queries comprises parsing the query into sub-queries based on a type of media specified by the content parameters.

14. The method of claim 8, wherein selecting the respective search engines for each of the sub-queries further comprises selecting the respective search engines based on the temporal parameters and selecting a first search engine that searches a first time period of a multimedia item based on the confidence values and selecting a second search engine that searches a second time period of the multimedia item based on the confidence values.

15. The method of claim 8, wherein combining the first set of results into the second of results based on satisfying the temporal parameters and content parameters of the query comprises combining the first set of results into a set of references to multimedia data items that are ranked in terms of their relevance to the temporal parameters and the content parameters.

16. The method of claim 8, wherein the at least two sub-queries indicate two distinct events in the searchable format of the multimedia data;

and the specified set of temporal parameters represent a time period between the two distinct events,

wherein the first set of results comprises at least one first event received in response to a first sub-query among the at least two sub-queries from a first search engine among the at least two different search engines, and at least one second event received in response to a second sub-query among the at least two sub-queries from a second search engine among the at least two different search engines, and

wherein the combining the first set of results into the second set of results further comprises generating the second set of results by classifying the at least one first event and the at least one second event based on the time period.

17. A system, executed on a processor, for storing and searching multimedia data, the system comprising:

a detector that detects a multimedia data signal having content that progresses based on a timing;

a converter, coupled to the detector, that converts the multimedia data signal into a searchable format;

an analyzer, coupled to the converter, that identifies a set of features in the searchable format and outputs the set of features as content index data, and the analyzer further determines a set of confidence values for the set of features in the searchable format, wherein the set of confidence values estimate the accuracy of the set of features in the searchable format as a function of the timing of the multimedia data signal, the confidence values output as confidence index data;

a module that links the content index data and the confidence index data with the multimedia data signal as a function of the timing of the multimedia data signal;

a set of storage devices that store the content index data and the confidence index data into a database that is indexed based on the set of features and the confidence values; and

a server configured to receive, from a requestor, a query that specifies a set of temporal parameters and content parameters, parse the query into a set of sub-queries, wherein the set of the sub-queries comprises at least two sub-queries, the server further selecting respective search engines for each of the sub-queries, wherein the at least two sub-queries are transmitted to at least two different search engines based on confidence index data among the stored confidence index data which corresponds to the specified set of at least the content parameters, and the server, in response to the transmitted at least two sub-queries, receives results from the at least two different search engines, and transmits the received results to the requestor,

wherein the server is further configured to gather a first set of results from each of the search engines that satisfy their respective sub-queries, and combine the first set of results into a second set of results based on satisfying the temporal parameters and content parameters of the query,

wherein the at least two sub-queries indicate two distinct events in the searchable format of the multimedia data; and

the specified set of temporal parameters represent a time period between the two distinct events,

wherein the first set of results comprises at least one first event received in response to a first sub-query among the at least two sub-queries from a first search engine among the at least two different search engines, and at least one second event received in response to a second sub-query among the at least two sub-queries from a second search engine among the at least two different search engines, and

the server combines the first set of results into the second set of results by classifying the at least one first event and the at least one second event based on the time period to generate the second set of results.

18. A system, executed on a processor, for storing and searching multimedia data, the system comprising:

means for receiving a multimedia data signal having content that progresses based on a timing;

means for converting the multimedia data signal into a searchable format;

means for identifying a set of features in the searchable format and outputting the set of features as content index data;

means for determining a set of confidence values for the set of features in the searchable format that estimates the accuracy of the set of features in the searchable format as a function of the timing of the multimedia data signal and outputting the confidence values as confidence index data;

means for linking the content index data and the confidence index data with the multimedia data signal as a function of the timing of the multimedia data signal;

means for storing the content index data and the confidence index data into a database that is indexed based on the set of features and the confidence values; and

means for receiving, from a requestor, a query that specifies a set of temporal parameters and content parameters, parsing the query into a set of sub-queries, wherein the set of the sub-queries comprises at least two sub-queries, the means for receiving further selecting respective search engines for each of the sub-queries, wherein the at least two sub-queries are transmitted to two different search engines based on confidence index data among the stored confidence index data which corresponds to the specified set of at least the content parameters, and the means for receiving, in response to the transmitted at least two sub-queries, receives results from the two different search engines, and transmits the received results to the requestor,

wherein the means for converting converts the multimedia data signal into the searchable format by receiving at least one of the set of confidence values for a previous portion of the multimedia data signal, and adjusts the conversion of a current portion of the multimedia data signal based on the received at least one set of confidence values,

wherein the means for converting adjusts the conversion of the current portion of the multimedia data signal by comparing the at least one set of confidence values to a threshold, and adjusts the conversion of the current portion of the multimedia data signal based on the comparison of the at least one set of confidence values to the threshold.