Method for combining multi-modal queries for search of multimedia data using time overlap or co-occurrence and relevance scores

• US 6,507,838 B1
• Filed: 06/14/2000
• Issued: 01/14/2003
• Est. Priority Date: 06/14/2000
• Status: Expired due to Term

• Current Assignee: Terrace Licensing LLC (IP Edge LLC)
• Original Assignee: International Business Machines Corporation
• Inventors: Syeda-Mahmood, Tanveer Fathima
• Primary Examiner(s): AMSBURY, WAYNE P
• Application Number:
  US09/593,206
• Time in Patent Office: 944 Days
• Field of Search: 707/3, 707/6, 707/102
• US Class Current: 1/1
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A method and structure for searching multi-media data having different modes using a query, includes processing the multi-media data to extract text and time reference points, identifying overlapping time periods when text in two or more of the modes corresponds to the query, and ranking a relevance of the overlapping time periods.

Claims

1. A method of searching multi-media data having different modes using a query, said method comprising:

processing said multi-media data to extract relevance scores and time reference points of matches to individual media modes;

identifying overlapping time periods when two or more of said modal matches correspond to said query; and

ranking a relevance of said overlapping time periods.

2. The method in claim 1, wherein said ranking includes finding an overlapping time period having a highest relevance score.

3. The method in claim 1, wherein said ranking includes segmenting said overlapping time period to identify beginning and ending events.

4. The method in claim 3, wherein said ranking includes calculating a relevance distribution based on a frequency of occurrence of said query in a time period.

5. The method in claim 1, wherein said ranking includes finding a largest number of different modes of overlap.

6. The method in claim 1, wherein said modes comprise two or more of audio, video, text, and graphic display.

7. The method in claim 1, wherein said query can have an input mode based on any of said modes and said method further comprises outputting results of said query in a mode consistent with said input mode.

8. A method of searching multi-media including audio, video, graphic display and written data using a query, said method comprising:

processing said multi-media data to extract relevance scores and time reference points;

identifying portions of said matching media modes that correspond to said query;

determining a relevance score for said matching mode;

assigning time periods for said matching mode;

identifying overlapping time periods;

determining a relevance timing score of said overlapping time periods; and

ranking said matching modes based on said relevance score and said relevance timing score.

9. The method in claim 8, wherein said ranking includes finding an overlapping time period having a highest relevance score.

10. The method in claim 8, wherein said relevance score comprises a relevance distribution based on a frequency of occurrence of said query in a time period.

11. The method in claim 8, wherein said ranking includes finding a largest number of different modes of overlap.

12. The method in claim 11, wherein said modes comprise two or more of said audio, video, graphic display, and written data.

13. The method in claim 8, wherein said query can have an input mode based on any of said modes and said method further comprises outputting results of said query in a mode consistent with said input mode.

14. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for searching multi-media data having different modes using a query, said method comprising:

processing said multi-media data to extract scores and time reference points;

identifying overlapping time periods when matches in two or more of said modes correspond to said query; and

ranking a relevance of said overlapping time periods.

15. The program storage device in claim 14, wherein said ranking includes finding an overlapping time period having a highest relevance score.

16. The program storage device in claim 14, wherein said ranking includes developing a relevance score of said modal data matching said query.

17. The program storage device in claim 16, wherein said ranking includes calculating a relevance distribution based on a frequency of occurrence of said query in a time period.

18. The program storage device in claim 14, wherein said ranking includes finding a largest number of different modes of overlap.

19. The program storage device in claim 14, wherein said modes comprise two or more of audio, video, text, and graphic display.

20. The program storage device in claim 14, wherein said query can have an input mode based on any of said modes and said method further comprises outputting results of said query in a mode consistent with said input mode.