Method and apparatus for indexing of topics using foils

• US 6,578,040 B1
• Filed: 06/14/2000
• Issued: 06/10/2003
• Est. Priority Date: 06/14/2000
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Syeda-Mahmood, Tanveer Fathima
• Primary Examiner(s): AMSBURY, WAYNE P
• Application Number:
  US09/593,207
• Time in Patent Office: 1,091 Days
• Field of Search: 707/102
• US Class Current: 1/1
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/7854 using shape G06F16/7837 takes precedence
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A method and structure for indexing multi-media data comprising deriving keywords from a first media type (slides), matching the keywords to a second media type (video), identifying an appearance of the first media type in the second media type, and calculating a co-occurrence of the keywords and the appearance of the first media type in the second media type. The invention produces an index of the second media type based on the co-occurrence.

Claims

1. A computer implemented method of indexing multi-media data comprising:

deriving keywords from a first media type;

matching said keywords to a second media type;

identifying an appearance of said first media type in said second media type; and

calculating a co-occurrence of said keywords and said appearance of said first media type in said second media type.

2. The method in claim 1, further comprising producing an index of said second media type based on said co-occurrence.

3. The method in claim 1, wherein said identifying comprises:

generating keyframes from said second media type;

extracting geometric keyframe features from said keyframes and geometric slide features from said first media type; and

matching said geometric slide features and said geometric keyframe features.

4. The method in claim 3, further comprising identifying background matching regions in said keyframes having colors matching colors of backgrounds in said second media type.

5. The method in claim 4, wherein said extracting of said geometric keyframe features is performed only in said background matching regions.

6. The method in claim 3, wherein said matching identifies which portion of said first media type has a highest number of geometric slide features matching geometric keyframe features in a keyframe of said keyframes.

7. The method in claim 3, wherein said extracting includes:

identifying changes in image intensity as edges;

forming curves connecting said edges;

identifying corners where said curves change direction;

grouping said curves into curve-groups; and

designating a sequence of three consecutive features in each of said curve-groups as basis triples.

8. The method in claim 7, wherein said matching comprises:

computing coordinates of said basis triples;

identifying which portion of said first media type has a highest number of basis triples matching basis triples in a keyframe of said keyframes.

9. The method in claim 3, wherein said generating of said keyframes comprises:

dividing said second media type into portions based upon scene changes; and

selecting one frame from each portion of said second media type as a keyframe.

10. The method in claim 1, wherein said calculating comprises:

processing said multi-media data to extract relevance scores and time reference points of matches to individual media modes;

identifying overlapping time periods of matching keywords and said appearance of said first media type in said second media type; and

ranking a relevance of said overlapping time periods.

11. The method in claim 10, wherein said ranking includes finding an overlapping time period having a highest relevance score.

12. The method in claim 10, wherein said ranking includes segmenting said overlapping time period to identify beginning and ending events.

13. The method in claim 10, wherein said ranking includes finding a largest number of different modes of overlap.

14. The method in claim 10, wherein said modes comprise two or more of audio, video, text, and graphic display.

15. A computer implemented method of indexing multi-media data comprising:

deriving keywords from slides;

matching said keywords to an audio portion of video data;

identifying an appearance of said slides in a video portion of said video data; and

calculating a co-occurrence of said keywords and said appearance of said slides in said video data.

16. The method in claim 15, further comprising producing an index of said video data based on said co-occurrence.

17. The method in claim 15, wherein said identifying comprises:

generating keyframes from said video portion;

extracting geometric keyframe features from said keyframes and geometric slide features from said slides; and

matching said geometric slide features and said geometric keyframe features.

18. The method in claim 17, further comprising identifying background matching regions in said keyframes having colors matching colors of backgrounds in said slides.

19. The method in claim 18, wherein said extracting of said geometric keyframe features is performed only in said background matching regions.

20. The method in claim 17, wherein said matching identifies which slide of said slides has a highest number of geometric slide features matching geometric keyframe features in a keyframe of said keyframes.

21. The method in claim 17, wherein said extracting includes:

identifying changes in image intensity as edges;

forming curves connecting said edges;

identifying corners where said curves change direction;

grouping said curves into curve-groups; and

designating a sequence of three consecutive features in each of said curve-groups as basis triples.

22. The method in claim 21, wherein said matching comprises:

computing coordinates of said basis triples;

identifying which slide of said slides has a highest number of basis triples matching basis triples in a key frame of said keyframes.

23. The method in claim 17, wherein said generating of said keyframes comprises:

dividing said video data into portions based upon scene changes; and

selecting one frame from each portion of said video as a keyframe.

24. The method in claim 15, wherein said calculating comprises:

processing said video data to extract relevance scores and time reference points of matching keywords and said appearance of said slide in said video portion;

identifying overlapping time periods of matching keywords and said appearance of said slides in said video portion; and

ranking a relevance of said overlapping time periods.

25. The method in claim 24, wherein said ranking includes finding an overlapping time period having a highest relevance score.

26. The method in claim 24, wherein said ranking includes segmenting said overlapping time period to identify beginning and ending events.

27. The method in claim 24, wherein said ranking includes finding a largest number of different modes of overlap.

28. The method in claim 24, wherein said modes comprise two or more of audio, video, text, and graphic display.