Method and apparatus for creating a searchable digital video library and a system and method of using such a library

• US 5,835,667 A
• Filed: 10/14/1994
• Issued: 11/10/1998
• Est. Priority Date: 10/14/1994
• Status: Expired due to Term

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Smith, Michael A., Wactlar, Howard D., Reddy, D. Raj, Kanade, Takeo, Mauldin, Michael L., Stevens, Scott M., Christel, Michael G.
• Primary Examiner(s): Tran, Thai
• Assistant Examiner(s): NGUYEN, HUY THANH
• Application Number:
  US08/324,076
• Time in Patent Office: 1,488 Days
• Field of Search: 358/341, 358/343, 358/335, 358/311, 360/19.1, 360/13, 348/552, 369/53, 364/419, 395/2, 386/95, 386/96, 386/52, 386/55, 386/46, 386/105, 386/106, 386/97, 386/111
• US Class Current: 386/241
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

An apparatus and method of creating a digital library from audio data and video images. The method includes the steps of transcribing the audio data and marking the transcribed audio data with a first set of time-stamps and indexing the transcribed audio data. The method also includes the steps of digitizing the video data and marking the digitized video data with a second set of time-stamps related to the first set of time-stamps and segmenting the digitized video data into paragraphs according to a set of rules. The steps of storing the indexed audio data and the digitized video data with their respective sets of time-stamps is also provided. The method also includes the step of passing the transcribed audio data through a natural language interpreter before indexing the transcribed audio data. A method and apparatus for searching the digital library is disclosed.

Claims

1. A method of creating an independent digital library from existing audio data and video images comprising the steps of:

transcribing said audio data and marking said transcribed audio data with a first set of time-stamps;

indexing said transcribed audio data;

digitizing said video data and marking said digitized video data with a second set of time-stamps related to said first set of time-stamps;

segmenting said digitized video data into video paragraphs according to a set of rules based upon scene characterization of said video images and the processing of said audio data; and

storing said indexed audio data and said segmented digitized video data with their respective sets of time-stamps to create the digital library which can be accessed through said indexed audio data without returning to the existing audio data and video images.

2. The method of claim 1 additionally comprising the step of passing said transcribed audio data through a natural language interpreter before indexing said transcribed audio data.

3. The method of claim 2 wherein said natural language interpreter updates said set of rules.

7. The method of claim 1 additionally comprising the step of generating a set of icons after segmenting said digitized video data into video paragraphs according to said set of rules.

8. The method of claim 7 wherein said set of icons is a set of intelligent moving icons.

9. The method of claim 8 wherein said set of intelligent moving icons is generated using data-dependent heuristics.

10. The method of claim 1 additionally comprising the step of compressing said digitized video data before storing said indexed audio data and said digitized video data with their respective sets of time-stamps.

11. The method of claim 1 wherein the step of transcribing said audio data and marking said transcribed audio data with a first set of time stamps includes the steps of:

producing a set of possible word occurrences, with each word occurrence having a start time and a plurality of possible end times;

producing a plurality of possible begin times for each of said end times;

generating a set of N-best hypotheses for said audio data; and

selecting a best-scoring hypothesis from said set of N-best hypotheses to produce said transcribed audio data.

12. The method of claim 11 wherein said set of possible word occurrences is produced using a forward time synchronous pass function.

13. The method of claim 11 wherein said plurality of possible begin times are produced using a reverse time synchronous function.

14. The method of claim 2 wherein the step of passing said transcribed audio data through a natural language interpreter before indexing said transcribed audio data includes the steps of:

summarizing said transcribed audio data;

tagging said transcribed audio data using data extraction techniques; and

correcting said tagged transcribed audio data using semantic and syntactic constraints and a phonetic knowledge base.

15. The method of claim 1 wherein said digitized video data are segmented into video paragraphs using comprehensive image statistic rules.

16. The method of claim 1 wherein said digitized video data are segmented into video paragraphs using camera motion rules.

17. The method of claim 1 wherein said digitized video data are segmented into video paragraphs using object motion rules.

18. The method of claim 1 wherein said digitized video data are segmented into video paragraphs using deformable templates and filtering rules.

19. The method of claim 1 wherein said digitized video data are segmented into video paragraphs using object presence rules.

20. The method of claim 1 wherein said digitized video data are segmented into video paragraphs using three-dimensional understanding rules.

4. An apparatus for creating an independent digital library from existing audio data and video images, comprising:

means for transcribing said audio data and marking said transcribed audio data with a first set of time-stamps;

means for indexing said transcribed audio data;

means for digitizing said video data and marking said digitized video data with a second set of time-stamps related to said first set of time-stamps;

means for storing a set of rules based upon scene characterization of said video images and the processing of said audio data;

means for segmenting said digitized video data into video paragraphs according to said stored set of rules; and

means for storing said indexed audio data and said segmented digitized video data with their respective sets of time-stamps to create the digital library which can be accessed through said indexed audio data without returning to the existing audio data and video images.

5. The apparatus of claim 4 additionally comprising natural language interpreter means for processing said transcribed audio data before said data is indexed.

6. The apparatus of claim 4 wherein said natural language interpreter means updates said set of rules.

21. The apparatus of claim 4 additionally comprising means for generating a set of icons after said digitized video data is segmented into paragraphs according to said set of rules.

22. The apparatus of claim 21 wherein said set of icons is a set of intelligent moving icons.

23. The apparatus of claim 22 wherein said means for generating said set of intelligent moving icons uses data-dependent heuristics.

24. The apparatus of claim 4 additionally comprising means for compressing said digitized video data before said indexed audio data and said digitized video data are stored with their respective sets of time-stamps.

25. The apparatus of claim 4 wherein said means for transcribing said audio data and marking said transcribed audio data with a first set of time stamps comprises:

means for producing a set of possible word occurrences, with each word occurrence having a start time and a plurality of possible end times;

means for producing a plurality of possible begin times for each of said end times;

means for generating a set of N-best hypotheses for said audio data; and

means for selecting a best-scoring hypothesis from said set of N-best hypotheses to produce said transcribed audio data.

26. The apparatus of claim 25 wherein said means for producing said set of possible word occurrences uses a forward time synchronous pass function.

27. The apparatus of claim 25 wherein said means for producing said plurality of possible begin times uses a reverse time synchronous function.

28. The apparatus of claim 5 wherein said means for passing said transcribed audio data through a natural language interpreter before indexing said transcribed audio data comprises:

means for summarizing said transcribed audio data;

means for tagging said transcribed audio data using data extraction techniques; and

means for correcting said tagged transcribed audio data using semantic and syntactic constraints and a phonetic knowledge base.

29. The apparatus of claim 4 wherein said means for segmenting said digitized video data into video paragraphs uses comprehensive image statistic rules.

30. The apparatus of claim 4 wherein said means for segmenting said digitized video data into video paragraphs uses camera motion rules.

31. The apparatus of claim 4 wherein said means for segmenting said digitized video data into video paragraphs uses object motion rules.

32. The apparatus of claim 4 wherein said means for segmenting said digitized video data into video paragraphs uses deformable templates and an filtering rules.

33. The apparatus of claim 4 wherein said means for segmenting said digitized video data into video paragraphs uses object presence rules.

34. The apparatus of claim 4 wherein said means for segmenting said digitized video data into video paragraphs uses three-dimensional understanding rules.