Embedding content-based searchable indexes in multimedia files

• US 20080089665A1
• Filed: 10/16/2006
• Published: 04/17/2008
• Est. Priority Date: 10/16/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Seide, Frank Torstein, Thambiratnam, Albert J.K., Yu, Roger Peng
• Granted Patent: US 8,121,198 B2
• Time in Patent Office: Days
• US Class Current: 386/95
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection
• CPC Class Codes: G10L 15/065 Adaptation
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 5/76 Television signal recording
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 5/85 on discs or drums
• CPC Class Codes: H04N 5/907 using static stores, e.g. storage tubes or semiconductor memories H04N5/91 takes precedence
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

Content-based analysis is performed on multimedia content prior to encoding the multimedia content in the rendering chain of processing. A content-based index stream is generated based on the content-based analysis and the content-based index stream is embedded in the multimedia file during rendering. The content-based index stream can be used to generate a content-based searchable index when necessary.

Claims

1. A system for indexing multimedia content, comprising:

a rendering engine configured to receive multimedia source tracks and generate rendered source streams;

a content analysis engine configured to receive the source tracks and generate a content-based index stream based on the source tracks;

a set of encoders coupled to the rendering component and configured to encode the rendered-source streams and the content-based index stream into encoded multimedia streams and an encoded content-based index stream; and

a multimedia file writer component configured to receive the encoded multimedia streams and the encoded content-based index stream and to write the encoded multimedia streams to a first multimedia file with the encoded content-based index stream associated with the first multimedia file.

2. The system of claim 1 wherein the multimedia file writer is configured to write the first multimedia file to include the content-based index stream within it.

3. The system of claim 1 and further comprising:

an indexing component configured to receive the first multimedia file and associated content-based index stream and generate a content-based searchable index from the content-based index stream.

4. The system of claim 3 wherein the indexing component is configured to embed the content-based searchable index into the first multimedia file.

5. The system of claim 1 wherein the content analysis engine is configured to receive a second multimedia file that includes the first multimedia file and to identify a portion of the second multimedia file that does not have a content-based index stream generated therefore.

6. The system of claim 5 wherein the content analysis engine is configured to generate a second content-based index stream for only that portion of the second multimedia file identified.

7. The system of claim 6 and further comprising an indexing component configured to receive the second multimedia file and generate a content-based searchable index for only the second content-based index stream.

8. A method of generating a multimedia file, comprising:

receiving a plurality of source tracks;

rendering the source tracks to obtain a plurality of multimedia streams;

generating a content-based index stream based on the source tracks;

after generating the content-based index stream, encoding the multimedia streams; and

writing the encoded multimedia streams to a multimedia file and associating the content-based index stream with the multimedia file.

9. The method of claim 8 wherein associating the content-based index stream with the multimedia file comprises:

writing the content-based index stream to the multimedia file.

10. The method of claim 8 and further comprising:

generating a content-based searchable index for the multimedia file based on the content-based index stream.

11. The method of claim 8 wherein generating a content-based index stream comprises:

analyzing content of the source tracks to obtain content-based index information used to perform content-based retrieval of the multimedia file; and

generating a time-based stream including the content-based index information.

12. The method of claim 8 and further comprising:

receiving the multimedia file with incorporated multimedia content that has no content-based index stream associated therewith; and

generating a content-based index stream for only the incorporated multimedia content.

13. The method of claim 8 wherein generating a content-based index stream comprises:

generating the content-based index stream based on speech content in the source tracks.

14. The method of claim 8 wherein generating a content-based index stream comprises:

generating the content-based index stream based on non-speech audio content in the source tracks.

15. The method of claim 8 wherein generating a content-based index stream comprises:

generating the content-based index stream based on video content in the source tracks.

16. A method of generating searchable multimedia content, comprising:

receiving a plurality of source tracks;

receiving a multimedia file with a first content-based searchable index therein;

combining the source tracks and multimedia file to obtain a combined file;

generating a content-based index stream only for a portion of the combined file that is without the content-based index; and

generating a second content-based searchable index based on the content-based index stream.

17. The method of claim 16 and further comprising:

generating a combined multimedia file by merging the first and second content-based searchable indexes therein.

18. The method of claim 16 and further comprising:

after receiving the multimedia file with the content-based searchable index, loading the content-based searchable index into a search index.

19. The method of claim 18 and further comprising:

loading the second content-based index into the search index.

20. The method of claim 19 and further comprising:

performing multimedia content retrieval by searching the search index.