Embedding content-based searchable indexes in multimedia files

• US 10,095,694 B2
• Filed: 05/12/2016
• Issued: 10/09/2018
• Est. Priority Date: 10/16/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Thambiratnam, Albert J. K., Seide, Frank Torsten Bernd, Yu, Roger Peng
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US15/152,995
• Publication Number:
  US 20160259782A1
• Time in Patent Office: 880 Days
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection
• CPC Class Codes: G10L 15/065 Adaptation
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 5/76 Television signal recording
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 5/85 on discs or drums
• CPC Class Codes: H04N 5/907 using static stores, e.g. storage tubes or semiconductor memories H04N5/91 takes precedence
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

Content-based analysis is performed on multimedia content prior to encoding the multimedia content in the rendering chain of processing. A content-based index stream is generated based on the content-based analysis and the content-based index stream is embedded in the multimedia file during rendering. The content-based index stream can be used to generate a content-based searchable index when necessary.

Claims

1. A computing system comprising:

a processor; and

memory storing instructions executable by the processor, wherein the instructions, when executed, configure the computing system to provide;

a multimedia file generation system configured to receive a combined file that includes a plurality of source tracks and a multimedia file with a first content-based searchable index therein; and

a content analysis engine configured to;

generate a content-based index stream only for a portion of the combined file that is without a corresponding content-based index; and

generate a second content-based searchable index based on the content-based index stream only for the portion of the combined file that is without the corresponding content-based index.

2. The system of claim 1, wherein the content analysis engine is configured to identify the portion of the combined file that is without the corresponding content-based index.

3. The system of claim 2, wherein the instructions configure the computing system to provide:

a file writer configured to write a combined multimedia file by merging the first and second content-based searchable indices therein.

4. The system of claim 3, wherein the instructions configure the computing system to provide:

a rendering component configured to render the plurality of source tracks.

5. The system of claim 1, wherein the content analysis engine is configured to generate the second content-based searchable index based on speech content in the source tracks.

6. The system of claim 1, wherein the content analysis engine is configured to generate the second content-based searchable index based on non-speech audio content in the source tracks.

7. The system of claim 1, wherein the content analysis engine is configured to generate the second content-based searchable index based on video content in the source tracks.

8. The system of claim 1, wherein the content analysis engine is configured to generate the content-based index stream by analyzing content of the source tracks to obtain content-based index information used to perform content-based retrieval of the combined file and generating a time-based stream including the content-based index information.

9. A computing system comprising:

a processor; and

memory storing instructions executable by the processor, wherein the instructions, when executed, configure the computing system to;

receive a combined file that includes a plurality of source tracks and a multimedia file with a first content-based searchable index therein;

identify a portion of the combined file that is without a corresponding content-based index;

generate a content-based index stream only for the identified portion of the combined file that is without a corresponding content-based index, based on both speech content and non-speech audio content in the identified portion of the combined file;

generate a second content-based searchable index based on the content-based index stream only for the portion of the combined file that is without the corresponding content-based index; and

write a combined multimedia file by merging the first and second content-based searchable indices therein.

10. The computing system of claim 9, wherein the instructions configure the computing system:

a rendering component configured to render the plurality of source tracks.

11. The computing system of claim 9, wherein the instructions configure the computing system to generate the second content-based searchable index based on speech content in the source tracks.

12. The computing system of claim 9, wherein the instructions configure the computing system to generate the second content-based searchable index based on non-speech audio content in the source tracks.

13. The computing system of claim 9, wherein the instructions configure the computing system to generate the second content-based searchable index based on video content in the source tracks.

14. The computing system of claim 9, wherein the instructions configure the computing system to generate the content-based index stream by

analyzing content of the source tracks to obtain content-based index information used to perform content-based retrieval of the combined file, and

generating a time-based stream including the content-based index information.

15. A computing system comprising:

a processor; and

memory storing instructions executable by the processor, wherein the instructions, when executed, configure the computing system to;

receive a plurality of source tracks and a first content-based searchable index associated with the plurality of source tracks;

generate a content-based index stream only for a portion of the plurality of source tracks that is without a corresponding content-based index; and

generate a second content-based searchable index based on the content-based index stream only for the portion of the plurality of source tracks that is without the corresponding content-based index.

16. The computing system of claim 15, wherein the instructions configure the computing system to identify the portion of the plurality of source tracks that is without the corresponding content-based index.

17. The computing system of claim 15, wherein the instructions configure the computing system to write a combined multimedia file by merging the first and second content-based searchable indices therein.

18. The computing system of claim 15, wherein the instructions configure the computing system to generate the second content-based searchable index based on at least one of:

speech content in the source tracks;

non-speech audio content in the source tracks;

or

video content in the source tracks.

19. The computing system of claim 15, wherein the instructions configure the computing system to:

generate the content-based index stream by analyzing content of the source tracks to obtain content-based index information used to perform content-based retrieval of the source tracks and generating a time-based stream including the content-based index information.

20. The computing system of claim 19, wherein the instructions configure the computing system to:

encode the content-based index stream into an encoded content-based index stream; and

generate the second content-based searchable index based on the encoded content-based index stream.